Observability, cost, and operations
Export PDFLaunching is the beginning. In production you need to see what every request did, know when quality or latency drifts, and keep the bill predictable. This lesson covers tracing with OpenTelemetry and Application Insights, the metrics to watch, cost drivers and the levers that reduce them, reliability patterns for quota and outages, and how to automate deployment with CI/CD and govern usage with an AI gateway.
Tracing: see every step
A record of one request's journey, made of nested spans: the agent run, each model call (with tokens and latency), each tool call (with arguments and result), and your own functions. Foundry uses the open OpenTelemetry standard and its GenAI semantic conventions.
Azure Monitor's application performance service. Connect one to your Foundry project and traces appear both in the Foundry portal's tracing view and in Application Insights, where you can query them with KQL and build alerts.
Agents run by the Agent Service are traced automatically once Application Insights is connected (portal: your project → Tracing → connect). To trace your own client code too:
python -m pip install azure-monitor-opentelemetry opentelemetry-instrumentation-openai-v2This example calls your Azure resources, so it needs a deployed project and a signed-in identity. Output will differ on every run.
import os
from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor, trace_function
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
project = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
credential=DefaultAzureCredential())
# send OpenTelemetry data to the project's Application Insights
configure_azure_monitor(connection_string=project.telemetry.get_application_insights_connection_string())
AIProjectInstrumentor().instrument(enable_content_recording=False) # Foundry SDK spans
OpenAIInstrumentor().instrument() # OpenAI client spans
@trace_function() # your own function becomes a span with its arguments
def answer(question: str) -> str:
with project.get_openai_client() as client:
return client.responses.create(model=os.environ["FOUNDRY_MODEL_DEPLOYMENT"],
input=question).output_textContent recording captures full prompts and responses in traces — invaluable for debugging, risky for personal data. Keep it off in production unless you have a data-handling basis, or enable it on a sampled, access-controlled basis. The environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT controls it for OpenAI instrumentation.
Metrics to watch
| Metric | Why it matters | Typical alert |
|---|---|---|
| Requests and errors by status code | Spot outages, auth problems, and 429s | Error rate above 2% for 5 minutes |
| Rate-limited requests (429) | You are out of quota | Any sustained 429s on a production deployment |
| Time to first token / total latency (p50, p95) | User experience | p95 latency doubles versus last week |
| Input, output, and cached tokens | Cost and prompt growth | Daily tokens 50% above forecast |
| Provisioned utilisation | PTU sizing | Above 90% (add capacity) or below 40% (overpaying) |
| Guardrail blocks | Abuse or overly strict policy | Sudden spike in blocks |
| Tool failures and rounds per request | Agent health | Tool error rate up, or average rounds creeping up |
| Online evaluation scores | Quality drift | Groundedness average drops below the release gate |
Model deployment metrics live in Azure Monitor on the Foundry resource; agent and application metrics come from traces in Application Insights; the Foundry portal shows an overview dashboard per project and agent.
What drives cost
| Cost driver | Billed by | Notes |
|---|---|---|
| Model tokens | Input, cached input, and output tokens per deployment | Usually the largest line; output tokens cost several times more than input |
| Provisioned throughput | PTU-hours or reservations | Fixed, whether used or not |
| Agent tools | Per session or per call (code interpreter, web search, file search storage) | Check each tool's pricing |
| Azure AI Search | Tier and replicas/partitions per hour, plus semantic ranker queries | Often the second-largest line for RAG |
| Vector store and file storage | GB per day | Clean up unused stores |
| Evaluation | Judge-model tokens for AI-assisted evaluators | Large datasets add up |
| Application Insights | GB ingested | Content recording increases volume |
Cost levers
- Right-size the model. Use the smallest model that passes your evaluation; route simple traffic to small models (or the model router).
- Shorten outputs. Ask for concise answers and set a realistic
max_output_tokens. - Trim inputs. Retrieve fewer, better chunks; summarise long histories.
- Use prompt caching. Put stable content (instructions, tool definitions, few-shot examples) at the start of the prompt so repeated prefixes are billed at the cached-input discount.
- Batch offline work with Global Batch deployments for roughly half the per-token price.
- Move steady high volume to PTU once the break-even point is reached.
This offline calculator compares the levers for a workload of 2,000 input and 400 output tokens per request, with illustrative prices:
# Illustrative prices only; look up current numbers on the Azure pricing page.
PRICE_IN_PER_M = 0.40 # USD per 1M input tokens (pay-as-you-go)
PRICE_OUT_PER_M = 1.60 # USD per 1M output tokens
CACHED_DISCOUNT = 0.75 # cached input tokens billed at 25% of the input price
PTU_MONTHLY = 2_000.0 # USD per provisioned unit per month (hypothetical)
PTU_UNITS = 15 # smallest purchase for this model (hypothetical)
def paygo_monthly(req_per_day, tokens_in, tokens_out, cache_hit=0.0, batch=False):
days = 30
cached = tokens_in * cache_hit
fresh = tokens_in - cached
cost_in = (fresh + cached * (1 - CACHED_DISCOUNT)) * PRICE_IN_PER_M / 1e6
cost_out = tokens_out * PRICE_OUT_PER_M / 1e6
per_request = cost_in + cost_out
if batch:
per_request *= 0.5 # Global Batch: ~50% off, 24h turnaround
return per_request * req_per_day * days
ptu = PTU_MONTHLY * PTU_UNITS
for volume in (10_000, 200_000, 1_000_000, 2_000_000):
base = paygo_monthly(volume, 2_000, 400)
cached = paygo_monthly(volume, 2_000, 400, cache_hit=0.6)
batch = paygo_monthly(volume, 2_000, 400, batch=True)
better = "PTU" if ptu < cached else "pay-as-you-go"
print(f"{volume:>9,}/day paygo=${base:>9,.0f} +cache=${cached:>9,.0f} "
f"batch=${batch:>9,.0f} PTU=${ptu:,.0f} -> {better}")At low volume, pay-as-you-go wins easily and caching saves 25%. Around a million requests a day, the fixed PTU price becomes cheaper than even cached pay-as-you-go — if those PTUs can actually serve the traffic, which you must confirm with the capacity calculator. Batch is cheapest of all for work that can wait.
Use Azure Cost Management to set budgets and alerts on the resource group, and tag resources (app, env, owner) so costs can be split by team.
Reliability patterns
| Problem | Pattern |
|---|---|
| 429s during peaks | Retry with backoff; raise quota; spread across deployments; PTU with spillover to Standard |
| Regional outage | A second Foundry resource in another region; fail over on repeated 5xx |
| Model version retired | Pinned versions, a calendar of retirement dates, evaluation before upgrade |
| Slow responses | Streaming, smaller models for simple tasks, shorter outputs, lower reasoning effort |
| Agent stuck in tool loops | Cap tool rounds and tokens; alert on rounds per request |
Governance with an AI gateway
A proxy — typically Azure API Management with its generative AI policies — placed between apps and Foundry endpoints. It centralises authentication, enforces per-app token limits, load-balances across deployments and regions, caches semantically similar requests, and emits per-consumer token metrics for chargeback.
Introduce a gateway when several teams share Foundry resources and you need fair usage and cost attribution. Foundry also integrates with Azure Policy (for example, allowing only approved models or requiring private endpoints) and with Microsoft Defender and Purview for security and data governance.
CI/CD for AI apps
pull request
├─ lint + unit tests (tool functions, parsers)
├─ deploy to dev project (Bicep/Terraform + new agent version)
├─ run evaluation on the golden set ──► gate: groundedness ≥ 4.0, safety = 0 flags
└─ (optional) red-team scan
merge to main
├─ deploy infra to prod (same templates, prod parameters)
├─ create agent version in prod project, tagged with git SHA
├─ smoke test + switch traffic
└─ continuous evaluation + alerts on production trafficStore agent definitions (instructions, tool schemas, model name) as files in Git, not just in the portal, so every change is reviewed and reproducible. The Azure Developer CLI (azd) and Foundry templates help bootstrap this setup.
- Change
cache_hitto 0.9 and find the new volume where PTU wins. - Connect Application Insights to your project, run an agent conversation, and open the trace to see each span's tokens and latency.
- Create a budget alert on your course resource group at 10 USD.
- Connect Application Insights for OpenTelemetry traces of agents, model calls, tools, and your own functions.
- Watch errors, 429s, latency, tokens, PTU utilisation, guardrail blocks, tool failures, and online evaluation scores.
- Tokens dominate cost: right-size models, shorten outputs, trim inputs, cache stable prefixes, batch offline work, and move steady volume to PTU.
- Plan for quota limits, outages, and version retirements with retries, multi-region failover, and pinned versions.
- Use CI/CD with evaluation gates, and an AI gateway when many teams share capacity.