Advanced

πŸ“Š OpenTelemetry

Instrument your agents with distributed tracing, custom metrics, and structured logging β€” then visualise everything in the free .NET Aspire Dashboard.

Agent Framework RC-1 exposes OpenTelemetry hooks at both the chat client level and the agent level via the .UseOpenTelemetry() builder extension. This means you get automatic spans for every LLM call, tool invocation, and streaming chunk β€” without writing instrumentation code yourself.

The example below wires up a full observability stack: traces, metrics (interaction count + response latency), and structured logs β€” all exported to an OTLP endpoint. You can use the .NET Aspire Dashboard (one Docker command), Application Insights, or any OTLP-compatible backend.

Key Concepts

  • .UseOpenTelemetry() β€” builder extension that auto-instruments agent and chat-client spans
  • ChatClientAgent β€” concrete agent class that accepts a pre-built IChatClient pipeline
  • ActivitySource β€” add custom spans around your own code (e.g. a "session" span)
  • Meter / Counter / Histogram β€” record business metrics (interactions, latency)
  • OTLP exporter β€” sends telemetry to any compatible backend (Aspire, Jaeger, Grafana, Application Insights)

Start the Aspire Dashboard (optional, one command)

docker run -d --name aspire-dashboard \
  -p 18888:18888 \
  -p 4317:18889 \
  -p 4318:18890 \
  -e DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true \
  mcr.microsoft.com/dotnet/aspire-dashboard:latest

# Dashboard UI  β†’ http://localhost:18888
# OTLP/gRPC     β†’ http://localhost:4317
# OTLP/HTTP     β†’ http://localhost:4318  ← used by this sample

NuGet Packages

dotnet add package Microsoft.Agents.AI.OpenAI --prerelease
dotnet add package Azure.AI.OpenAI --prerelease
dotnet add package Azure.Identity
dotnet add package OpenTelemetry --prerelease
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.Http --prerelease
dotnet add package OpenTelemetry.Instrumentation.Runtime --prerelease
dotnet add package Microsoft.Extensions.Logging
dotnet add package Microsoft.Extensions.DependencyInjection

Environment Variables

# PowerShell
$env:AZURE_OPENAI_ENDPOINT        = "https://<your-resource>.openai.azure.com/"
$env:AZURE_OPENAI_DEPLOYMENT_NAME = "gpt-4o-mini"
$env:OTEL_EXPORTER_OTLP_ENDPOINT  = "http://localhost:4318"  # Aspire Dashboard OTLP/HTTP port

Code Sample

using System.Diagnostics;
using System.Diagnostics.Metrics;
using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

const string SourceName  = "MyAgentApp";
const string ServiceName = "AgentDemo";

var otlpEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
                   ?? "http://localhost:4318";

// ── Tracing ────────────────────────────────────────────────────────
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName))
    .AddSource(SourceName)
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(o => o.Endpoint = new Uri(otlpEndpoint))
    .Build();

// ── Metrics ────────────────────────────────────────────────────────
using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName))
    .AddMeter(SourceName)
    .AddHttpClientInstrumentation()
    .AddRuntimeInstrumentation()
    .AddOtlpExporter(o => o.Endpoint = new Uri(otlpEndpoint))
    .Build();

// ── Logging ────────────────────────────────────────────────────────
var services = new ServiceCollection();
services.AddLogging(b => b
    .SetMinimumLevel(LogLevel.Debug)
    .AddOpenTelemetry(o =>
    {
        o.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName));
        o.AddOtlpExporter(x => x.Endpoint = new Uri(otlpEndpoint));
        o.IncludeFormattedMessage = true;
    }));

using var serviceProvider = services.BuildServiceProvider();
var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger<Program>();

// ── Custom instruments ─────────────────────────────────────────────
using var activitySource       = new ActivitySource(SourceName);
using var meter                = new Meter(SourceName);
var interactionCounter         = meter.CreateCounter<int>("agent.interactions");
var responseTimeHistogram      = meter.CreateHistogram<double>("agent.response_time_seconds");

// ── Agent setup ────────────────────────────────────────────────────
var endpoint   = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")!;
var deployment = Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT_NAME") ?? "gpt-4o-mini";

// Build an instrumented IChatClient pipeline, then wrap it in ChatClientAgent.
using var instrumentedClient = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential())
    .GetChatClient(deployment)
    .AsIChatClient()
    .AsBuilder()
    .UseFunctionInvocation()
    .UseOpenTelemetry(sourceName: SourceName, configure: cfg => cfg.EnableSensitiveData = true)
    .Build();

// ChatClientAgent accepts a pre-built IChatClient and also supports the agent-level builder.
var agent = new ChatClientAgent(
        instrumentedClient,
        name: "DemoAgent",
        instructions: "You are a concise, helpful assistant.")
    .AsBuilder()
    .UseOpenTelemetry(sourceName: SourceName, configure: cfg => cfg.EnableSensitiveData = true)
    .Build();

var session = await agent.CreateSessionAsync();
logger.LogInformation("Agent session started");

// ── Interactive loop ───────────────────────────────────────────────
// Parent span wraps the entire conversation session.
using var sessionActivity = activitySource.StartActivity("Agent Session");
sessionActivity?.SetTag("agent.name", "DemoAgent");

while (true)
{
    Console.Write("You: ");
    var input = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(input) || input == "exit") break;

    var sw = Stopwatch.StartNew();

    // Child span per interaction.
    using var interactionActivity = activitySource.StartActivity("Agent Interaction");
    interactionActivity?.SetTag("user.input", input);

    try
    {
        Console.Write("Agent: ");
        await foreach (var chunk in agent.RunStreamingAsync(input, session))
            Console.Write(chunk.Text);
        Console.WriteLine();

        sw.Stop();
        interactionCounter.Add(1, new KeyValuePair<string, object?>("status", "success"));
        responseTimeHistogram.Record(sw.Elapsed.TotalSeconds,
            new KeyValuePair<string, object?>("status", "success"));

        logger.LogInformation("Interaction completed in {Elapsed:F2}s", sw.Elapsed.TotalSeconds);
    }
    catch (Exception ex)
    {
        sw.Stop();
        interactionCounter.Add(1, new KeyValuePair<string, object?>("status", "error"));
        responseTimeHistogram.Record(sw.Elapsed.TotalSeconds,
            new KeyValuePair<string, object?>("status", "error"));
        interactionActivity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        logger.LogError(ex, "Interaction failed: {Message}", ex.Message);
        Console.WriteLine($"Error: {ex.Message}");
    }
}

What You'll See in the Aspire Dashboard

  • Traces tab β€” one parent span per session, child spans per interaction, and inner spans from Agent Framework and HTTP calls to Azure OpenAI.
  • Metrics tab β€” agent.interactions counter and agent.response_time_seconds histogram, plus HTTP client and .NET runtime metrics.
  • Structured Logs tab β€” log entries correlated to their trace IDs, with formatted messages and scoped context.

Sending Telemetry to Application Insights

Add the Azure Monitor exporter package and pass your connection string:

// dotnet add package Azure.Monitor.OpenTelemetry.Exporter

var appInsightsConnStr = Environment.GetEnvironmentVariable("APPLICATIONINSIGHTS_CONNECTION_STRING");

Sdk.CreateTracerProviderBuilder()
    // ... (existing configuration)
    .AddAzureMonitorTraceExporter(o => o.ConnectionString = appInsightsConnStr)
    .Build();

Next Steps