How to log request & response body to Application Insights

How to log request & response body to Application Insights

Introduction

Although Application Insights does log HTTP requests out-of-the-box, it doesn't provide a turn-key ready solution that also logs the content of a request/response body.

This however might be required for some debugging scenarios and luckily we can extend the telemetry data with our own metrics and logs.

In this article I am showing how to log the body of an HTTP POST or PUT request and also the response to Azure's Application Insights by leveraging custom middleware components.

A word of caution

Writing the content of an HTTP body to Application Insight might reveal sensitive user information that otherwise would be hidden and protected in transfer via TLS. So use this with care and only during debugging or developing!

With that being out of the way lets see some code!

Writing a custom middleware

When using ASP.NET Core an HTTP request travels through a chain of request delegates that allows us to tap into and log the HTTP body content away.

ASP.NET Core Request Pipeline made up of several request delegates - Source: docs.microsoft.com

There are two ways to write a custom middleware. One is by following a convention-based approach and the other one implementing IMiddleware that gives us strong typing and per request activation. I prefer the later, although it lacks the option to pass arguments to it.

With the great documentation that Microsoft provides, it all seems straight forward. However there is one banana skin 🍌to get around.

When reading from HttpContext.Request.Body via StreamReader() we need to keep a copy of the data and reset the stream position to the beginning so that other request delegates (middleware components) are also able to read from it. If you don't reset the position you will end up with a HTTP 400 or even HTTP 500 response.

Also when dealing with the HTTP response HttpContext.Response.Body we need to swap out the stream with one that is buffered and that supports seeking. We are going to use MemoryStream for this purpose.

So without further ceremony here is the code that I put together with the help of this great article here.

public class RequestBodyLoggingMiddleware : IMiddleware
{
    public async Task InvokeAsync(HttpContext context, RequestDelegate next)
    {
        var method = context.Request.Method;

        // Ensure the request body can be read multiple times
        context.Request.EnableBuffering();

        // Only if we are dealing with POST or PUT, GET and others shouldn't have a body
        if (context.Request.Body.CanRead && (method == HttpMethods.Post || method == HttpMethods.Put))
        {
            // Leave stream open so next middleware can read it
            using var reader = new StreamReader(
                context.Request.Body,
                Encoding.UTF8,
                detectEncodingFromByteOrderMarks: false,
                bufferSize: 512, leaveOpen: true);

            var requestBody = await reader.ReadToEndAsync();

            // Reset stream position, so next middleware can read it
            context.Request.Body.Position = 0;

            // Write request body to App Insights
            var requestTelemetry = context.Features.Get<RequestTelemetry>();                              
            requestTelemetry?.Properties.Add("RequestBody", requestBody);
        }

        // Call next middleware in the pipeline
        await next(context);
    }
}
RequestBodyLoggingMiddleware.cs
public class ResponseBodyLoggingMiddleware : IMiddleware
{
    public async Task InvokeAsync(HttpContext context, RequestDelegate next)
    {
        var originalBodyStream = context.Response.Body;

        try
        {
            // Swap out stream with one that is buffered and suports seeking
            using var memoryStream = new MemoryStream();
            context.Response.Body = memoryStream;

            // hand over to the next middleware and wait for the call to return
            await next(context);

            // Read response body from memory stream
            memoryStream.Position = 0;
            var reader = new StreamReader(memoryStream);
            var responseBody = await reader.ReadToEndAsync();

            // Copy body back to so its available to the user agent
            memoryStream.Position = 0;
            await memoryStream.CopyToAsync(originalBodyStream);

            // Write response body to App Insights
            var requestTelemetry = context.Features.Get<RequestTelemetry>();
            requestTelemetry?.Properties.Add("ResponseBody", responseBody);
        }
        finally
        {
            context.Response.Body = originalBodyStream;
        }
    }
}
ResponseBodyLoggingMiddleware.cs

For enhanced comfort I suggest to create two extension methods that allow for clean integration in Startup.cs with a call to app.UseRequestBodyLogging() or app.UseResponseBodyLogging() accordingly.

public static class ApplicationInsightExtensions
{
    public static IApplicationBuilder UseRequestBodyLogging(this IApplicationBuilder builder)
    {
        return builder.UseMiddleware<RequestBodyLoggingMiddleware>();
    }

    public static IApplicationBuilder UseResponseBodyLogging(this IApplicationBuilder builder)
    {
        return builder.UseMiddleware<ResponseBodyLoggingMiddleware>();
    }            
}
ApplicationInsightExtension.cs

Don't forget to register the new types with the DI container.

public void ConfigureServices(IServiceCollection services)
{
    // ...
    services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
            
    services.AddTransient<RequestBodyLoggingMiddleware>();
    services.AddTransient<ResponseBodyLoggingMiddleware>();
}
Startup.cs

To ensure logging does not get enabled in production environments only enable the custom middleware components inside the condition that checks if we are running in a development environment.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        
        // Enable our custom middleware
        app.UseRequestBodyLogging();
        app.UseResponseBodyLogging();
    }
    
    // ...
}
Startup.cs

And that's it! 😎 Lets run through a quick test and see the results!

Testing

One of the great things about Application Insight is that the results show up really quick in the Azure Portal. So fire-up your ASP.NET Core project and send a POST request to a corresponding endpoint.  Here is a sample controller and simple DTO for your convenience.

[Route("api/[controller]")]
[ApiController]
public class UserController : ControllerBase
{

    private readonly ILogger logger;

    public UserController(ILoggerFactory loggerFactory)
    {
        this.logger = loggerFactory.CreateLogger<UserController>();
    }

    [HttpPost]
    public ActionResult Index([FromBody]User user)
    {
        return Ok(user);
    }
}
UserController.cs
public class User
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
}
UserDto.cs

Here we are sending the HTTP Post Request with the help of cURL.

curl --header "Content-Type: application/json" --request POST --data @body.json https://localhost:44331/api/user  
Posting JSON to REST API using cURL

End-to-end transaction details

Goto your Application Insights resource and navigate to Investigate > Transaction Search. Optionally set Event Types = Request and click on one of the logged items.

You will end up with a view like this showing our custom properties which we defined as RequestBody and ResponseBody.

Custom Properties showing up in Application Insights

Log Queries

Alternatively you can use Log Analytics in Azure Monitor which gives you more control over your attributes and better adapts to specific needs.

requests 
| where name startswith "POST"
| order by timestamp
| project timestamp, customDimensions.RequestBody
Querying data provided by Application Insights

Conclusion

For sure many roads lead to Rome. Others might want to implement their own version of ITelemetryInitializer or use an ActionFilterAttribute from Microsoft.AspNetCore.Mvc.Filters instead. However I found it convenient that HttpContext is already available when implementing IMiddleware and doesn't need to be injected.  

As we have seen, adding custom properties to our Application Insights telemetry data is no big deal. Also extending an ASP.NET Core pipeline can be fun, however it needs some care when dealing with the streams!

I hope you enjoyed this short article, in case you have any questions or comments please leave them in the comment section or drop me a mail.

Further reading

Azure Application Insights for ASP.NET Core applications - Azure Monitor
Monitor ASP.NET Core web applications for availability, performance, and usage.
Filtering and preprocessing in the Application Insights SDK - Azure Monitor
Write telemetry processors and telemetry initializers for the SDK to filter or add properties to the data before the telemetry is sent to the Application Insights portal.
Write custom ASP.NET Core middleware
Learn how to write custom ASP.NET Core middleware.
Re-reading ASP.Net Core request bodies with EnableBuffering() | ASP.NET Blog
In some scenarios there’s a need to read the request body multiple times. Some examples include Logging the raw requests to replay in load test environment Middleware that read the request body multiple times to process it Usually Request.Body does not support rewinding,
Show Comments