2 posts core - Log request/response headers

Recently we deployed a new application to our clients production environment. Their hosting partner uses a API gateway in front of the application that allows them to rate limit calls, load balance the application and stuff like that. We were told that the gateway would be "100% non intrusive" and "just work". Yeah right.

We noticed right away that our application simply did not work in the production environment. Everything worked perfectly in our own test environments but as soon as the application was deployed to production, it stopped working...
One thing I noticed was that NO headers at all were forwarded to our application...well that sucks.

So I told the hosting company "Hey, please forward all headers to and from our application and everything will start working right away".

Hah. They responded with "Sorry we can't allow all headers, you need to explicitly specify ALL the headers that you want to passthrough, so please send us a list of all the headers asap"


Anyhow, our application is an core application, we are using the authorization middleware and also a couple of other middlewares, my point is, I don't KNOW from the top of my head which headers our application use/needs, sure, I can make an educated guess and probably come close, but I will most certainly forget one or two.

Time to start coding. Since we are using core I created a simple middleware that logs all unique request/response headers that I can turn on/off in appsettings.

Actually, I created two middlewares, one responsible for saving the unique headers and one that could display the headers.
Here's the first one responsible for the "logging" part, it just saves the headers in two lists, one for the request headers and one for the response headers.

public class LogHeadersMiddleware  
    private readonly RequestDelegate next;
    public static readonly List<string> RequestHeaders = new List<string>();
    public static readonly List<string> ResponseHeaders = new List<string>();

    public LogHeadersMiddleware(RequestDelegate next)
    { = next;

    public async Task Invoke(HttpContext context)
        var uniqueRequestHeaders = context.Request.Headers
            .Where(x => RequestHeaders.All(r => r != x.Key))
            .Select(x => x.Key);

        var uniqueResponseHeaders = context.Response.Headers
            .Where(x => ResponseHeaders.All(r => r != x.Key))
            .Select(x => x.Key);

Since I made the lists public I can then access them in my other middleware responsible for displaying the headers.

In my startup.cs file I register the logging middleware like this, I also map the url /show-headers to the ShowHeadersMiddleware

public void Configure(  
            IApplicationBuilder app, 
            IHostingEnvironment env, 
            ILoggerFactory loggerFactory,
            IOptions<ApplicationSettings> applicationOptions) 
    // Appsetting that toggles the logheaders middleware.
        // Extension method that enables the logging middleware
        app.Map("/show-headers", ShowHeaders);

private static void ShowHeaders(IApplicationBuilder app)  
    app.Run(async context => {
        var requestHeaders = string.Join("<br/>", LogHeadersMiddleware.RequestHeaders.OrderBy(x => x));
        var responseHeaders = string.Join("<br />", LogHeadersMiddleware.ResponseHeaders.OrderBy(x => x));
        var output = $"<h2>Unique request headers</h2>{requestHeaders} <h2>Unique response headers</h2> {responseHeaders}";
        context.Response.ContentType = "text/html";
        await context.Response.WriteAsync(output);

I then enabled the logging in our test environment and had our QA-people browse around in one of our other applications that uses the API so I could get a list of all the unique headers. I then sent the list to the hosting provider and sure started working right away. core appsettings transformation with environment variables on unix

TL;DR Replace dots with underscore and colons with double underscores.

Consider the following IOptions<MetroSettings> implementation

public class MetroSettings  
    public RealTimeDepartures RealTimeDepartures { get; set; }

public class RealTimeDepartures  
    public string ApiKey { get; set; }
    public string BaseUrl { get; set; }

In my Startup I have the following code:

var builder = new ConfigurationBuilder()  
        .AddJsonFile("appsettings.json", true, true)
Configuration = builder.Build();  

And my appsettings.json looks like this:

    "Metro": {
        "RealTimeDepartures": {
            "BaseUrl": ""

I don't like having API keys and other sensitive data in config files, I prefer environment variables.

In ASP.Net core it's pretty straight forward to transform the appsettings file with environment variables.

On windows you just run this in your terminal:

set MyNamespace.MyPrefix.Metro:RealTimeDepartures:ApiKey=MySecretApiKeyHere  

and your good to go, at least in a windows environment.

I, however, wanted to run my application on a Ubuntu server and I ran into some problems.

When I ran the following command

export MyNamespace.MyPrefix.Metro:RealTimeDepartures:ApiKey=TESTING  

I got the following error:

export: MyNamespace.Metro:RealTimeDepartures:ApiKey=TESTING': not a valid identifier  

Apparently you are not allowed to have dots or colons in the environment variable name.

So, what to do?

Luckily I found this issue on github (open source ftw!) and I was able to make it work by doing the following:

  • Change my prefix from MyNamespace.MyPrefix. to MyNamespace_MyPrefix_ in the Startup file.
  • Replace the colons with double underscores instead




  • Setting the correct environment variables in windows and ubuntu.


set MyNamespace_MyPrefix_Metro__RealTimeDepartures__ApiKey=MySecretApiKeyHere  


export MyNamespace_MyPrefix_Metro__RealTimeDepartures__ApiKey=MySecretApiKeyHere  

(I actually ended up adding the variable directly in the /etc/environment file)