Reading Environment Variables, AppSettings, LocalSettings, and User Secrets Seamlessly Across Environments in ASP.NET

One of the biggest challenges we face in modern cloud solutions is making it so sensitive configuration data can be read seamlessly across all environments from the local dev environment all the way to production. This isn't really a new issue, but one that has multiple ways to accomplish some easier than others.

For a sample use case, we might have a database connection string with sensitive information that needs to work seamlessly in code for local, Dev, QA, Stage, Prod, etc. without a lot of special hooks or handling in code. One way to handle this in the cloud regardless of deployment resource or setup is to expose a single key name with unique value per-environment. This can be configured via scripting in deployment pipelines per environment to take care of the DevOps side of this equation. However in code we want this single-read of a key to work in all environments. This is where in .NET the Configuration provider offered in Microsoft.Extensions.Configuration works so well.

Let's say the following environment variable has been configured across Dev, QA, Stage, and Prod environments with respective values already:

CosmosDbConnectionString | AccountEndpoint=https://cosmos-acct-{env}-eus-001.documents.azure.com:443/;AccountKey=abc123...

For local development we have a couple options as well to configure this value:

1. Create your own environment variable on your local OS with the identical key name and whatever value you choose
2. Add the identical key and again whatever value you require as a 'User Secret' within Visual Studio or VS Code (VS Code requires an extension download to work with secrets)

The above 2 methods are preferred as they do not add to files that get committed to source control. While you can add the key/value pairs to appSettings.json or launchSettings.json (as env variables), these are not advised as it breaks the requirement that we should not be committing secrets to source control. Note - you could also write a PowerShell script to be added to the solution that developers could run initially to pull environment variable key names from the cloud modifying with a local value for an added bonus to be even more automated.

So moving back to our .NET code, we would like to have a single line that uses 'CosmosDbConnectionString' that will work for all environments.

To get started, in current versions of ASP.NET, this single line of code in program.cs sets us up out of the box:

The absolutely fantastic feature of this black-box is that it will automagically handle the hierarchy of running through multiple different configuration sources including environmental variables! This single line of code (after injecting the IConfiguration configuration service) is all that's needed:

Here is the official documentation on this from Microsoft:

Using the default configuration, the EnvironmentVariablesConfigurationProvider loads configuration from environment variable key-value pairs after reading appsettings.json, appsettings.Environment.json, and Secret manager. Therefore, key values read from the environment override values read from appsettings.json, appsettings.Environment.json, and Secret manager.

This is great, because now we can add our connection string as a local secret (or local environment variable) and have it read at runtime during debugging as required, but once deployed the configured environment variables per-environment will supersede anything from appSettings, secrets, etc. and continue to work with the same code.

This is also the case for non-sensitive data stored in appSettings.json locally vs configuration in a deployed environment like Azure. Any configuration settings in Azure per-environment using the same key name will supersede locally configured values. This use case is more well-known but worth mentioning as it's an anti-pattern to have different environment configs managed at the source control level; it should be managed in deployed environments via DevOps.

A last point to review is naming conventions when creating environment variables so they work platform agnostic. Per the same MSFT docs linked above:

The : delimiter doesn't work with environment variable hierarchical keys on all platforms. For example, the : delimiter is not supported by Bash. The double underscore (__), which is supported on all platforms, automatically replaces any : delimiters in environment variables.

Therefore if you need a hierarchical key consider using the following naming structure which will work on Windows and Linux:

CosmosDb__ConnectionString

This can be read regardless of deployed environment in code with the following line:

Thanks to the .NET configuration provider having all of this included functionality, we can work across 1..n environments using a multitude of configuration options, yet being able to garner that data at runtime with a single line of code!