I’m sure I’m not the only one that has “temporarily” stored passwords, application keys or other development secrets in appsettings.json simply because it was easier or faster than doing it the right way (although if I am, this first sentence will surely come back to haunt me).

I get it. It happens! This blog post actually comes from my efforts to update a project I’m working on with some friends so that we’ve got a more consistent approach to keeping secrets… secret.

So how are we going to go about this? We’ll be using Secret Manager, a fantastically helpful tool built into the configuration API in ASP.NET Core that will let us keep the secrets out of appsettings while not require a change to how we access the settings in code. Sound good?

Throughout the rest of this post I’ll be referring to appsettings.json, appsettings.Development.json and any other configured appsettings files collectively as “appsettings” or “the appsettings files” for simplicity.

Overview

The steps we’ll be taking look as follows:

0. Configure the project in question to use Secret Manager
1. Create a secrets.json file containing only the secrets you’re migrating from appsettings
2. Import that file into Secret Manager
3. Remove the secrets from appsettings
4. Test the migration
5. Commit your changes

Secret Manager

Before we get into the migration itself, it’s worth quickly looking at what Secret Manager is and a bit about how it works.

What is it?

Secret Manager is a tool built into the configuration API in ASP.NET Core. It is designed for exactly the purposes we need it for; storing development secrets in a way that makes them easily accessible in code while keeping them clear of settings files that are committed as part of the project.

How Does it Work?

By default the configuration API will attempt to load values from the configured providers in the following order (with values loaded later overriding previously loaded values):

1. appsettings.json file
2. appsettings.{env.EnvironmentName}.json file
3. The local User Secrets File #Only in local development environment
4. Environment Variables
5. Command Line Arguments

What this means is that if you store a value in Secret Manager (the “User Secrets File”), that value will be loaded after (and take precedence over) whatever is in your appsettings files.

The magic of this approach is that (after initial configuration) it’s essentially invisible; the configuration API does all the heavy lifting behind the scenes to retrieve the values from Secret Manager instead of the JSON configuration files without requiring any change to how you access them in code!

Steps

0: Setup

If you’re using CreateDefaultBuilder to create a Host you should already be set up and ready to go, but if you’re using the configuration API directly you’ll need to make sure you register Secret Manager as a configuration provider. The full instructions for setting up your project to support Secret Manager are available in the documentation.

You’ll also need to initialise Secret Manager for any of the projects you’re migrating by doing the following:

1. Open a command prompt or terminal (depending on your operating system)
2. Change directory (cd) to the project you’re setting up (wherever your .csproj is stored)
3. Run dotnet user-secrets init

If everything worked correctly, you should see a message saying “Set UserSecretsId to <GUID> for MSBuild project <ProjectPath>”. If not, you’ll need to make sure you’ve gone through the whole process for setting up your project to support Secret Manager.

You can commit the resultant change to the .csproj that sets the UserSecretsId to source control since the value is only used in the path for the JSON file Secret Manager uses in the background.

1: Preparing the Secrets File

Rather than importing values one-by-one, we’ll be putting together a file with all of the secrets we want to migrate and then importing that file into the Secret Manager tool, so creating this file is our initial goal.

It’s important to remember that secrets in appsettings.Development.json take precedence over those in appsettings.json meaning that if the same secret is present in both files, the value found in appsettings.Development.json is the one that will be used. As an example, let’s say we’ve got the following two files.

appsettings.json

{
  "AllowedHosts": "*",
  "SuperSecretConfigItems": {
    "Password": "ISureHopeThisStaysSecret",
    "AppKey": "12345"
  },
  "NotSecret": {
    "ItemsLoadedPerPage": 20
  }
}

appsettings.Development.json

{
  "SuperSecretConfigItems": {
    "ConnectionString": "ConnectPlease",
    "AppKey": "99999"
  },
  "NotSecret": {
    "ItemsLoadedPerPage": 1000
  }
}

If you were to access the values in the SuperSecretConfigItems element through the configuration API, you’d get the following:

• Password: “ISureHopeThisStaysSecret” (as it’s only present in appsettings.json)
• ConnectionString: “ConnectPlease” (as it’s only present in appsettings.Development.json)
• AppKey: 99999 (as it’s present in both but the appsettings.Development.json value takes precedence)

The fastest way to produce your own secrets file then is as follows:

1. Make a copy of appsettings.json for the project you’re working on (for simplicity we’ll name the new file “secrets.json”, but if you’re migrating a few projects it might be worth giving it a more specific name)
2. If you are using appsettings.Development.json (or any other JSON files), copy any secrets into your newly created settings file (secrets.json), overwriting duplicate elements where applicable
3. Remove any non-secret values (since otherwise they’ll end up in Secret Manager)

Using the above example files, we could expect our secrets.json file to look as follows:

secrets.json

{
  "SuperSecretConfigItems": {
    "Password": "ISureHopeThisStaysSecret",
    "ConnectionString": "ConnectPlease",
    "AppKey": "99999"
  }
}

We’ve removed non-secret values such as AllowedHosts and the creatively named NotSecret element while making sure we get the AppKey value from appsettings.Development.json.

Now that we’ve put our secrets file together, we’re finally ready to move onto the next step!

2: Importing into Secret Manager

Importing secrets.json is thankfully a pretty simple process:

1. Open a command prompt or terminal (depending on your operating system)
2. Change directory (cd) to the project you’re setting up (wherever your .csproj is stored)
3. Import the file by running one of the following commands (depending on operating system)

Windows:

type .\secrets.json | dotnet user-secrets set

Linux/macOS

cat ./secrets.json | dotnet user-secrets set

If the secrets were imported successfully, you’ll see the message “Successfully saved x secrets to the secret store”.

3: Removing the Secrets from appsettings

Our goal with this post is to remove secrets from appsettings and this is the step where we actually get to do that!

Although you can remove the secrets entirely I prefer to replace them with empty strings, mainly to avoid breaking your required configuration into multiple places. With empty strings (or maybe a placeholder if you prefer) you can see at a glance how the configuration is laid out for the whole project without having to reconstruct it from various sources. Thanks to the previously discussed precedence of configuration sources the values in appsettings will be overwritten by those in Secret Manager, meaning that the actual values you leave behind don’t matter!

Let’s take a look at how our appsettings files look now after that change:

appsettings.json

{
  "AllowedHosts": "*",
  "SuperSecretConfigItems": {
    "Password": "",
    "AppKey": ""
  },
  "NotSecret": {
    "ItemsLoadedPerPage": 20
  }
}

appsettings.Development.json

{
  "SuperSecretConfigItems": {
    "ConnectionString": "",
    "AppKey": ""
  },
  "NotSecret": {
    "ItemsLoadedPerPage": 1000
  }
}

You may also want to remove duplicate entries at this point (such as appkey), preferring to keep the blank entry in appsettings.json over appsettings.Development.json.

4: Testing your Changes

An important step in making any sweeping change like this is to make sure it hasn’t broken anything! How you go about this will depend on your project and what you use the secrets for, but you’ll want to make sure that values are still being loaded and that they are what you expect.

You’ll also have to consider how these changes may impact your testing/staging/production environment(s). If at all possible you’ll also want to test your deployment process to make sure that your environments aren’t relying on the values you’ve removed from appsettings.json!

5: Committing your Changes

If you’ve committed your secrets to source control, this is the time to remove them! My advice is to do this in a separate branch to give other members of your team a chance to get their systems configured correctly before it gets merged into your develop or main branches since it will break the build for anyone who hasn’t moved their secrets to Secret Manager.

Once you’ve managed to get most/all of the potentially impacted developers to set up their environment and all seems to be working correctly you can merge the branch into develop, (relatively) safe in the knowledge that it won’t cause issues for everyone else.

Next Steps

If you’ve made it this far, congratulations! You’ve finished the main process of migrating your development secrets out of your appsettings files and into a more suitable location.

There are now a few more things that you might want to do.

Refresh Secrets

If any of your secrets have made their way into appsettings (especially if they’ve been committed to source control) I highly recommend refreshing/re-issuing them. Consider them compromised! Although you may think they’re safe to keep using, other people with repository access now (or anyone who gets access later) can see and use them.

Once you’ve got the new values, the process to update the settings you migrated is thankfully very simple: modify your secrets.json file and run the import again to overwrite them!

Documenting Setup/Setting up your Team

Now that you’ve migrated your development secrets out of appsettings.json you’ll need to modify any documentation you have for setting up a development environment to include the configuration of Secret Manager.

The process is as simple as taking your secrets.json file and blanking out all of the secret values to turn it into a template. For example, using the secrets.json file from earlier we could produce the following:

{
  "SuperSecretConfigItems": {
    "Password": "",
    "ConnectionString": "",
    "AppKey": ""
  }
}

You can safely commit this file to source control alongside your appsettings files and include steps to fill out and import it as part of initial setup. Committing the file and keeping it up to date is also a very useful way of keeping the setup process as simple as possible to avoid anyone dropping values back into appsettings and undoing your hard work!

Conclusion

You’re done! The above process can take time and attention to do properly, but it’s an important step in ensuring that your development secrets are kept safe.