How to secure ASP.NET Core apps with Azure Key Vault and Sentry

Joao Grassi — a .NET developer, front-end hobbyist, and friend of Sentry — likes .NET very much. So do we. With the help of one of Sentry’s top 10 SDKs, .NET developer teams process roughly half a billion .NET events every month. In this post, we strive for app security with Azure Key Vault and Sentry.

By now, it's not big news that ASP.NET Core is the future of web development with .NET. Of all the great additions that ASP.NET Core has introduced, one of my favorites is everything around Configuration.

Configuration is easy and practical — configuring our apps in ASP.NET Core feels natural. I could spend this entire blog post just talking about why I think it's so great, but let's leave it for maybe another post. :)

One thing missing from Configuration is a way to "secure" the settings. If you use Azure App Service, the settings and connection strings are stored encrypted in Azure, and they are decrypted only before being injected into your app's process memory when the app starts, as explained here.

This is all great, but one could argue that the settings values are still visible in plain text in the Configurations page on the App Service, and anyone who can access your App Service could potentially see the values. As a practical person, I argue back that if someone you don't trust has access to your App Service configurations page, then you probably have more serious issues. 🤷

Still, let's say you trust no one and you want to have more control over the settings which contain sensitive values in your App Service / Function app settings. Imagine settings like API Keys, external services secrets and so on. How can we achieve a more fine grained control over our settings? Enter Azure Key Vault.

Azure Key Vault: what is it?

The official definition by Microsoft:

Azure Key Vault is a tool for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. A vault is logical group of secrets.

In essence, we can think of Azure Key vault as, well, a vault! You put your secret things in, and the vault keeps them secure. Of course, Azure Key Vault is more complex than that and offers many features, but for the scope of this post, I'd say that's enough to understand. Feel free to dig more into the details in case you are interested: What is Azure Key Vault?

Using Azure Key Vault with your ASP.NET Core apps

If you want to use Azure Key Vault as one of your app's configuration providers you would need to do some work, like add specific NuGet packages, get the URL of the Vault, create your clientId and secret (more on resolve this chicken-or-egg issue with Azure system-assigned identity later), connect to the vault, read the settings... you get the idea.

The point is: the process isn’t trivial. I had to go through these steps recently, and I spent more time on it than I'm proud to share. Several Microsoft docs and blog posts later, I managed to change my app to make it work. BUT, not everyone out there can change their apps that easily. We need to consider the number of people running critical components in production. Requiring such code changes just for the sake of consuming settings from the Key Vault seems a bit too much to ask.

As you maybe guessed at this point, there's an easier way (if you are using App Service/Azure Functions at least): Azure Key Vault references. 🔐

Azure Key Vault references

With Key Vault references you can reference your secrets from the Key Vault directly in your normal app settings, be it either an app hosted on App Service or an Azure Function. So, your app can just work as it normally would. You don't need to change code to be able to use settings that are stored in the Key Vault!

I should point out that at the time of writing this blog post, this feature is in public preview and only works for App Service and Azure Functions. I suspect that the syntax will not change that much, but we all know that preview is preview and not suitable to use in production.

Now that we know the theory behind Key Vault references and understand the issue it tries to solve, let's see how we can actually use this thing in combination with an ASP.NET Core App Service.

Steps of our demo

1. Creating the Key Vault

2. Adding a connection string to our Key Vault

3. Get the "reference" of our secret

4. Our demo app

5. Using the Key Vault reference as the source of our connection string

6. Testing our app

7. Creating a system-assigned identity for our App Service

8. Configure the access policy in our Key Vault for our App Service (GET only)

9. Testing our app: part II

10. Conclusion

Ten steps seem like a lot, but it's not that complicated. I will try to guide you and explain each step in more detail, so you are not just blindly following me.

Creating the Key Vault

On the Azure Portal, go to Resource groups > _your resource group_ > Add new resource > Key Vault > Create.

1. Fill in the basic details: Name, Region and pricing tier.

2. You can skip the other tabs for now. We will come back to Access policy later.

3. Review the things and finally click on Create.

Adding a connection string to our Key Vault

Navigate to your Key Vault when the deployment finishes. Once inside the Key Vault page, look at the left menu and click: Secrets > Generate/Import. Enter the connection string to your database. In my case, I'm entering a dummy connection string to a SQL Server Database. The page looks like this:

Get the "reference" of our secret

After the secret is created, you are redirected to the list of secrets. Click on the secret you just created. The next page shows the versions of that secret. A single secret can have multiple versions inside the Key Vault. Since we just created ours, we only have one version. Proceed by clicking on it.

Once inside the details of that version of the secret, you'll see the Secret Identifier (it's just a URL). That's the identifier we will be using later. Azure nicely offers to copy it to the clipboard for us. So, copy it and store it somewhere.

Our demo app

We will be using an ASP.NET Core API to demonstrate consuming the secret (our connection string) from Azure Key Vault. To avoid making this post too long, I'll not show the code here. Basically, it's an API using EF Core + SQL Server to read a simple Movies table.

I have made the code available on my GitHub. You can check it out here: appservice-keyvaultreference-sentry.

Configuring Sentry error tracking

While I was experimenting with Key Vault references, I've encountered a rather "odd" issue with it (more on it later). Luckily, I was using Sentry on my app, so finding the actual issue was a breeze. You can check their samples on GitHub to see how you can add it to your ASP.NET Core apps.

Deploying the API to Azure

As mentioned before, to use Key Vault references your app must be running on App Service or be an Azure Function.
To be able to continue with this tutorial you'll need to deploy your ASP.NET Core app to Azure. You can use the one I provided above or create your own. I'll also skip the deploy part for brevity. You can find many tutorials online for this, or just publish from Visual Studio.

Tip: you can use the F1 pricing tier for your service plan to host it for free. 😉

Quick recap — At this point, you need to have on Azure:

1. A Key Vault with our connection string

2. A SQL Database set up

3. Your ASP.NET Core App deployed to App Service

Let's move on and use the Key Vault reference in our app.

Using the Key Vault reference as the source of our connection string

Remember I told you to copy the secret identifier when we first added our connection string to our Key Vault? We are going to use it now. If you lost it or haven't copied it yet, don't worry. Just go back to your Key Vault > Secrets. Then follow the instructions on the step: Get the "reference" of our secret.

The Key Vault references feature has a special syntax which will contain the secret identifier inside. This is how the syntax looks like:

@Microsoft.KeyVault(SecretUri=<YOUR_SECRET_IDENTIFIER_HERE>)

What you need to do is: Replace the YOUR_SECRET_IDENTIFIER_HERE in the code above with the Secret URI/Identifier we copied from our Key Vault. With our secret reference built, we can head to our App Service Settings.

Just edit (or add) a Connection String to your App Service app using the Key Vault reference as the value. Make sure you save it!

Testing our app

Let's see if our api is now working. If you used the sample I provided, you can navigate to: api/movies to see a list of movies retrieved from our SQL Server database.

Investigating what went wrong

The app is not working and we have no idea why. This is where error monitoring tools like Sentry shine. Since I used it on my app, I can just switch to my Sentry's dashboard and see what's going on. In fact, I already got an email. Nice!

Interesting. It seems it tried to use our Key Vault reference value as it is for the connection string and, of course, didn't work. This is the "odd" behavior I encountered. Fetching the value from the Key Vault failed for some reason which we'll see in a bit, but it failed silently.

This example might not be the best, since without the connection string nothing would initially work, but imagined you used this in some setting of your app which is not used frequently. To make matters worse, if you don't have any sort of tool to help you find out the bugs in your app, like Sentry, you are out of luck.

The behavior I would expect in this case is either fail or return something else, but definitely not the Key Vault reference in plain text. Of course, there are policies on the Key Vault, but you could, in the end, leak your stuff. Not so cool.

The actual issue: permissions

What went wrong is: We created the Key Vault, added a secret (connection string) to it and used it as a reference in our App Service app. But: we never granted Key Vault access for our App Service!

The official docs for Azure Key Vault References describe this problem under the Troubleshooting Key Vault References section. Once I saw the error on Sentry I immediately knew what the issue was.

The most important part on the docs regarding this particular problem is:

If a reference is not resolved properly, the reference value will be used instead. This means that for application settings, an environment variable would be created whose value has the @Microsoft.KeyVault(...) syntax. This may cause the application to throw errors, as it was expecting a secret of a certain structure.

Now that we know what caused the issue, let's fix it!

Creating a system-assigned identity for our App Service

To be able to consume the secrets from the Key Vault, our App Service needs to have access to it. In Azure, you can configure one resource to access another by creating what's called a managed identity. Once your resource has a managed identity, you can modify another resource and allow access to it.

Maybe my explanation sucks, so here are the official words:

A managed identity from Azure Active Directory allows your app to easily access other AAD-protected resources such as Azure Key Vault. The identity is managed by the Azure platform and does not require you to provision or rotate any secrets.

This also solves the Chicken or the egg issue where you have to have the Key Vault user and password in your settings before you're able to connect to it.

Let's add a managed identity for our App Service:

On the "Identity" page, make sure the tab System assigned is selected. Then, toggle the Status to On and press Save. That's it. Wait until Azure finishes configuring everything to move to the next step.

Configuring the access policy on the Key Vault for our App Service

One of the nicest things about Azure Key Vault revolves around its access policy feature. The granularity is huge so we can achieve a lot of things by creating several policies with different access levels. In our case, we're going to keep it simple and allow only GET access to secrets to our App Service.

The previous step was important because now we can search for our managed identity and configure its access to the Vault. To do this, go back now to our Key Vault page, and on the left menu click on Access policies.

Once in the Access policies page, click on Add access policy. Select Get for the Secret permissions field. Next, click on the Select principal field. A side-bar will appear on the right side of the screen. There, you need to search for our managed identity, which matches the name of your App Service. Click Select on the window. At the end it should look like this:

Click on Add then on Save to save the changes.

Now our App Service has GET access to Secrets in the Vault.

Testing our app: part II

Now that we have everything in place, let's test it again. Head over to api/movies, or if you have used a different app go to where you access the database.

Now it works! (If it still doesn't, try restarting your App Service.)

Conclusion

It might seem like a lot, but let's take a step back for a second. Imagine that your app is already in production and, with just a few clicks, we were able to reference a setting value which is securely stored and managed by Azure Key Vault, without any code change. Awesome, isn't it?

Limitations:

Here are some limitations that you may or may not care:

1. As of it now, you have to use the whole secret URL, with the version at the end. This isn't great, because if you change the secret, the AppService/Function will not use the new value. It seems they are working on to support not having to have the version, but there's no ETA.

2. The Key Vault references is still in preview, so use it with care. According to this comment the GA might be coming soon. You can also vote and subscribe to this so you are notified when it's available for production.

Good to know

The access to the key vault is priced by transactions. For secrets, the current pricing page shows secrets operations: €0.026/10,000 transactions.

Well, that looks pretty cheap. But still, I wouldn't like to have a surprise at some point after starting using it. The first thing I was curious was:

While using Key Vault references, does it count as a transaction for every time I read the setting which holds the reference value? Turns out someone already asked it over on GitHub and the response was:

The settings/values are pulled from KeyVault once the app service process starts [...]

Here is the link for the whole discussion: KeyVault references - are returned values cached in the App Service.

Automation

I demonstrated here in this post how to configure Key Vault references in your AppService or Function. Along with the post, I created several resources and changed configurations, all via the Azure Portal. It's okay to do it like that a few times, but as soon as you need to scale your services you have to start thinking about how to automate the deployment of it. Most of the things I showed here, you can do via the Azure CLI. In case you are interested, the links below have more details on this.

References/links

It's not possible to cover all the things in a single blog post. If I got you curious/interested, you can go into more details using the links below.

Announcement of Key Vault References

Key Vault references docs

Managed identities using Azure CLI

CLI samples for Azure App Service

Key Vault docs

And that's about it. Thanks for reading; I hope it was somewhat useful.

Now that you’re done, you might as well debug .NET apps and prevent crashes across your entire stack with Sentry’s .NET SDK.