Managing Secrets in a Cloud World – Introduction

by Dave Bechberger

If your application does anything interesting, it has secrets. No, not that no one remembers how your code works, or that that one dev put a curse word in his checkin once. Not those kind of secrets. We’re talking about database passwords, secret API keys, per-user encryption keys, and the like. For a long time, web developers put those in configuration files or read them from files which programmers promised never to accidentally check in. Cross your heart.

These methods can have problems. If secrets are contained in files that you promise not to check in (like that SSH key or that encryption private key), they have to be transported from machine to machine, probably in an unsafe way and possibly in violation of industry standards and/or federal regulations  (PCI DSS, HIPAA, etc.). Perhaps you email another developer the passwords for a development environment, or copy a file to production during installation, hopefully from a safe place. And where is your change history? Can you survive an audit from a Very Serious Security Team?

‍

An age-old solution for many ecosystems — encrypting the data in a configuration file and checking it in — creates problems. How do you share the encryption key? Assuming you can share it safely, you have a new problem: the things that cause passwords and secrets to change aren’t the same things that cause code to be checked into your source control repository. Developers check in code when they’re done with it. It might be on a feature branch, it might be a production-ready patch, but it’s not directly tied to deployments. Secrets change for all kinds of reasons: an account was compromised, the master database password is rotated every morning, your infrastructure provider has moved to a new system. Shouting “get latest and redeploy” doesn’t scale very well.

No, these secrets need to be secured and managed in such a way that access to them is constrained to only the users and applications that need them. This requirement may be for legal reasons, a matter of corporate information/risk management policies, or a desire to maintain a clear separation of information (e.g. developers never have access to a production database).

‍

‍

Once you have more than a few developers or deployment environments, you need an efficient way to manage and secure configuration information. You’ve probably got tools on Linux and Windows, whether in servers or desktops. Add in a modern deployment environment built on containerization, SOA architecture based on micro-services and deployed to cloud infrastructure the challenges of managing, securing, and automating the deployment of this data is amplified.

‍

‍

‍

‍

‍

In this multi-part blog series we are going to be looking at several different methods for achieving this goal in a variety of ecosystems: Windows, Linux, AWS, Azure, and using several key management frameworks — both commercial and free.

• Part 1 – We will examine a simple scenario consisting of deployment of a .NET Azure Web Appvia Visual Studio publish with secrets managed using Azure Key Vault. This solution is appropriate for simple deployments, but has some shortcomings.
• Part 2 – We will examine a different take on a complex scenario by using Visual Studio Online, Octopus Deploy, and Conjur to deploy to an AWS environment. This solution makes sense for companies with diverse development teams pushing to large numbers of sensitive environments or looking for a hybrid onsite/cloud environment.
• Part 3 – We will examine a complex deployment scenario pushing a nodejs application to an Amazon Web Service (AWS) environment using AWS Code Deploy with secrets managed with Hashicorp Vault . This is a pretty typical setup for a startup that’s looking to be ‘cloud-native’ and is comfortable using open-source tools.
• Part 4 – Roundup. We’ll discuss pros & cons of the different approaches, and look to some of the devops capabilities you need once you’ve figured out you need a secrets manager. What about ssh proxying? Software firewalls? Distributed configuration?

In each case we will examine a method of retrieving secret information through access credentials that are not hard coded into the application, environment variable, or configuration file but are provided to the application at deployment time. The application will then use these credentials to retrieve the secrets at runtime.

There are a ton of secret management systems and deployment scenarios out there; we tried to choose a representative variety from the spectrum of options available. If you’re interested in a particular combination of tools — Chef? Backspace? Strongloop? — give us a buzz. We’d love to hack on it with you.