Managing Secrets in a Cloud World – Part 2

(This is part 2 in series on Managing Secrets in a Cloud World.  You can read the introduction here and part 1 is here)

In this post we are going to explore a slightly more complicated yet common scenario involving continuous deployment of a .NET web application.  We chose a common set of tooling used in Windows based deployments consisting of Visual Studio Online (VSO) for Source Code Control and Build, Octopus Deploy for Automated Deployment, Amazon EC2 for a hosting environment.  For secrets management we are going to be using Conjur, which is more commonly associated with Linux/Unix based environments but, as you will see, can be easily integrated into both Windows and Hybrid (Linux/Unix) environments using their provided REST API. Palladium Consulting is a Conjur partner. All code for this post can be found on GitHub here.

Our goal was not to store any sort of critical information (usernames/passwords) in source code control (remember “Keys are Code, Values are Data”) or make them available in any sort of shared configuration.  To accomplish this we will be creating unique credentials for each deployment and automatically adding them to the web.config during a deployment.  The steps we are going to use are:

1. Check code into Visual Studio Online

1. Use Visual Studio Online to build and publish release package to Octopus Deploy
2. Octopus Deploy will automatically create and start a deployment. (This could also be a tool like Jenkins or a TFS deployment step.)
3. During the deployment process a Powershell script will automatically create the unique user in Conjur and provide it the appropriate access to the secret
4. The unique user credentials will be added to web applications configuration by Octopus Deploy during the deploy to the web server
5. The web application is deployed to web server and properly configured
6. The web application starts up and uses uses the provided credentials to retrieve the appropriate secret

As you can see this is a lot more complicated than the “click to deploy” demo in our last example. In particular, there are several steps taken to securely generate a one-time “secret zero” for the deployed application. If you’re a small enough company where “click to deploy” works, then enjoy while it lasts. But if you manage multiple development teams and deployment environments — analytics, storefront, API, data warehouse, production, testing, staging, research — then you quickly determine you can’t leave secrets management to each team. A secure solution starts to need to look like more than just a bucket to put secrets in. Conjur is a commercial take on this problem that we’ve been impressed with.

Setting it all up

Visual Studio Online

Configuring Visual Studio Online required little more than more than signing up for an account and adding a simple build for the project (instructions here) that runs each time a check-in to the project occurs.

Amazon EC2 Servers

For this scenario we provisioned two Windows 2012 machine EC2 machines.  The first one was configured for use as a web server of our final deployed solution.  The second one was configured to run the Octopus Deploy server.  Additionally a SQL Server Express database was provisioned using Amazon RDS for Octopus Deploy to store its configuration information.

Note: For expediency here we set up these machines by hand. In a real cloud deployment scenario, we’d fully expect to use a tool like Chef , Puppet or Terraform. Integration of secrets management tools like Conjur with tools like this will be the subject of a future blog series.

Octopus Deploy

Octopus Deploy is an automation deployment tool which is familiar to many .NET houses. It can be installed either onsite or in the cloud, making it appropriate for a both hybrid and pure cloud solutions. It’s a good choice for .NET applications; other tools might be appropriate for other ecosystems. Like many deployment tools — AWS CodeDeploy and Chef among them — you must also deploy a lightweight agent on each potential deployment target. Octopus calls this agent a “tentacle”. Cute.

For our scenario we installed a tentacle on both the web server (for the deployment of the web application) and the deployment server (for interacting with Conjur).  The installation of the Octopus Deploy Server was done on the deployment machine by following the instructions found here.   After the installation of the server was complete we installed a Tentacle on both the web and deployment server so that each would be available to be used by the deployment process.  Once we had completed the server and tentacle installations we configured a basic deployment project consisting of a new Environment (instructions here), Lifecycle (instructions here) and added Deployment targets for both our deployment server tentacle and web server tentacle.

Conjur

We used an AWS EC2 Machine that was kindly provided to us by the folks at Conjur; it was configured following the instructions from the Conjur documentation (here).  Once we had it up and running we needed to setup a user and a group for our application (tutorial found here), added our secret (which is called a Variable in Conjur) using the tutorial found here and added a layer and granted that layer access to fetch our secret using the Conjur Command Line Interface (CLI).

Where Azure Key Vault supported per-principal security on each secret, Conjur supports a more sophisticated role-based security mechanism for its secrets. For a demo like ours where we are retrieving one secret, it seems over the top. But users, layers, and groups serve to manage a multi-group or multi-target environment much more easily.  In Conjur a layer is a concept that is used for grouping and defining access permissions in bulk to a user or host. You might create a layer to represent all the production web servers, one for all testing web servers and one for all development web servers and give each individual layer appropriate access to the secrets it needs.  Then by granting/revoking access to specific host/user to a layers you can control access to these secrets in a far more manageable and repeatable manner .

Wiring it all together

Now that we have setup the basic configuration of pieces we start the interesting work of wiring it all together.

Step 1 – Automated Build, Packaging and Publishing

Octopus Deploy works by using NuGet packages as the container you use to publish your executables, libraries, configuration files, setup scripts, etc. for a deployment.  While there are several ways (manually, external script, etc.) to create these,  we chose to use an extension that Octopus Deploy provides called Octopack.   Octopack adds custom MSBuild hooks into your project that you can later use via MSBuild arguments properly package and publish your project for use by Octopus Deploy.  You install Octopack into each project that you are going to want to deploy (instructions here).  Installation is a straightforward matter of adding the package using the Nuget Package Manager in Visual Studio to the appropriate project(s) in your solution.  Once you have added Octopack you need to update your  build configuration in Visual Studio Online by adding a few custom MSBuild Arguments to package and send your project to Octopus Deploy as shown below:

[code language="powershell"]/t:Build /p:RunOctoPack=true /p:OctoPackPublishPackageToHttp={{Your Server Address}} /p:OctoPackPublishApiKey={{Your API Key}}</code><code>[/code]

Here you will need to insert the address and API Key for your Octopus Deploy Server.  Now whenever you check in code to your project it will be automatically built, packaged and sent to Octopus Deploy for deployment.  In our scenario we chose to publish to Octopus Deploy’s own internal Nuget repository, however you can publish to an external Nuget repo as well.

Note: With this command you can also publish to an external repository but only the internal repository allows for the “Automatic Release Creation” we will discuss in the next step.  The major downside to using the Octopus Deploy repository is that it is not accessible from other applications such as Visual Studio.  This can be a major drawback if you are using Nuget for internally sharing of packages as you will now have to run and maintain 2 separate repos.    While not the biggest pain in the world this is something I would like to see the Octopus Deploy team fix in future releases.

Step 2 – Automating Deployment

Next we automated the deployment of the our newly published deployment package.  Newer version of Octopus Deploy (>2.6) have a nice feature known as Automatic Release Creation.  This feature allows for automatic creation and deployment of Projects that use the internal Nuget repository.

To enable this on a Project and add first add the step using the “Deploy a Nuget Package” option.  You then configure the step to deploy the package you have published from Step 1 by selecting the Nuget package from your Octopus Server feed, setting any needed configuration parameters.  Once you have do this you will see (at the Project level) the ability to turn on Automatic Release Creation as well as a dropdown to select the step.

Once this is enabled each time a new version of your selected Nuget package is added to the feed a new Release will be created and deployed.

Step 3 – Retrieve Conjur Host Token

Now that we have our automated deployment building and deploying to the web we need to add functionality to interact with our Conjur instance to create , configure and retrieve our unique host identity and authentication token.  We are going to do this by adding a scripting step before our Deploy step in our Process in Octopus Deploy.  In this step we are going to write a Powershell script that will make several calls into the Conjur REST API to create and retrieve a unique Host and API authentication token for our deployment.

Note: In Conjur a Host is an identity that is used when the interaction is being done by some sort of automation such as a program or a script.  While a Host and a user are virtually identical the main difference is that a Host can only authenticate via a username and token while a User can authenticate using a username/token or a username/password.

The script for creating a host consists of a few calls to the Conjur REST API using Invoke-WebRequest cmdlet and some minor some manipulation of the returned data to URL encode or Base64 encode the resulting data for future use.

First, we authenticate a user and create a reusable header (Authorization: Token {{insert Base64 encoded token}}) which contains a Base64 encoded version of the API Key as the token.  Once we have the initial authorization token we add a new host to Conjur and then add the Host to both a security group and our previously configured layer.  We then return the value of the new Host including the Id and API token.

[code language="powershell"]function Get-Authentication(){Param([string]$userName, [string]$apiKey, [string]$urlBase, [boolean]$isHost)#get authenitcate tokenIf ($isHost) {$userName = "host/" + $userName}$userName = [System.Web.HttpUtility]::UrlEncode($userName)$uri = "https://" + $urlBase + "/api/authn/users/" + $userName + "/authenticate"Write-Host "Authenticate Uri: " $uri$response = Invoke-WebRequest $uri -Body $apiKey -Method Post -UseBasicParsing#base64 encode the response$token = Out-String -InputObject $response.Content#convert the token to a Base64 encoded string$b = [System.Text.Encoding]::UTF8.GetBytes($token)$token = [System.Convert]::ToBase64String($b)#create header for token authentication$tokenString = 'Token token="' + $token + '"'$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"$headers.Add("Authorization", $tokenString)$headers}function Create-Host(){Param($userName, $password, $urlBase, $groupName)#setup initial authorization$b = [System.Text.Encoding]::UTF8.GetBytes($userName + ":" + $password)$authString = [System.Convert]::ToBase64String($b)$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"$headers.Add("Authorization", "Basic " + $authString)#get token$uri = "https://" + $urlBase + "/api/authn/users/login"$userApiKey = Invoke-RestMethod $uri -Headers $headers$headers = Get-Authentication -userName $userName -apiKey $userApiKey -urlBase $urlBase#get API Key$uri= "https://" + $urlBase + "/api/hosts?ownerid=" + $groupName$apiKey = Invoke-RestMethod $uri -Headers $headers -Method Post#add the host to the Test Layer$hostUri = "https://" + $urlBase + "/api/layers/Test/hosts?hostid=" + [System.Web.HttpUtility]::UrlEncode("host:" + $apiKey.id)$resp = Invoke-RestMethod $hostUri -Headers $headers -Method Postreturn $apiKey;}$key = Create-Host -userName {{insert username}} -password {{insert password}} -urlBase {{insert Conjur URL}} -groupName {{insert group name}}Set-OctopusVariable -name "APIKey" -value $key.api_keySet-OctopusVariable -name "APIUser" -value $key.id[/code]

Once nice feature of Octopus Deploy has the ability to programmatically access and share data, called output variables, between steps at deployment time (see here).  In our script we are setting the variables “APIKey” and “APIUser” to the Host id and Host API Key using the Set-OctopusVariable cmdlet that is available within the Octopus Deploy runtime.

Note: Conjur also has the concept of a Host Factory which enables scripts to create hosts and add them to layers without needing full administrative privileges as shown in the code sample above.  Using this methodology the amount of trust required by the script is significantly less than the method I have shown above, however these is no method for creating the host factory token via the REST API only via the Cojur Command Line Interface (CLI).  The host factory token is ephemeral so this poses an interesting dilemma since we would not want to set this as a variable anywhere on the deployment server and there is currently no CLI available for Windows. 

Edit - 11/17/15 - Our friends at Conjur have informed us that creation of a Host Factory was always possible through the REST API and that the lack of documentation was a simple oversight.  They have corrected this and the documentation to create a Host Factory is available here.

Now that we have successfully created our new host, given it the required access and saved those values into Octopus Deploy our next step is to get those output variables saved into our web.config during the deployment.  Octopus Deploy has functionality built in that automatically does a replacement of appsetting and connectionstring values, based on key names, with Variables stored in Octopus Deploy during a deployment.  This functionality allows us to bind those Variables to the Output Variables which we can set in our Powershell script.  To do this we setup a Variable in the Project to named “APIKey” and “APIUser”  and specify the value to be the output variables specified in the “Setup” step using the Variable Substitution Syntax provided by Octopus Deploy.  Now that we have the Variables bound to our Host Id and API Key we need to turn on the “Configuration Variables” option in our deploy script and voila, now it all works seamlessly.

Step 4 – Retrieve Secrets on Startup

The last step is to retrieve the host identity and authentication token from our application using the “APIUser” and “APIKey” that we added to the web.config by the deployment.  To retrieve our secrets it was a simple matter of using the Conjur REST API to authenticate the Host and retrieve the value of the variable (named “TestVariable” in this case).

[code language="csharp"]var authURI = "https://XX.XX.XX.XX/api/authn/users/"+ HttpUtility.UrlEncode("host/" + System.Configuration.ConfigurationManager.AppSettings["APIUser"])+ "/authenticate";var client = new HttpClient();var content = new StringContent(System.Configuration.ConfigurationManager.AppSettings["APIKey"]);var resp = await client.PostAsync(authURI, content);resp.EnsureSuccessStatusCode();var token = await resp.Content.ReadAsStringAsync();token = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(token.Replace(Environment.NewLine, "")));var variableUri = "https://XX.XX.XX.XX/api/variables/TestVariable/value";client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Token", "token=\"" + token + "\"");resp = await client.GetAsync(variableUri);resp.EnsureSuccessStatusCode();[/code]

Native Tooling

Like Azure Key Vault, using a centralized solution requires you to change the parts of your code that read secrets. For a well-factored application, this might be straightforward, but lets be honest things rarely work this way.   However a few additional tools that made using a centralized server more ‘plug and play’ in the .NET world might be attractive. For example:

• A custom configuration provider which fetched secrets on demand (presumably caching them for efficiency), allowing programs to continue to use the app.config methodology they’re used to.
• A tool like summon which edited a configuration file in place upon deployment by using secret zero to fetch all other secrets and then encrypt them in place using the local machine key.
• Kerberos Authentication to handle the authentication of a user/host in Conjur

Secret zero is still a bit awkward in this scenario. It’s similar to what we see in many toolsets do (see, for instance the ‘cubbyhole’ model of Hashicorp Vault 0.3.), but it’s still awkward. If you are deploying applications into Windows/Azure environment, you are often installing server programs as “service accounts” whose very identity ought to serve as secret zero. A Kerberos/Active Directory authentication back-end for Conjur would make this smoother, eliminating several complex steps.

Summary

Overall the experience with using Visual Studio Online, Octopus Deploy and Conjur was very pleasant one, once you got over the initial learning curve.  This toolset is overkill for a small to medium sized projects, however more complex projects involving Enterprise systems or SaaS based offerings that consist of multiple deployments, layers, clients or deployment scenarios involving a Hybrid Windows/Linux environment would certainly benefit from the added power, flexibility, and fine grained secret management that comes with this tooling.

Pros

• Powerful and more flexible that some other tools
• After initial learning curve the process to manage items was straight forward
• Provides robust encryption of secrets
• Unlike Azure Key Vault or Amazons Key Management Service all your secrets are stored on hardware that is under your control
• Would be useful in SaaS and Enterprise level situations
• Robust Audit trail capabilities
• Conjur provides additional functionality such as Software Firewall and SSH Management that are powerful, especially in a Hybrid enviroment

Cons

• Initial learning curve is non-trivial
• Lack of UI tooling for adding/managing is lacking, current UI is mainly focused on Read Only
• Main focus has been on the Linux side of things so no client libraries exist for .NET or Powershell
• Lack of Command Line Interface for Windows
• Lack of Kerberos support for authentication