Secrets in Google App Engine

Google App Engine makes it easy to deploy NodeJS applications. The GAE Standard Environment and SDK support NodeJS out of the box. This makes Google App Engine a great choice ahead of competitors such as Heroku, AWS or Microsoft Azure. Unfortunately though, there’s no support for managing secrets in Google App Engine. When I deployed Dog n Bone to GAE, I found this single shortcoming the main source of complexity.

There are however some workarounds. None of them is particularly nice though.

Why we need secrets in Google App Engine applications

My use case is fairly typical. I want my source code to be publicly visible. It should therefore contain no secrets. I don’t want passwords, auth tokens, phone numbers and so on in source control. 12 Factor methodology suggests storing secrets (and other config) in environment variables.

Heroku has excellent support for managing environment variables per running instance. You can deploy a single application to many environments and then use Heroku’s config vars to store secrets and other per-environment configuration.

Google App Engine however has no such support. Yes, you can set environment variables for a GAE application but the mechanism is to store them in the app.yml application descriptor. GAE requires this descriptor to deploy your application so it can’t really be kept secret. What’s more, we’d like to use a standard application descriptor to deploy to all environments. In my case, the app.yml file is in source control and used by Travis CI to deploy my application. This is not the right place to manage secrets or environment specific configuration.

Google App Engine has no other mechanism for managing environment variables. There is no CLI, API or web console means of setting environment variables per instance. So we need some other way to manage secrets.

Rejected solutions: Import at runtime

It is possible to manage secrets within the Google Cloud Platform stack. For example, you could put your .env file in a Google Cloud Storage bucket and have your application download and import it. Gunar C. Gessner described this nicely in his brutal honesty blog article How to use Environment Variables in GCloud App Engine (Node.js). This seems far too heavy for my requirement. It also ties my application to Google App Engine. I’d prefer it to be vendor agnostic.

It’s also possible to manage secrets in a Google Cloud Datastore and import from there. Again I rejected this as too heavy. My application does not require a file system or database so I don’t want to use Cloud Storage or Cloud Datastore just to manage two secrets.

My solution: Inject at build time

I use Travis CI to build and deploy my application. Travis CI has capability to set environment variables at build time. My solution is to copy build environment variables into a .env file and then deploy the .env file. This means that we do not need to store the secret .env file in (public) source control. The secrets are protected by the (private) CI build system.

The secrets I want Travis CI to manage are:

Google service account json file. This is the authentication token that Travis CI uses to push new builds to Google App Engine. This is created in the Google Cloud Console dashboard.
Twilio App SID and phone number. These are created in Twilio and tie my application to my Twilio account.

I use two different techniques to manage these secrets.

Technique 1: Travis CI encrypted file

I followed the example on Travis CI’s guide to Google App Engine Deployment to encrypt my Google service account json file. The travis CLI takes care of the encryption / decryption process:

travis encrypt-file travisci_google_service_account.json --add

This command:

creates an encrypted copy of the file (travisci_google_service_account.json.enc);
adds a section to the .travis.yml build definition to decrypt the file at build time and;
stores the encryption key and IV as environment variables in your Travis CI project.

After running the command, check your Travis CI project for the new environment variables:

Travis CI managing encryption keys

Let’s just be clear about what is secret here.

travisci_google_service_account.json is secret. As I now have an encryped copy of it, it’s best to delete this to prevent it accidentally being added to source control. If you get stuck, you can always create a new one in the Google Cloud Console dashboard
travisci_google_service_account.json.enc is public. It’s protected by Travis CI’s encryption so can safely be sorted in a public source control repo.
encrypted_f2e7221f9a5f_key and encrypted_f2e7221f9a5f_iv are secret. Travis CI created these secret values and stored them in your build project.

That takes care of build time secrets.

Technique 2: Create an .env file at build time

I want my Twilio App SID and phone number available as environment variables at run time. Travis CI offers a way of configuring environment variables at build time. So it’s fairly straightforward to get Travis CI build a .env file and deploy it with the application.

First, create a script that pipes environment variables to a file:

#!/bin/bash

# Create a .env file from environment variables
echo TWILIO_TWIML_APP_SID=$TWILIO_TWIML_APP_SID >> .env
echo TWILIO_NUMBER=$TWILIO_NUMBER >> .env

Then invoke the script from Travis CI on deploy:

before_deploy: "./deploy/create_env_file.sh"

Finally, add them to the Travis CI project in the Environment Variables section:

Travis CI managed secrets in Google App Engine

Again, lets be clear about where our secrets are:

TWILIO_TWIML_APP_SID and TWILIO_NUMBER are my secrets. I don’t want them in source control. They’re in my Travis CI project as environment variables.
.env contains the secrets. I don’t want this in source control so I have Travis CI create it at build time.

A working example of secrets in Google App Engine

This example showing how I manage secrets in Google App Engine is available in Dog n Bone 1.0.4 on Github. In particular take a look at:

.travis.yml: See the decryption of the Google service account JSON in the before_install section and creation of the .env file in the before_deploy section.
create_env_file.sh: This builds the .env file containing environment secrets.
app.yml: The Google App Engine deployment file. This is how Google suggests you manage environment variables. However, as we want to source control this, we can’t use this for sensitive information.

One final word. In subsequent versions of Dog n Bone, it pulls the Twilio App SID and phone number from the Twilio API. As a result, from version 1.1.0 it’s no longer necessary to build the .env file at build time.