Managing Github Secrets with Vault

I’m a huge fan of GitHub actions and organization level secrets, but you can’t see what the current secret is w/i GitHub (w/o deploying a container somewhere with it exported as an ENV var). I tried to keep a copy in my former secret management tool clipperz.is/app but drift is real. It was always too scary to change a secret; I didn’t really know where it was being used and, thus, which applications I should redeploy to ensure proper operations.

This pain of manual “secret management” (syncing between clipperz and GitHub) + the stagnant codebase of clipperz (last commit was Oct 2019), made me realize it was time to take control of the situation. So, I started looking for “off-the-shelf” services.

1Password Secrets Automation ($29 / month) + Keeper Secrets Manager ($10 / month) are probably highly architected and professionally maintained tools, but should I really give up Netflix and/or Amazon Prime to migrate to one of them for my personal projects? And even if I did this, it doesn’t mean that 5 years from now when they double their prices (or go bankrupt and disappear), I’m stuck yet again trying to figure out how best to manage my secrets.

Instead, I spent the last 9 months learning a lot about running my own Vault instance, setting up a few API tokens and automating the whole secret management flow. It all starts from this crontab entry:

32 * * * * /home/ubuntu/my-ca/update_github_secrets.sh

After ingesting the requisite environment variables, it calls vault_update_secrets.py which initially gets the last-modified dates of all my Vault application secrets. Then it goes over to Github and grabs the same data in order to compare which secrets are out-of-sync (meaning updated in Vault where I now manage my secrets).

Updated (or created) secrets are highlighted via Slackbot message with a super helpful pointer to which repos use those secrets (and may need to be redeployed) via e.g. https://api.github.com/search/code?q=org%3Aackersonde+{secret_name}&type=Code. Here’s a diagram to better explain what’s going on:

Vault manages secret versions OOTB so I can rollback w/ confidence if something breaks. The automatic sync is a lifesaver and I no longer stress about rolling over & updating my secrets on a regular basis. The next, logical automation step is auto-deploying the repos identified to be using updated secrets.

If you want to learn more about running your own Vault server, just lemme know and we can chat. If there’s enough interest, I’ll write a separate post on what I’ve learned in the past year.