Secure GitHub Deployments to Your Home

I’m a cheap bastard when it comes to online services. All my repositories have been publicly hosted on GitHub well before Microsoft swooped in and offered unlimited private repos.

Since around 2016, this had been my build/deploy pipeline:

You might ask how could that ssh key be comprised. Well, back in the day, CircleCI would happily echo anything you (stupidly) told it right into the publicly available build log including secrets. Yeah, fun times…

Last summer when the new Raspberry Pi 4 was released, my scrooge senses went into overdrive realizing I could recoup the $5 month cost for my Digital Ocean droplet in less than a year and a half.

What I didn’t consider was that it would also take me a year and a half to figure out how-to securely & reliably connect my well established CI/CD pipeline to my home.

I wanted to share how I architected my working setup this summer, but that led me to think even harder about security. How secure was my VPN/ssh deploy from CircleCI? Exposing your home network on the internet is one thing. Bragging about it, quite another. I needed to do an honest assessment and go the extra mile.

Secrets

First things first – I am ruthless about stripping out secrets or any hardcoded sensitive data from my source code. I was always a big fan of using CircleCI secrets and greatly benefited from their shift to Organization wide ones (e.g. not having to copy/paste the same keys into every repository). GitHub employs a mirror copy of this setup.

Since I push all my work to public repositories, I extract all secrets and re-inject at build time. It’s definitely extra effort, but I’m happy (and proud) to share my work online, to inspire others and give back to the amazing community of opensource developers.

Using secrets is good coding practice – not only is it more secure, but it gives you flexibility for deployments in the long term.

VPN

For my deploys from CircleCI, I was connecting via some kind of Cisco IPSec on my router that:

• the manufacture provided no versioning information or update cycle for – basically, it was none of my business
• the credentials were difficult to create on the router via a script so rotating these would be painful – hurdles to automation suck

I had heard a lot about Wireguard and, within an hour of installing it, had successfully connected from my mobile network into my home. Promising! Now to hook it up to CircleCI and away we’d go.

Or so I thought. Not only did CircleCI not have the latest Ubuntu 20.04 available (which has full support for Wireguard packages by default), even after bootstrapping this OS w/i a docker container, I couldn’t get the networking right. It seemed that CircleCI was actively blocking this type of outbound VPN access from docker containers.

As the build was already running well over 5 minutes and wasn’t even close to working, I decided to shop around with other CI/CD providers. First place I went was Semaphore CI. Although I could use an off-the-shelf Ubuntu 20.04, the wireguard connection was also blocked by Semaphore.

Slowly giving up hope, I thought about Github Actions. Now, I don’t like having all my eggs in one basket, but within a few hours of setting up my configuration (which was very comparable to CircleCI), I had a successful ssh connection running ls -lrt on the k3s master node sitting in my hall. I haven’t looked back since (sorry CircleCI \o we had a lot of good years together).

Using the Wireguard container from linuxserverio.io, it’s a breeze to reissue client keys, and I do this on the same monthly cadence as regenerating my deployment SSH certificates:

sudo kubectl exec -it $(sudo kubectl get po | grep wireguard | awk '{print $1; exit}' | tr -d \\n) -- rm -Rf /config/peer_githubActions /config/wg0.conf

As you can tell, I’m running this container as a kubernetes service/deployment. The entire client peer config is uploaded as a secret to my Github org allowing me to access it from any of my repositories during deployment:

- name: Create wireguard tunnel configuration
  run: |
    sudo apt-get install wireguard resolvconf
    echo "${{ secrets.CTX_WIREGUARD_GITHUB_ACTIONS_CLIENT_CONFIG }}" | sudo tee /etc/wireguard/tunnel.conf

To save you some hair pulling, remember to uncomment net.ipv4.ip_forward=1 in your /etc/sysctl.conf on any nodes you’d like to have Wireguard running.

SSH access

Ok, we’ve got rotating Wireguard client creds – time to move on to the main event: ssh access to deploy your new code. Rotating keys here would mean carefully managing each node’s ~/.ssh/authorized_keys file, but why automate yack shaving?

Searching around the Internets led me to Aakash Yadav’s article about SSH certificates which does a great job explaining the why’s and how’s of using them for authentication. With this, I can regenerate the ssh keys used for GitHub deployments every month, create a certificate and upload these artifacts to my organization’s secrets store.

The authentication magic happens because I created a Certificate Authority (CA) keypair, adding the public key to the bottom of every node’s /etc/ssh/sshd_config file with this simple line: TrustedUserCAKeys /etc/ssh/github_deploy_ca.pub and using the private CA key (ca_key) to sign & certify any ssh keypair:

ssh-keygen -t ed25519 -a 100 -f $DIR/id_ed25519_github_deploy -q -N ""
ssh-keygen -s $DIR/ca_key -I github -n ubuntu,user -P "$CACERT_KEY_PASS" -V +5w -z $(date +%s) $DIR/id_ed25519_github_deploy

If these credentials are compromised, they’ll automatically be invalidated after some time period. I chose 5 weeks to ensure my 4 week update cron (shown below) is always one step ahead of the expiration.

Automation

# m h dom mon dow command
50 07 02 * * /home/ubuntu/my-ca/gen_new_deploy_keys.sh > /var/log/gen_new_deploy_keys.log 2>&1

We’ve done our homework when it comes to security, but now it’s time to put our money where are mouth is and automate it. By setting an expiration timer on the cert validity, we’ve forced ourselves to do the right thing. The cron executes this script which rotates both the GitHub wireguard VPN client credentials and ssh deploy key & certificate on the 2nd day of every month.

Updating GitHub secrets is not very straightforward, so I ended up writing this python script todo the heavy lifting.

Figuring out how-to actually get a token to allow this automation was a project in and of itself! Yes, I definitely do recommend you set yourself up with a GitHub organization as this will allow you to share secrets across repository builds.

I’ve created some dummy URLs in the steps below to help you out. You’ll need to replace <your-org> & <your-app> after navigating to them.

0. Create a new GitHub App in your organization. I called mine “Secrets manager” (aka <your-app>):

1. Add the following permissions:

• R/W access to Repository & Org Secrets (update the keys)
• R/W access to Repository Actions (trigger repo workflow CI/CD w/ new keys)

2. To sign access token requests, generate & download a private key for your newly created app:

3. Install the application to your organization:

4. Finally, note the install ID by navigating to your new app and looking at the URL. You’ll need this to generate your Access Token with JWT.

Securely automating your home

Now, I run my homepage, slackbot, pihole, private VPN, transmission, plex media server & wireguard from my 2 node cluster. Deployment performance is snappy and my staging environment is just a kubernetes node label.

Next up: Syncthing and taking back control of my photos.