Securely tunneling Munin traffic

This is a guest post by Thomas Eisenbarth. Thomas studied computer science at the University of Augsburg, currently works at BINconsult GmbH, Berlin and co-founded makandra GmbH in Augsburg. He and his teams develop and operate web applications.

As Dan discussed already in Getting a Grip on your Operations with Munin and 10 Seconds A Day With Munin, Munin is a quite nifty tool for monitoring the inner life of your machines.

We’re also using Munin for some time now to monitor our staging and production stuff. There is only one little problem. Most of you guys out there have machines at multiple locations and nameservers which may not be collocated. Most setups that I’ve seen so far do their testing and development at the office while the production sites hopefully run out of a datacenter with reasonable bandwidth, security, etc.

So you have your munin-master running on one server and might want to setup a node running “somewhere on the internet” (and not in the LAN with munin-master). munin-node offers a socket (defaulting to port 4949/tcp) for data polling by the master.

Security paranoia!

Usually, you don’t want to open this munin port to the entire unsafe world out there on the web. Sure, munin has the “allow”-directive, but it doesn’t sound like a bullet-proof solution to rely on this alone – especially as the traffic from/to munin-nodes isn’t even encrypted.
If you have an in-house sysadmin staff or a provider maintaining your firewalls they might ask WTF you’re planning to do with port 4949 after you ask them to open it up. So what can we do?

Solution 1: The “big” one: Use VPNs. Done.
Solution 2: Use SSH to tunnel your munin-traffic:

Especially when you have a smaller setup (or even a single remote machine), SSH is a solid, (mostly) secure way to tunnel authenticated & encrypted traffic.

Using SSH with munin

To shorten it up:
1) Give user “munin” on $munin-node a public key w/o password:
“man ssh-keygen” & put it into ~munin/.ssh/authorized_keys on the remote machine.

2) Fire up the tunnel with (e.g.) autossh to ensure that the tunnel is up even if it dies for some reason:

/usr/local/bin/autossh -f -M 5122 -g -N -C -L 4950:$REMOTE_IP:4949 $REMOTE_IP -i $KEYFILE -l munin

I have no deeper experience with autossh, but it seems to do it’s job pretty well. Just be sure to start the autossh-script above on server reboot. It’s a good idea to do this automatically with (e.g.) rc when using FreeBSD (as I do) or Sys-V-style when rocking on Linux.

3) Configure munin-node to accept from the public ip interface. (Use tcpdump port 4949 to see where the traffic comes from in case of doubt)

4) Configure munin-master to collect through the tunnel:

[munin_node_hostname]
      address 127.0.0.1
      port 4950
      use_node_name yes

You should see the new host shortly after modifying the configuration. Usually the munin-master runs periodically launched by cron.

See you next time!
Thomas

You can get in touch with me through awo@thomas-eisenbarth.de.