Tailoring Your Munin Installation

After following Dan’s tutorial on installing munin on your servers, you already get the benefits of munin’s default plugins. You have graphs showing your CPU, RAM, I/O, as well as MySQL, Exim, and quite some other stats. But most of the time you run some additional software which you also want to montior.

Install Some Custom Munin Plugins

In our case, we have a full Ruby on Rails stack featuring nginx front end web server, HA Proxy load balancer, a cluster of thin instances for the Rails application, memcached for caching page fragments and a MySQL database backend.
No matter how your stack looks like, you’ll find some good munin plugins at MuninExchange. Some of them are pretty basic (or not working as required) but give you a great head start. Just grab one of them and put it into the /usr/share/munin/plugins dir on any munin node. Make it executable (chmod 755 thin_memory) and enable the plugin by symlinking it to /etc/munin/plugins (you must restart your munin-node after adding new plugins).

Alternatively to manually symlinking your plugin, just call root@node:/usr/share/munin/plugins# munin-node-configure --shell to do it for you. It will only link plugins compatible with your configuration (see the munin-node config docs for details)

Test Your Custom Munin Plugins

You’ve got basically three ways of making sure your plugin works:

1. Run the munin plugin script directly:
   This is the most basic way to ensure, that your munin plugin returns reasonable values. Just execute the script directly on your munin node.

   root@node:/etc/munin/plugins# ./thin_memory
   

   Your script should return something like this (depending on how many thin instances you run on which ports):

   thin_8000.value 180
   thin_8001.value 178
   thin_8002.value 175
   thin_8003.value 169
   

2. Use munin-run to execute the plugin locally:
   The next stage is to test whether the munin plugin still returns reasonable values when executed as a munin-node process:

   root@node:/etc/munin/plugins# munin-run thin_memory --debug
   

   This should return something like this:

   # Saving mysql*->env->mysqlopts = --defaults-extra-file=/etc/mysql/debian.cnf...
   ...
   # file: 'thin_memory'
   # Checking thin_*...
   # Checking thin_*...
   # Checking thin_*...
   # Checking thin_*...
   # Want to run as euid/egid 0/106
   # Running as uid/gid/euid/egid 0/106 106/0/106 106
   # DEBUG: About to exec "/etc/munin/plugins/thin_memory"
   thin_8000.value 180
   thin_8001.value 178
   thin_8002.value 175
   thin_8003.value 169
   

3. Run munin-update at your munin server: The final stage is to manually invoke the munin server process to see, whether it is able to collect data from your munin plugin remotely:

   root@munin-server:~# /usr/share/munin/munin-update --debug --nofork --stdout --host node.example.com --service thin_memory
   

   Watch out for lines like these:

   ...
   Dec 17 18:41:31 - [17775] Fetching service: node.example.com->thin_memory
   Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8000-g.rrd with 180
   Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8001-g.rrd with 179
   Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8002-g.rrd with 175
   Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8003-g.rrd with 169
   Dec 17 18:41:31 - [17775] Fetched service: node.example.com -> thin_memory (0.08 sec)
   ...
   

And, last, but not least, there’s good, old telnet which you can use to contact and query any munin-node:

me@anywhere:~$ telnet node 4949

You may use list to see, which services are monitored at your node, and fetch thin_memory to get the current values. Just type help to see what else you can do there.

Customizing your munin installation with additional munin plugins is quite easy and straight forward. You just need some scripts from MuninExchange and enable them for use via munin-node. The various ways of debugging your plugins really help to get your monitoring up fast.