Resque: Email Notifications On Failed Jobs

I’m currently using Github’s resque, a Redis backed Ruby library for creating and running asynchronous and scheduled jobs. As we’re running business critical jobs with the resque scheduler plugin, I really want to know via email if a job fails. In this post I want to share my solution for making resque send emails on failed jobs

Resque Failure Handling

By default, resque uses the Redis failure handler to record failures. They are then available in the “Failed” tab in the resque web application. Resque comes with some other helpful failure hanlders, namely one for hoptoad and one to be able to combine multiple failure handlers. As I want to keep storing failures in Redis and want to be alerted via email, I use a combination of the multiple failure handler as well as my own Notifier handler.

The Resque Email Notifier Handler

Using the Notifier handler found at labnotes.org as a starting point, I evolved it into a solution which finally works for me. The Notifier as found at labnotes did not work for me – I assume that resque has evolved enough since January 2010 to break it.

Here is my version of the resque email notifier:

require 'net/smtp'
module Resque
  module Failure
    # Send an email to the developer, so we know something went foul.
    class Notifier < Base
      
      class << self
        attr_accessor :smtp, :sender, :recipients
      end
      
      def self.configure
        yield self
        Resque::Failure.backend = self unless Resque::Failure.backend == Resque::Failure::Multiple
      end
            
      def save
        # Create notification email
        msgstr = <<END_OF_MESSAGE
Subject: [Resque] #{exception}

Queue:    #{queue}
Worker:   #{worker}
 
#{payload.inspect}
 
#{exception}
#{exception.backtrace.join("n")}
END_OF_MESSAGE
        
        Net::SMTP.start(self.class.smtp[:address], self.class.smtp[:port], self.class.smtp[:domain]) do |smtp|
          smtp.send_message(msgstr, self.class.sender, self.class.recipients)
        end
      rescue
      end
    end
  end
end

If you’re using resque as part of a Ruby on Rails application, like I do, just put that file under lib/resque/failure/notifier.rb.

Using Multiple Failure Handlers In Resque

To be able to use your brand new resque email notifier without losing the default behaviour, you need to configure resque to use the multiple failure handler with Redis and the notifier failure handlers in your initializer or rake task:

require 'resque/failure/notifier'
require 'resque/failure/multiple'
require 'resque/failure/redis'

Resque::Failure::Multiple.configure do |config|
  config.classes = [Resque::Failure::Redis, Resque::Failure::Notifier]
end

Resque::Failure::Notifier.configure do |config|
  config.smtp = {:address => 'mail.example.com', :port => 25, :domain => 'example.com'}
  config.sender = 'resque@example.com'
  config.recipients = ['me@example.com', 'you@example.com']
end

config.classes of the multiple failure handler takes an array of failure handlers. Please be aware of the fact that it delegates all calls e.g. for getting counters etc. to the first plugin in the chain. So you should make sure that the first plugin has all of them implemented (see the Base failure handler for the defaults).

My Notifier accepts a hash for the SMTP settings, a sender, and an array of recipients. You could use it stand alone as well, but you would lose recording Exceptions in Redis then. Your call.

If you really need to know if a resque job fails and you want to get an email notification (instead of e.g. using the bundled hoptoad handler) the above should help you get there. Does it work for you? Or do you have a better solution? Please let us know in the comments!