Selenium Testing in the Cloud with Sauce Labs

One small comment for Matthias, one giant leap for our testing infrastructure. In my last post about anti-fixes, I expressed my reservations about selenium test automation. Matthias mentioned the companies Sauce Labs and Cloud Testing maintained virtual test server farms so that I wouldn’t have to. Here’s why Sauce Labs made my choice super easy:

Hudson Integration

Like I mentioned, any solution needs to fit into my automated build. No extra tests or scripts please – I have enough of those lying around. Sauce Labs’ excellent documentation makes it integrating with Hudson stupid simple. The Sauce OnDemand plugin, written by Kohsuke Kawaguchi, even features Sauce Connect functionality for creating ssh tunnels to reach your firewalled test environment from the outside.

The only thing I’m missing here are some nice back links within the Hudson job to the finished test results on the Sauce Labs site. But plugins are made for tinkering with and this should be an easy implementation.

$50/month for 1000 minutes testing

I did a bit of arithmetic and figured out at 1 minute / test * 10 tests * 5 builds / day * 20 days = 1000 minutes. So the bare minimum is taken care of. At 5 cents / minute over, say you have 10 builds a day. Your monthly bill would be $50 + (.05 * 1000) = $100 / month. This means you should be pretty damn careful about automatic build integration!

A few words about test timings. Of course the number one factor will be your site’s responsiveness. But, I’ve definitely noticed a bit of a performance hit for using ssh tunnels (particularly if you have an SSL page – double encrypted fun). Notable is the ~33% test time increase for video recordings. Hmm? Video recordings, you ask? Why, yes, Sauce Labs can make a video of your test session…

Video recording of your selenium test case

How cool is that? Want to get a better understanding of why a particular test failed? Screenshots in Selenium generally are enough, but a video just feels so much more alive. You get an idea of that lag between clicks (along with any strange javascript or browser popups). Even cooler are the live streaming broadcasts of your tests under Windows via a nice VNC browser plugin. Here’s a sample test run I did last week showing a simple login (advance to around 20 secs to save a bit of boredom):

Last but not least, you get 30 days to try all this stuff out for free. Another great example of the power of communities – without this blog and your comments I would have just kept my head in the sand, oblivious of a very viable alternative to procuring and configuring my own selenium server farm.