A/B Testing with Webrat

dez&john3313

It all began a year ago. We wanted to introduce a new navigation bar, but gradually, gaining feedback (and confidence) that it was the right upgrade for our users. My idea at that time was a random cookie which would show 20% of our users the new navigation while giving them a separate google analytics code for tracking their visit. And it worked well. Within a few weeks we were able to increase the cookie up to 100% and then simply throw it away.

GUI Based A/B testing in a custom CMS

But the idea remained. We now had a successful prototype for A/B testing. The next hurdle was introducing it into the CMS. Implementing a new page in the code for every hairbrained, well-intentioned GUI change is not exactly exciting stuff for web developers to do. But, this is exactly what our half-assed solution required. A product manager added a new A/B test to the backlog for a developer to implement. How about introducing /page/A/ and /page/B/ params to the CMS system?

Development was fairly straightforward. A single new table in the DB to maintain the new A/B mapping, and a few day’s development later, the product management and editorial staff were able to create their own A/B tests!

The next question was how to ensure we served A/B pages as requested, not to mention the myriad of edge cases (B page requests for pages that only existed in A form, etc.). We needed acceptance tests.

Webrat saves the day (again)

As our backend system is behind an SSL authentication mechanism, Mechanize traps the 401 code and we authenticate with our own do_visit (patch is here). Our build server is a Debian Lenny using Ruby 1.8.7 and Webrat 0.6.0.

Now that we have stable access to the CMS, we can write a webrat test for our A/B testing functionality. Let’s create a B page:

def test_1_cms_create_B_Case
  do_visit($cms_url)
  assert_contain "Login"
  assert_not_contain(PHP_ERROR)
 
  fill_in "username", :with => "xyz"
  fill_in "password", :with => "123"
  click_button "Login"
   
  # create Test Case B in CMS
  do_visit($cms_url + 'structure/page-edit/2366/page/B/')
  assert_contain "caseId=B"
  assert_not_contain(PHP_ERROR)
end

Let’s visit the frontend and ensure we see the B page after overwriting the cookie:

def test_2_visit_demo_page_B
   do_visit($net_url + '/Demo/')
   assert_not_contain(PHP_ERROR)
    
   # overwrite the pageAB cookie to 'B'
   cookie_jar = webrat.adapter.mechanize.cookie_jar
   url = URI.parse($net_url)
   cookie_jar.to_a.each do |cookie|
     if cookie.name == 'pageAB'
       cookie.value = 'B'
       our_cookie = cookie.clone()
       cookie_jar.add(url, our_cookie);
       break;
     end
   end
    
   page = do_visit($net_url + 'ND-intern/Demo/')
   assert_not_contain(PHP_ERROR)
   # Verify we get a B page with our B cookie!
   unless page.root().to_s.include? 'pageB'
     assert false
   end
end

And that’s it. Of course, you need to do some extra testing for the boundary cases, but I’ll leave this as an excercise for the reader 😉 …
Have any webrat tricks you’d like to share with us? Let us know in the comments!