Debugging Cucumber tests on Codeship

I’ve been running into a few problems recently where my cucumber tests fail in the build but not on my local machine. It’s often related to the tests running slightly faster than the javascript on the build machine, but it can be really hard to catch.

A couple of things have worked well. First of all, I’ve added capybara-screenshot gem to my cucumber tests – setting it up was as simple as adding it to the gemfile and putting one line in the env.rb for my tests:

require 'capybara-screenshot/cucumber'

Every time a test fails, it captures an HTML output and an image. I’ve discovered that running tests without javascript – which uses the default headless Capybara driver – does not create an image, and also using the Selenium driver in Chrome throws an error, but that hasn’t really caused an issue.

The next step was accessing the details of the fails on the build. I’m using Codeship which is an awesome cloud-based CI platform that offers 100 builds a month for FREE (plenty for a solo developer on a startup!) Codeship allows me to create a debug build and access it via SSH, which was great because now I can generate images and HTML from the failed tests.

The last problem to solve was getting the images. There’s probably a few ways to skin this cat – I started off just using the HTML, but a picture not only tells a thousand words, it usually is a much faster way of seeing the root of the problem than picking through the HTML code. Turns out SFTP works perfectly – I just use the server IP and port from the SSH command that Codeship provides:

sftp -P [port number] user@ip_address
get [image file name]

I do still need to ssh in and run the failing tests, but finally I can see the broken screens! Much faster.