A new twist on testing Ruby

posted by cjh, 22 June 2012

How do you know your tests are testing what you think they’re testing? That is, they pass when your code is right, but will they always fail when your code is wrong? Mutation testing is the only way to find out.

The estimable Ryan Davis, always a master of the art of the improbable (and often impenetrable!) gave the Ruby world Heckle. By hooking in to the internals of Ruby, it could return a snapshot of the compiled code, in the form of an Abstract Syntax Tree (or parse tree), modify it, and re-run the tests to make sure that something breaks.

Unfortunately for us, Ruby moved on, and no longer generates an AST. Instead it generates machine code for a virtual machine (VM). So the last working version of Heckle was for Ruby 1.8.7. It you’re still using that, perhaps you haven’t heard of the huge pointy stone things the Egyptians made recently.

Enter Twister. By installing a patched version of Ruby 1.9, the kinds of code mutation performed by Heckle become a core capability of the Ruby language. A single kernel method called set_parse_func provides a window into the parser, the part of Ruby that figures out what your code means. Not only can set_parse_func see things it probably shouldn’t, it can tweak them to make Ruby think it saw something different.

But wait, do I really want download and install Ruby with patches? Well, relax because it’s almost as easy as installing any other Ruby version, using RVM. Just download twister.patch to the file twister.patch and say

rvm install –head –patch twister.patch%1 1.9.3

and you’ll have a not-quite-normal version of Ruby 1.9.3 installed and ready to twist. A word of caution though: Make sure the path to your copy of ‘twister.patch’ doesn’t have any spaces in it, it seems to confuse RVM. Check that it says “Applying patch ‘twister.patch’ (located at…”

Now, you want RSpec. Well, I’m still working on that, but you can track at my work in progress on Github. The –twist option is the one you want to try.