The Problem

At Cloverhound, I was able to choose the testing framework and implementation that would best suit my projects. Armed with the conviction that I ought to be testing behaviors from the user’s perspective, I reached for my familiar arsenal of Rails testing tools: RSpec, Capybara, Selenium, and Database Cleaner. RSpec feature tests are great for behavior testing, but they are notoriously slow, partially due to the need to clean the database in between each spec run. So, it makes sense to reuse as much code as possible and write long specs in order to eliminate costly reads and writes from the test database.

For the purposes of this post, imagine that we are building a meeting attendance app that allows a user to log in with their username, password, and a meeting ID assigned to each meeting. We want to test the login functionality, so we write a test that verifies that a user cannot log in to our app if they do not enter the correct information. That hypothetical spec might look something like this:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user    = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting = Meeting.create(meeting_id: 1234)
  end

  context "unsuccessfully" do
    scenario "without a correct meeting ID, username and password", js: true do
    visit login_path

    fill_in name: "meeting_id", with: "WRONG MEETING ID"
    fill_in name: "username",   with: "WRONG USERNAME"
    fill_in name: "password",   with: "WRONG PASSWORD"
    click_on "Submit"

    expect(page).to have_content "Meeting ID is invalid"
    expect(page).to have_content "Username is invalid"
    expect(page).to have_content "Password is invalid"
    expect(current_path).to eq(login_path)
  end
end

Pretty simple. We enter the wrong credentials and some error messages display on the screen. But wait! If the user enters no credentials at all and clicks submit, they should see some different errors. Let’s add some expectations for that scenario:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user    = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting = Meeting.create(meeting_id: 1234)
  end

  context "unsuccessfully" do
    scenario "unsuccessfully without a correct meeting ID, username and password", js: true do
      visit login_path

      fill_in name: "meeting_id", with: "WRONG MEETING ID"
      fill_in name: "username",   with: "WRONG USERNAME"
      fill_in name: "password",   with: "WRONG PASSWORD"
      click_on "Submit"

      expect(page).to have_content "Meeting ID is invalid"
      expect(page).to have_content "Username is invalid"
      expect(page).to have_content "Password is invalid"
      expect(current_path).to eq(login_path)

      fill_in name: "meeting_id", with: ""
      fill_in name: "username",   with: ""
      fill_in name: "password",   with: ""
      click_on "Submit"

      expect(page).to have_content "Meeting ID cannot be blank"
      expect(page).to have_content "Username cannot be blank"
      expect(page).to have_content "Password cannot be blank"
      expect(current_path).to eq(login_path)
    end
  end
end

Great! But, oh… not quite. There is also a Terms and Conditions modal that pops up after the user enters the correct credentials, prompting the user to accept or decline the terms. If they decline the terms, they are redirected to the login page. Let’s make sure to include that example in our spec:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user    = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting = Meeting.create(meeting_id: 1234)
  end

  context "unsuccessfully" do
    scenario "without a correct meeting ID, username and password", js: true do
      visit login_path

      fill_in name: "meeting_id", with: "WRONG MEETING ID"
      fill_in name: "username",   with: "WRONG USERNAME"
      fill_in name: "password",   with: "WRONG PASSWORD"
      click_on "Submit"

      expect(page).to have_content "Meeting ID is invalid"
      expect(page).to have_content "Username is invalid"
      expect(page).to have_content "Password is invalid"
      expect(current_path).to eq(login_path)    

      fill_in name: "meeting_id", with: ""
      fill_in name: "username",   with: ""
      fill_in name: "password",   with: ""
      click_on "Submit"

      expect(page).to have_content "Meeting ID cannot be blank"
      expect(page).to have_content "Username cannot be blank"
      expect(page).to have_content "Password cannot be blank"
      expect(current_path).to eq(login_path)
    
      fill_in name: "meeting_id", with: @meeting.meeting_id
      fill_in name: "username",   with: @user.username
      fill_in name: "password",   with: @user.password
      click_on "Submit"
    
      expect(page).to have_selector(".terms-and-conditions-modal")
    
      within ".terms-and-conditions-modal" do
        click_on "Decline"  
      end
    
      expect(current_path).to eq(login_path)
    end
  end
end

Wait! we forgot to make sure that the meeting ID is not only valid, but also that the meeting is scheduled for the current day, since occasionally the meeting IDs are reused by the client, and they don’t want users logging in to stale meetings that happen to be leftover in the database. We’ll have to put that in our spec, too, making sure we have two meetings scheduled for different days:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user          = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting       = Meeting.create(meeting_id: 1234)
    @stale_meeting = Meeting.create(meeting_id: 5678, date: Date.today - 1.day)
  end

  context "unsuccessfully" do
    scenario "without a correct meeting ID, username and password", js: true do
      visit login_path

      fill_in name: "meeting_id", with: "WRONG MEETING ID"
      fill_in name: "username",   with: "WRONG USERNAME"
      fill_in name: "password",   with: "WRONG PASSWORD"
      click_on "Submit"

      expect(page).to have_content "Meeting ID is invalid"
      expect(page).to have_content "Username is invalid"
      expect(page).to have_content "Password is invalid"
      expect(current_path).to eq(login_path)    

      fill_in name: "meeting_id", with: ""
      fill_in name: "username",   with: ""
      fill_in name: "password",   with: ""
      click_on "Submit"

      expect(page).to have_content "Meeting ID cannot be blank"
      expect(page).to have_content "Username cannot be blank"
      expect(page).to have_content "Password cannot be blank"
      expect(current_path).to eq(login_path)
    
      fill_in name: "meeting_id", with: @meeting.meeting_id
      fill_in name: "username",   with: @user.username
      fill_in name: "password",   with: @user.password
      click_on "Submit"
    
      expect(page).to have_selector(".terms-and-conditions-modal")
    
      within ".terms-and-conditions-modal" do
        click_on "Decline"  
      end
    
      expect(current_path).to eq(login_path)
    
      fill_in name: "meeting_id", with: @stale_meeting.meeting_id
      fill_in name: "username",   with: @user.username
      fill_in name: "password",   with: @user.password
      click_on "Submit"

      expect(page).to have_content "Meeting is not scheduled for today"
      expect(current_path).to eq(login_path)
    end
  end
end

Oh, one more thing… actually you probably get the idea by now. These specs can quickly balloon out of control as the tester struggles to spec out examples to meet every requirement in the project design doc. Logging in is a relatively uncomplicated behavior that is pretty intuitive and straightforward to most developers, so this spec proliferation is not a big deal in our example above. But when behaviors become more complex with branching paths that a user might take, it becomes difficult to keep track of which component of the overall behavior is being tested at each point in the scenario.

One possible solution is to make the spec example description more verbose, giving you something like:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user          = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting       = Meeting.create(meeting_id: 1234)
    @stale_meeting = Meeting.create(meeting_id: 5678, date: Date.today - 1.day)
  end

  context "unsuccessfully" do
    scenario "without a correct meeting ID, username and password, or if they enter a blank username and password, or if they decline the terms and conditions, or if the meeting is not scheduled for the current day", js: true do

    ...

    end
  end
end

But (1) that looks hideous, and (2) it still doesn’t solve the problem of identifying which line of code refers to which sub-behavior that is being tested. If I get a failure on line X, does it mean the user entered a blank username, or did they enter a correct meeting code on an incorrect day? Adding a comment above each example might be one option, but then you lose the benefit of RSpec’s lovely documentation formatting that prints successful example descriptions in green text to STDOUT. Again, we could split each of these cases into separate scenarios, but that would add seconds or even minutes to the test suite depending on how complex it is to create database fixtures for your examples.

The Solution

What we really want is a way to print semantic output from within the specs themselves to STDOUT. It would be even nicer if this semantic output conformed to existing RSpec formatting conventions. This would allow us to pinpoint what each line of code is actually testing, while also displaying some nice output in the console that informs the person running the test suite what each example covers. A simple puts statement can be used to print text to the console from within a spec, but the text will not be nested with other examples in its group. Also, you lose the nice green color of a passing scenario.

A novel solution would be to divide our spec example into sections that delimit each sub-behavior we are testing. In the example spec above, those sections might look something like this:

require 'rails_helper'

RSpec.feature "A User logs in to their dashboard" do
  before do
    @user          = User.create(username: "mrjaimisra", password: "thisisasecret")
    @meeting       = Meeting.create(meeting_id: 1234)
    @stale_meeting = Meeting.create(meeting_id: 5678, date: Date.today - 1.day)
  end

  context "unsuccessfully", js: true do
    scenario "without a correct meeting ID, username and password" do
      visit login_path

      fill_in name: "meeting_id", with: "WRONG MEETING ID"
      fill_in name: "username",   with: "WRONG USERNAME"
      fill_in name: "password",   with: "WRONG PASSWORD"
      click_on "Submit"

      expect(page).to have_content "Meeting ID is invalid"
      expect(page).to have_content "Username is invalid"
      expect(page).to have_content "Password is invalid"
      expect(current_path).to eq(login_path)    
    
      section "if the meeting ID, username, or password are blank" do
        fill_in name: "meeting_id", with: ""
        fill_in name: "username",   with: ""
        fill_in name: "password",   with: ""
        click_on "Submit"

        expect(page).to have_content "Meeting ID cannot be blank"
        expect(page).to have_content "Username cannot be blank"
        expect(page).to have_content "Password cannot be blank"
        expect(current_path).to eq(login_path)
      end
    
      section "if the user does not accept the terms and conditions" do
        fill_in name: "meeting_id", with: @meeting.meeting_id
        fill_in name: "username",   with: @user.username
        fill_in name: "password",   with: @user.password
        click_on "Submit"
    
       expect(page).to have_selector(".terms-and-conditions-modal")
    
        within ".terms-and-conditions-modal" do
          click_on "Decline"  
        end
    
        expect(current_path).to eq(login_path)
      end
    
      section "if the meeting is not scheduled for the current day" do
        fill_in name: "meeting_id", with: @stale_meeting.meeting_id
        fill_in name: "username",   with: @user.username
        fill_in name: "password",   with: @user.password
        click_on "Submit"

        expect(page).to have_content "Meeting is not scheduled for today"
        expect(current_path).to eq(login_path)
      end
    end
  end
end

This solution is superior to the others for a couple of reasons. First, it groups related code into namespaced blocks that help outside developers understand the thought process of the person who wrote the test. Remember, the person running your test suite is not always the person writing the specs being run. Once a spec has been added to the suite, future developers will need to tweak and maintain it as they layer on features that complexify the original behaviors being tested. In order to make those future developers’ work easier, it is helpful to include as much information as possible about why certain decisions were made in past iterations of the code.

Second, as we discussed earlier, it is preferable in many cases to write longer specs that touch the database as infrequently as possible in order to speed up each run of the test suite. At Cloverhound, we use a Continuous Integration server that runs the test suite against every PR we commit to version control, so running tests can be costly when pushing up new features. Being able to simply divide a scenario into semantically significant sections, rather than writing a whole new scenario that relies on time-consuming setup methods, saves developers time and allows them to do what they do best: write code.

The Method

So how do we accomplish this? In version 3, RSpec released an API for its Formatter class that allows developers to add custom formatting to their console output during each spec run. It is also possible to write custom Ruby modules that can be included in RSpec’s runtime configuration, like so:

module ExampleSections
  def section(name)
    RSpec.configuration.reporter.publish(:example_section_started, :name => name)
    yield
  ensure
    RSpec.configuration.reporter.publish(:example_section_finished, :name => name)
  end
end

RSpec.configure do |c|
  c.include ExampleSections
end

The real magic here is RSpec’s #publish method. The first argument is the name of the event that you want listen for during the spec run. So :example_section_started and :example_section_finished are methods that we will define in our custom Formatter class. The second argument :name is the example description that you pass into each section block (i.e. “if the meeting ID, username, or password are blank”, or “if the user does not accept the terms and conditions” in our spec example above). The :name attribute will be passed into the #example_section_started and #example_section_finished methods as the first argument.

The code example above can be added to a file in the spec/ directory, and then that file must be required in your spec_helper.rb in order for you to call the #section method in your feature specs.

Now, let’s define our custom formatter as described in the RSpec docs:

class CustomFormatter
  # This registers the notifications this formatter supports, and tells
  # us that this was written against the RSpec 3.x formatter API.
  RSpec::Core::Formatters.register self,
                                   :example_group_started,
                                   :example_section_finished,
                                   :example_group_finished
  def initialize(output)
    @output           = output
    @group_level      = 0
    @section_messages = []
)  end

  def example_group_started(notification)
    @group_level += 1
  end

  def example_section_finished(notification)
    @section_messages << passed_output(notification.name.strip)
  end

  def example_group_finished(notification)
    @group_level -= 1
    @section_messages.each { |message| @output.puts message }
    @section_messages = []
  end

  private

    def passed_output(example)
      RSpec::Core::Formatters::ConsoleCodes.wrap("#{current_indentation}#{example}", :success)
    end

    def current_indentation
      '  ' * @group_level
    end
end

What are we doing here? Let’s go through some of the important aspects of this CustomFormatter class (to see an example for writing your own documentation-style formatters, check out the RSpec DocumentationFormatter class).

One important thing to remember is that each custom method must be registered with the RSpec Formatters class (lines 4-7 above), regardless of whether you are overwriting an existing RSpec event like #example_group_started, or if you are writing a custom event like #example_section_finished. When you initialize your CustomFormatter class, the output variable (line 9) refers to what will be printed to the console, @group_level (line 10) is responsible for indenting your output text (each section should be nested underneath its relevant scenario in the console). And @section_messages (line 11) is an array of all the section descriptions that belong to each spec example.

The #example_group_started (line 14) method is called after each #context block begins, and adds indentation to group the scenario and section descriptions underneath each context description in the console.

#example_section_finished (line 18) collects the output from each specific section block being run and formats it with the #passed_output helper method.

#example_group_finished (line 22) resets the indentation level back to zero in preparation to run the next context block, prints out the descriptions for each of the sections that passed during the test run, and clears the previous @section_messages after they have been printed to the console.

Notice that all three of these methods #example_group_started#example_section_finished, and #example_group_finished, accept one argument notification, which is the name attribute that is passed into the section method defined in the ExampleSections module from the previous code block.

The #passed_output helper method (line 30) formats the output text in green (the color of a passing expectation in RSpec), and indents the text to the appropriate level based on the @group_level counter that gets translated into whitespace in the #current_indentation helper (line 34).

Finally, the last step is to include this formatter in your .rspec configuration file in the root directory of your project. To do so simply add the --format CustomFormatter flag on a new line in the config.

I probably should have mentioned earlier that I made the assumption that everyone reading this is using the --format documentation flag when running their specs, so you must add that flag into your .rspec file as well in order for all of the formatting to print correctly. One awesome thing about RSpec Formatters is that they can be layered on top of one another, providing complementary output streams that can be invoked and silenced individually.

(NOTE: You may notice we never defined and registered the :example_section_started method that is called in the ExampleSections module, but that is because we don’t need to print anything to the console or change indentation levels when an individual section begins.)

Now, when we run our specs, we should get something nice like this if everything passes:

Inspiration

I would love to say that I came up with all of this on my own, but after scouring the Googlenet for some indication of how to divide scenarios into sections, and after poking around the graveyards of some unmaintained gems that attempted to print more semantic output from within RSpec feature specs, I found this amazing issue thread. In the replies, RSpec maintainer @myronmarston provided the code for the ExampleSections module included in this post, and encouraged someone to use it to make section formatting work. Most of the CustomFormatter class code was lifted from RSpec’s DocumentationFormatter class referenced above.

In case you are wondering why dividing scenarios into sections is not already included in the RSpec library, another contributor in the issue thread discusses the team’s reluctance to test multiple behaviors in the same spec. I guess it all comes down to how one defines a single “behavior”.

Conclusion / Ideas for Future Iterations

I hope these few lines of code help add some semantic text to your RSpec test run output that assists in grouping complex behaviors within feature specs. I have long been searching for a way to write extended feature specs that reuse database fixtures to save time, but also provide context about what behavior is being tested on each line of code. Hopefully, the section syntax makes specs easier to read for developers who maintain your applications’ RSpec test suites. Ultimately, I would love to learn how to print more granular failures/error messages/stack traces that use the section names, but so far I have only been successful with printing out section names when the examples within a section block pass.

If anyone plays around with the section syntax, or learns anything cool about RSpec Formatters that they’d like to share, please reach out to jmisra@cloverhound.com and let me know what you discover!