Cukedoctor - when test automation meets living documentation

When I was reading one of Uncle Bob's books (Clean Code or The Clean Coder, don't remember which one) I've stumbled upon this project he created called FitNesse. I very liked the concept of it, let me explain why.

What is FitNesse?

The idea behind this tool is pretty simple:

  • You write Acceptance Tests in plain text file,
  • Each text file is also a markup wiki page so it serves both the role of documentation and Specification by Example,
  • You can launch the tests directly from the FitNesse wiki page directly and see the results. See below: An Example FitNesse Test live results

Problem solved?

Here is why FitNesse tool is neat:

  • It encourages collaboration between IT and business: the Development Team together with Product Owner can specify the tests and enrich them with proper wiki content so it's easy to understand for everyone,
  • Using FitNesse we can easily create a single source of thruth, since we store our tests and reference documentation in the same source material (the text files),
  • When we have a single source of truth, it's easier to maintain Domain's Ubiquitous Language.

It's sure nice, but...

FitNesse looks like a promising tool, but in addition to all the great things I've also found some issues with it:

  • The tests are written using "fixtures" defined by FIT Framework. That was a totally new concept for me which I needed to learn, a concept that I think is not broadly used within the industry,
  • These fixtures are organised in a table manner and are IMHO not easy to read and understand. The test may be just fine for very simple scenarios like the simple 2 Minute Example showed on the FitNesse web page, but I can see that describing some state transitions in a complex system using such tables is not going to work well,
  • Big overhead: In order to create your own test you need to have proper fixture table and glue code that will link the test contents with your production code, that's understandable. But it was crazy to me, HOW MUCH boilerplate code is needed in order to launch a simple test and integrate that into maven lifecycle.

Cukedoctor to the rescue

Once I knew what my problems with FitNesse were, I've started to look for some other tools that will scratch the same itch. It took me a while but I've finally found a satisfying solution to the problem: Cukedoctor.

The concept of the tool is the same but the details are what makes the difference. So here's the usual workflow with new solution:

  • You write some Acceptance Tests in BDD style, using Cucumber & Gherkin Language
  • Each .feature file also serves as a markup wiki page which uses AsciiDoc format. So essentially we still keep Specification by Example together with reference documentation (same as with FitNesse)
  • Once you have your tests & docs in place:
    • You launch the Cucumber tests,
    • Based on the Cucumber .json test results & enriched .feature files Cukedoctor is generating a living documentation document (html and/or pdf).

Benefits

Cukedoctor provides you with the following:

  • It Encourages collaboration between business and IT as well, but instead of using FIT fixtures, we're using plain BDD with Gherkin so the specification reads like a charm, and is much easier to get into (both from technical and non-technical people)
  • We still have (same as with FitNesse) single source of truth stored on our repository files (this time in .feature files instead of plain text),
  • Cukedoctor is much easier to integrate with build tools such as Maven or Gradle and almost no boilerplate code is required (I mean in addition to the actual test implementation).

However…

There is one feature missing in Cukedoctor that is available in FitNesse - that is running the tests from the browser level using the documentation website.

Since FitNesse is actually a live software component with http server etc. (whereas Cukedoctor produces static html and/or pdf file) you can play around with wikis & test fixtures and run your tests straight away from your browser. That's a nice feature however I'm wondering how often it's gonna be used - let's be honest, can you imagine that some business person is gonna write a table in markup language (using FIT fixtures) to run his own custom Acceptance Tests against your code? :). Also, please remember that FitNesse doesn't handle the versioning of wikis for you so if you have your documents available somewhere on a server, once somebody will adjust the wiki content & test values it's up to you to make sure that changes are reflected in your source code repository.

However regardless of it's downsides it's worth mentioning that there is such feature in FitNesse whereas in Cukedoctor there isn't - perhaps that's the one killer feature you need that will justify using one tool over the other.

Lets see it in action!

Below you can find the whole process I've followed to set up a project using Cukedoctor. Tools & technologies used:

For ready repo containing the project sources see link below:

Now, to the details.

Tools setup

Here are my maven dependencies:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>

  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-spring</artifactId>
    <version>${cucumber.version}</version>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-java</artifactId>
    <version>${cucumber.version}</version>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-junit-platform-engine</artifactId>
    <version>${cucumber.version}</version>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>org.junit.platform</groupId>
    <artifactId>junit-platform-suite</artifactId>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>org.assertj</groupId>
    <artifactId>assertj-core</artifactId>
    <version>${assertj.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies>

Those would allow me to run Cucumber tests. I also had to add junit-platform-suite and cucumber-junit-platform-engine dependencies in order to run Cucumber tests in JUnit 5 fashion. Now, to generate Cukedoctor doc, we need also to configure it. The way I did it was through cukedoctor-maven-plugin:

<plugin>
  <groupId>com.github.cukedoctor</groupId>
  <artifactId>cukedoctor-maven-plugin</artifactId>
  <version>3.7.0</version>
  <configuration>
    <outputFileName>documentation</outputFileName>
    <outputDir>docs</outputDir>
    <format>all</format>
    <toc>left</toc>
    <numbered>true</numbered>
    <docVersion>${project.version}</docVersion>
    <documentTitle>${project.name} living documentation generated by Cukedoctor</documentTitle>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>execute</goal>
      </goals>
        <phase>prepare-package</phase>
    </execution>
  </executions>
</plugin>

With such configuration the documentation will be created each time we prepare a package. You can find additional documentation on the plugin here.

Then I created a class wich configures JUnit 5 Suite to run Cucumber tests:

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameters({
    @ConfigurationParameter(key = Constants.GLUE_PROPERTY_NAME, value = "e2e"),
    @ConfigurationParameter(key = Constants.PLUGIN_PROPERTY_NAME, value = "json:target/cucumber.json")
})
public class CucumberTests {}
  • Line 2: @IncludeEngines annotation marks that this suite will use Cucumber engine, provided in cucumber-junit-platform-engine dependency,
  • Line 3: @SelectClasspathResource("features") annotation says that the .feature files are stored under the classpath /features resource path,
  • Line 5: @ConfigurationParameter annotation says that the glue code (Java classes with Gherkin step definitions) are to be searched within the java e2e package,
  • Line 6: @ConfigurationParameter annotation enables json plugin which will result in producing proper .json results to be later used by Cukedoctor.

We now have everything in place to do some actual BDD coding.

Let's write some Calculator app, no one has done that before!

Before we jump into creating our living documentation, few words about the "software" itself. As an exercise I've taken a simple software idea - creating a service that could add two numbers together. Here, I will reference it as Calculator.

Let us work on adding feature to our Calculator. Here's my initial feature file (just BDD Gherkin test, no asciidoc yet here):

Feature: Adding numbers
  Scenario: Calculator should add two numbers
    Given n1 is 3
    And n2 is 4
    When user performs add operation
    Then the outcome result should be 7

and the step definitions:

@CucumberContextConfiguration
@SpringBootTest(classes = CukedoctorShowcaseApplication.class)
public class StepDefinitions {

  @Autowired
  CalculatorService calculatorService;

  private BigInteger n1;
  private BigInteger n2;
  private BigInteger result;

  @Given("n1 is {}")
  public void n1Is(BigInteger number) {
    n1 = number;
  }

  @Given("n2 is {}")
  public void n2Is(BigInteger number) {
    n2 = number;
  }

  @When("user performs add operation")
  public void userPerformsAddOperation() {
    result = calculatorService.add(n1, n2);
  }

  @Then("the outcome result should be {}")
  public void theOutcomeResultShouldBe(BigInteger number) {
    assertThat(result).isEqualByComparingTo(number);
  }
}

With that we already have working tests in place. Here's the code for the Calculator itself (nothing fancy as you can see):

@Service
public class CalculatorService {
  public BigInteger add(BigInteger n1, BigInteger n2) {
    return n1.add(n2);
  }
}

This gave us following pdf result (when run mvn clean package): Cukedoctor rendered pdf content

As you can see, it already looks nice, but the whole point up until now was to being able to add some context to the BDD tests. Let's do that now. Below you can find reworked .feature file with some additional asciidoc markups:

Feature: Adding numbers
  Scenario: Calculator should add two numbers
  This scenario checks that the result of the add operation is correct. We have defined two variables *n1* and *n2* with respective values *3* and *4*. So the final result of adding should be *3 + 4 = 7*
    Given n1 is 3
    And n2 is 4
    When user performs add operation
    Then the outcome result should be 7

Notice the highlighted line, I've added: This scenario checks that (...). Now let's see what the Cukedoctor results are: Cukedoctor rendered pdf documents with processed asciidoc content

As you can clearly see, the asciidoc content was processed and is now included into the resulting pdf.

And that's pretty much it! Of course I didn't cover most of the cool features of the Cukedoctor (related mostly to Asciidoc features) - I'll leave it to you to discover. Happy testing!