mittos64/doc/5_Unit_Testing.md

Unit Testing Framework

In this chapter we'll build a unit testing framework that will help us developing the rest of the kernel.

Why build it yourself?

As far as I understand it, there's really no good arguments for building your own unit testing framework. There are lots of finished alternatives available allready, and using them will let you focus on your main project instead.

I'll write my own, simply because I feel like it. If you don't, then use an existing one.

Requirements

I don't have very many requirements for the testing framework. But it's important that:

• It runs fast
• It's clear what failed
• Tests are isolated so that they don't influence each other

I also want the tests to be "close" to the code. I was thinking of including them in the actual source file, but instead opted to put them in a separate file with the same name, but with a .tt extension. (tt for Thomas-Test).

Designing the framework

The framework will be in two parts; a c source file to include in each test, and a shell script to run all tests.

Let's start with the shell script.

It should scan the supplied directories for .tt files, compile them and run them:

ttest

#!/bin/sh

dirs = src/kernel

main()
{
  for dir in $dirs; do
    local files=`find $dir -name "*.tt"
    for suite in $files; do
      test_exec="${suite}"est
      cc -x c $suite.c -o $test_exec -I $dir/include -I toolchain -DTTEST`
      $test_exec
      rm $suite.c $test_exec
    done
  done
}

$test_exec contains the name of the compiled test executable. It will get the extension .ttest. Since gcc doesn't know how to compile .tt files we tell it that it contains c code with the -x switch.

Now let's take a look at the c source file (well... it's really a header file acting like a c source file...)

This should contain the main() function of the test which should run each test in turn and keep track of any failures. But first, we have a struct to keep track of everything related to a test.

toolchain/ttest.h

...
struct tt_test
{
  char *name;
  int(*test)(void);
  int status;
  char *output;
};

struct tt_test *tt_tests;
struct tt_test *tt_current;
int tt_test_count = 0;
...

Next for the main function. For insolation, each test is forked off and run as a separate process. Errors are passed back to the main process through a pipe.

toolchain/ttest.h

...
int tt_pipe[2];

int main(int argc, char **argv)
{
  for(int i = 0; i < tt_test_count; i++)
  {
    tt_current = &tt_tests[i];

    fflush(stdout);
    pipe(tt_pipe);

    int pid;
    if(!(pid = fork()))
    {
      close(tt_pipe[0]); // Close read end of pipe
      exit(tt_current->test());
    }

    close(tt_pipe[1]); // Close write end of pipe
    read(tt_pipe[0], tt_current->output, TT_BUFFER_SIZE);
    close(tt_pipe[0]);

    int status;
    waitpid(pid, &status, 0);
    if((tt_current->status = WEXITSTATUS(status)))
    {
      printf("F");
    } else {
      printf(".");
    }
  }

  for(int i = 0; i < tt_test_cound; i++)
  {
    if(tt_tests[i].status)
      printf("%s: %s\n", tt_tests[i].name, tt_tests[i].output)
  }
}

This is, of course severey simplified. I also have a lot of prettification, keeping track of the number of failures, checking if the test crashed before finishing and such...

Ok, so how is the tt_tests array populated?

Well. There's a macro to define each test in the .tt-file:

toolchain/ttest.h

...
#define TEST(name) \
  int ttt_##name(); \
  __attribute__((constructor)) void tttr_##name() { \
    tt_register(#name, ttt_##name); \
  } \
    int ttt_#name()

This looks... weird... What does it do? Let's try expanding it.

I'll tell you right away that it's supposed to be used for a function definition, so using it like:

TEST(adder_adds_two_numbers)
{
  // Test goes here
}

will expand to

int ttt_adder_adds_two_numbers();
__attribute__((constructor)) void tttr_adder_adds_two_numbers()
{
  tt_register("adder_adds_two_numbers", ttt_adder_adds_two_numbers);
}
int ttt_adder_adds_two_numbers()
{
  // Test goes here
}

So it declares, and then defines a function. But what's the stuff in between?

It's magic - that's what.

The __attribute__((constructor)) is a gcc specific attribute which tells the compiler that the function tttr_adder_adds_two_numbers should run as soon as it gets into scope - which is when the test loads and, most importantly, before main() is called.

So tt_register just has to set up an entry into the tt_tests array, and the rest will take care of itself:

toolchain/ttest.h

...
void tt_register(char *name, int (*fn)(void))
{
  tt_tests = realloc(tt_tests, (tt_test_count+1)*sizeof(struct tt_test));

  struct tt_test *t = &tt_tests[tt_test_count++];
  t->name = name;
  t->test = fn
  t->status = 1
  t->output = malloc(TT_BUFFER_SIZE)
}
...

The final part needed for a test is an assertion, which is yet another macro:

toolchain/ttest.h

...

#define ASSERT_EQUAL(lhs, rhs) do { \
  int tt_lhs = (int)(lhs); \
  int tt_rhs = (int)(rhs); \
  if(tt_lhs != tt_rhs) { \
    dprintf(tt_pipe[1], "Got <%d> but expected <%d>\n", tt_lhs, tt_rhs); \
    return 1; \
  } \
} while(0);
...

The arguments of the assertion macro are copied immediately. That way they are only evaluated once.

And that's it!

Using the framework

A typical source file with tests can look like this

adder.c

int adder(int a, int b)
{
  return 5;
}

adder.tt

#include <ttest.h>
#inclued "adder.c"

TEST(adder_adds_two_numbers)
{
  ASSERT_EQUAL(adder(2,3), 5);
}

TEST(adder_adds_two_other_numbers)
{
  ASSERT_EQUAL(adder(3,4), 7);
}

And running it can look like this

$ d ./ttest
.F
adder_adds_two_other_numbers: Got <5> but expected <7>

If you have a c compiler installed you should also be able to run the test framework locally. This speeds it up a bit.

Further improvements

Of course this is only a very rudimentary test framework, and a lot of improvements can be made. But this series isn't about designing a test framework, and I don't know anything about designing a test framework anyway, so I'll just leave it here.

I will however note some differences between this version and what's actually in my code:

• The ASSERT_EQUAL macro has a different signature and should in fact not be used directly.
• Instead use
  • ASSERT_EQ_INT(lhs, rhs) for integer comparison
  • ASSERT_EQ_CHR(lhs, rhs) for char comparison
  • ASSERT_EQ_STR(lhs, rhs, len) for string comparison
• There's also ASSERT_NEQ_INT and ASSERT_NEQ_CHR for not-equal assertions
• Macros BEFORE() and AFTER() define functions that are run before and after each test respectively. Use them for setup and teardown of the objects under test
• The printed errors contain the name of the test file and the faulting line in a format that can be processed automatically by vim and used in the quickfix list
• Output to terminal is colorized
• The output is colorized
• The shell script handles compilation errors of the tests

If you run the tests locally, everything should work if your c compiler is gcc.