Unit Testing Embedded C: On-Target with minunit and Off-Target with MS Test

Generally, the advice on unit testing in embedded environments is to run your tests on the PC host rather than on the target device. Whilst I agree that this is the most productive arrangement, there are a variety of reasons for needing to test on the target which can be convincing in certain situation. The technique described here allows for both.

Mike Long in his GOTO 2015 presentation Continuous Delivery for Embedded Systems says

"Test on your host because that's fast - it's a really fast way to develop. But also test on the target because behaviour can change in different ways, different compilers, different hardware..."

Niall Cooling in his talk at the EmbeddedOnlineConference 2020 "How agile is changing the face of embedded software development" says (at 46m) on the gap between testing on the host and the target

Things like TDD really are based on testing in the host, and really that's fine but of course we are typically using host compilers like host GCC and of course we know that at the moment this is typically going to be an Intel based processor. So we are compiling for the underlying OS. And it is good for finding a lot of functional issues at a certain level above the platform. But we know there are certain problems that cannot be addressed on the host within that. If we jump straight to target testing, it is quite a big gap in terms of levels, first is that we are using cross compilers. We've changed our compiler model straight away but also we've changed the compiler itself, from GCC on the host to Keil or IAR for the target, so completely different environment, and using things like RTOS.

First let's look at the pros and cons of testing on the host and testing on the target.

FactorHostTarget
Build & Run time<5 sec20-40 sec including programming
Code size restrictionunlimited(Target code space) - (application code space)
RAM size restrictionnot usually a problem(Target RAM size) - (application RAM usage)
Embedded specific macros, librariesNeed to separate logic functions from driver layers, add mocksnot a problem
Int size, packing, endiannesshaving to run on host forces portability issues early onnone if not worried about portability
Real timedifficultruns at full product speed, gives timing info
Hardware interactionnoyes

If you are trying to decide which is better so that you can only implement that method then I would advise -

This is one of those cases where the best thing to do is both. Implement as many tests as possible using the host framework where you have the space and execution time. But also run some tests on target, even if some of them duplicate the code coverage of the host test. And set up an on-target build which just tests your HAL and driver layers.

Don't forget that unit testing is for functional requirements, the logical behaviour of the code - given these inputs what are the outputs. It does not intend to test, and cannot be expected to test, the non-functional aspects of code, for example

  • timing and performance eg throughput, response time
  • security, data integrity
  • design issues like maintainability, stability, portability, testability, extensibility, documentation
  • reliability
  • cost, initial and life cycle
  • quality in terms of number of bugs
  • code and RAM space

But my code is structured so that I need to test on the target

Then your code is wrongly structured.

Matt Chernosky has a great article, For embedded TDD, don't worry so much about testing on the target where he explains that you should design your application so that hardware accesses are abstracted out to a specific layer so it can be mocked; that you don't need interrupts active to test the ISR code, and that RTOS calls can be abstracted out. Or you can take the approach of testing individual functions or modules without having to set up the whole application and its main loop.

Choosing a Unit Test Framework

There are a lot of unit test frameworks for C, Wikipedia lists 58 of them. But I have the requirement that the framework is supported on Windows because that is where the embedded IDE and tools run, I don't know Linux and am not about to learn it just for this purpose.

I decided to use Visual Studio C++ 2019 for code editing since it is widely regarded as one of the best IDEs and is so popular that there is an enormous number of helpful resources should you run into any problem. Although it does not initially appear to support C code, being a C++ tool, there is a way to use it with pure C files.

Create a new C++ project and when you add a new source file, use a file name with a .c extension (instead of .cpp). The compiler treats all files that end in .c as C source files. However, MSVC whilst being compatible with ISO C99 is not strictly compliant, but most portable C will compile and run as expected. Additionally you can add compiler flags /std:c11 or /std:c17 for those ISO versions.

Despite using a C++ IDE, I wanted to avoid getting involved with any C++ code as much as possible since it is an unknown area for many embedded C programmers. Microsoft includes their unit test framework which is easy to use, MS Unit Testing Framework for C++. Whilst the tests are written in C++, you can get away with knowing only a few simple constructs such as Assert::AreEqual(expected, actual); there is also Asserts for Are Same, Is Null; Not versions of those, and Is True. To write message to the Output Window, use the Logger class such as Logger::WriteMessage("In Module Initialize");

You don't have to run the full set of tests every time, there is support for custom test playlists. Unfortunately, code coverage analysis is only available in Visual Studio Enterprise edition.

Running Tests on the Host with MS Unit Test

It is easy to setup a Visual Studio C++ project to run tests as described by Microsoft. Assuming you have separated out your target specific layers (anything hardware dependent such as Board Support Package or RTOS) from your application, the resulting standalone C project can be compiled with Visual Studio and then you can add a test project.

  • Solution - Add New Project - Native Unit Test Project
  • Solution - Add Reference - your application project
  • In Unit Test .cpp file, add #include headers and C source files that you are testing, of your application under test 
  • Write tests inside TEST_METHODs and TEST_CLASSes in your unit test .cpp file.

Each test sets up conditions, calls the function under test then asserts that the results are as expected. Here's a very simple example, testing my ctof() function for one input value.

TEST_METHOD(BoilingPt)
        {
            uint8_t c, f;
            //boiling point of water
            c = 100;
            f = 212;
            Assert::AreEqual((uint32_t)f, (uint32_t)ctof(c));
        };

Run All Tests CTRL+R, A (release CTRL before typing A)

MS Test, 18 Unit Tests passing

All the tests should be very fast because you are running on your PC's processor unless you are doing a lot of looping or iterations. In the above screenshot, the only test taking more than a few milliseconds is AllNumeratorsAndDivisors which tests divRoundClosest() with all numerators in range [1..255] and all divisors [1..255] so we see 65025 iterations taking 2.4 sec.

When your code produces unexpected results, you will want to set a breakpoint in the function under test or in the test runner to step through it and see the execution flow and the variable values. To do this, you need to run the tests in debug using Debug All Tests, CTRL+R, CTRL+A (hold CTRL down while typing both R then A).

Additionally, you probably want to break on Assert failure. Open the Exception Settings pane using CTRL-ALT-EVisual Studio 2019 Exception Settings
Search for cse. Check the box to Break When Thrown on this exception
Microsoft::VisualStudio::CppUnitTestFramework::CSEException 
Break When Thrown CSEException

Tip: Make sure your project under test is set as the Startup Project (right click project in Solution Explorer - Set as Startup Project) rather than the Unit Test project, so that the Debug - Run works.

Running Tests On the Target or the Host PC with minunit

minunit is the simplest possible unit test system, with only 3 lines of code in a single h file. The source code and example usage are easy to follow, and there are no licence restrictions.

/* file: minunit.h from http://www.jera.com/techinfo/jtns/jtn002.html */
/* A MinUnit test case is just a function that returns 0 (null) if the tests pass. 
If the test fails, the function should return a string describing the failing test. 
mu_assert is simply a macro that returns a string if the expression passed to it is false. 
The mu_runtest macro calls another test case and returns if that test case fails. 
That's all there is to it! */

#define mu_assert(message, test) do { if (!(test)) return message; } while (0)
#define mu_run_test(test) do { char *message = test(); mu_tests_run++; \
                                if (message) return message; } while (0)
extern int mu_tests_run;

In use, main.c contains at least this code:

#define TESTING_WITH_MINUNIT    //include this if you want to run tests with minunit, otherwise that code is not included
#include <stdint.h>

#ifdef TESTING_WITH_MINUNIT
int mu_test_runner(void);
#endif

int main()
{
#ifdef TESTING_WITH_MINUNIT    //run the tests
    int mu_result = mu_test_runner();

    if (mu_result)
        return mu_result;
#endif    
}

All the test code resides in mu_test_main.c which is part of the application project but can be #defined out so that it doesn't take up code space in release builds. Let's have a look at a simple example of one test which logs it's version number and the results as it goes along.

#include <stdint.h>

#include "minunit.h"
#include "ctof.h"    //application include files for function prototypes

//minunit's variable
int mu_tests_run = 0;

#define MAJ 1
#define MIN 00
#define str(s) #s
#define xstr(s) str(s)

#define VERSION_STRING xstr(MAJ) "." xstr(MIN)

//minunit's tests
static char* mu_TestBoilingPt()
{
    uint8_t c, f;
    c = 100;
    f = 212;
    mu_assert("mu_TestBoilingPt Failed, f != 212", ctof(c) == f);
    return 0;
}

//TODO more tests go here

//top level test functions
static char* mu_all_tests()
{
    STD_OUTPUT_LOG(VERSION_STRING "\n");

    STD_OUTPUT_LOG("Running MINUNIT mu_TestBoilingPt...\n");
    mu_run_test(mu_TestBoilingPt);
    
    //TODO call more tests from here
    
    return 0;
}

int mu_test_runner()
{
    STD_OUTPUT_LOG("Starting MINUNIT Tests\n");

    char* result = mu_all_tests();
    if (result)
    {
        STD_OUTPUT_LOG("%s\n", result);
    }
    else
    {
        STD_OUTPUT_LOG("ALL MINUNIT TESTS PASSED\n");
    }
    STD_OUTPUT_LOG("MINUNIT Tests run: %d\n", mu_tests_run);
    return result != 0;
}

printf test reporting 

Test progress and failures are logged to the console using printf, which of course works fine in a host environment as long as stdio.h is included. Running this application within Visual Studio with the Local Windows Debugger (F5) produces

Visual Studio running minunit with 1 successful test

Deeply embedded devices tend not to have a display that supports printf so instead these strings can be printed through a serial port. An example of getting log messages through a serial port (over USB) is the Nordic nRF5 system with Segger where the macro NRF_LOG_INFO is defined. To allow easy use of either routing, I have set up #defines in mu_test_main.c like this:

//Options:
#define STD_OUTPUT_TO_PRINTF            //include this if on host PC platform with printf
//#define STD_OUTPUT_TO_NRF_LOG_INFO        //include this if using SES and Nordic using UART for output messages

#ifdef STD_OUTPUT_TO_PRINTF #include <stdio.h> //for printf #define STD_OUTPUT_LOG printf #endif #ifdef STD_OUTPUT_TO_NRF_LOG_INFO #define STD_OUTPUT_LOG NRF_LOG_INFO #endif

Test Logging using on-chip peripherals

If your micro has an otherwise unused UART, you only need to set it up for transmit and wire the Tx pin and GND to a FTDI RS-232 to USB cable and use a terminal program such as Hyperterminal etc which can be set to record to a file. If you don't have such a cable to hand, you can use the decoding facilities of a digital storage oscilloscope or logic analyser to see the messages go past in real-time. This can be particularly useful combined with the scope's trigger on serial message functionality because it gives you timing information and the ability to capture other signals simultaneously.

Another possibility is to send the log messages out through an unused SPI or I2C port via the appropriate peripheral, and either view them on a DSO with serial decoding facilities or use SPI to USB FTDI cables with Hyperterminal to log them.

As Mike says, you don't want your debug code to stall your code so you don't wait for the UART to be ready between messages - you just throw the data at it. Send data to the UART at a rate compatible with it baud rate and number of bytes in the peripheral buffer. Tune the baud rate to match the frequency that data occurs so that decoded values are readable on your display.

The UART method has the advantage of only using up one pin, but if you don't have a UART peripheral spare then it might not work well because a software-implemented UART will suffer from some bits being stretched by interrupt tasks/high priority processes. Fortunately there are techniques which don't suffer from this problem.

Getting log messages out of your target without any peripherals

If you don't have any UART, SPI or I2C peripherals available on your micro, it is still possible get messages out using just a couple of output pins. SPI and I2C can be done purely in software and these routines do not use timers or call backs so are simple to use but have the disadvantage that they are blocking calls.

Jack Ganssle demonstrates his I2C method on this video, which is coded fairly simply as

//Jack Ganssle's I2C log method from https://youtu.be/VfpLyJvaBmc
//bitbashed I2C send for debugging purposes when no peripheral available
//This does not require a free I2C peripheral - it just requires two GPIO pins in Output mode
//For displaying on a scope or logic analyser with I2C decode
// Ganssle's measurements are:
// 20us per pair {address, data} on Cortex M0+ @ 48MHz
// 11us per pair {address, data} on Cortex M3 @ 96MHz

#include <stdint.h>  
  
//Example of output pin allocation. Set these to Outputs before calling these functions
#define SDA LATB1;
#define SCL LATB2;

//output data polarity
#define HIGH 1
#define LOW 0

//optional blocking delay in case bits are too fast changing for your analyser. Inlined to remove any performance hit if it is not used
inline static void delay(void)
{
}  
  
static void sda_write(uint8_t value)
{
    SDA = value;    //drive the line high or low
    delay();        //if needed, delay to slow the signal to the analyser
}

static void scl_write(uint8_t value)
{
    SCL = value;    //drive the line high or low
    delay();        //if needed, delay to slow the signal to the analyser
}

static void i2c_start(void)
{
    sda_write(HIGH);
    scl_write(HIGH);
    sda_write(LOW);
    scl_write(LOW);
}

static void i2c_stop(void)
{
    scl_write(LOW);
    sda_write(LOW);
    scl_write(HIGH);
    sda_write(HIGH);
}

//Send 8 bits to the I2C port using bit bashing. Includes fake ack from a slave at the end of the byte
//so as to not confuse analyers since there is no slave.
static void i2c_write_byte(uint8_t data)
{
    uint8_t i;
    
    for (i = 0; i < 8; i++)
    {
        if (data & 0x80)
            sda_write(HIGH);
        else
            sda_write(LOW);
        data <<= 1;
        scl_write(HIGH);    //Toggle clock
        scl_write(LOW);
    }
    sda_write(LOW);            //fake ACK from slave
    scl_write(HIGH);
    scl_write(LOW);
}

//Send an address and a string to the I2C port.
//address is 7 bit. LSB will be forced to 0 because that indicates a write
//string is null terminated
//example of usage: i2c_send_string(0xc0, "error 29");
void i2c_send_string(uint8_t address, const char *string)
{
    i2c_start();
    i2c_write_byte(address & 0xFE);        //masks off LSB
    while (*string != 0)
        i2c_write_byte(*string++);
    i2c_stop();    
}

Sending "Test String" from an STM32F4 at 168MHz SYSCLK using this bitbashing I2C method without a delay() implementation takes 12us per character.

Ganssle's I2C method sending "Test String"

Mikeselectricstuff demonstrates his SPI method on this video, which is coded even more simply. I've renamed his version sendspi16and added an 8 bit version which can be sent a string using sendspistr

//mikeselectricstuff's method for SPI Out logging from https://www.youtube.com/watch?v=EdfHzpEKtZQ
//bitbashed SPI send for debugging purposes when no peripherals are available
//This does not require a free SPI peripheral - it just requires two GPIO pins in Output mode
//For displaying on a scope or logic analyser with SPI decode 

//Set your hardware pinout. Need to ensure these bits are set as Outputs before the function runs. This an example from a PIC
#define spi_clk LATB14
#define spi_data LATB13

static void sendspi16 (unsigned int d)
{//sends 16 bit data, MSB first, at rate dependent on micro clock
    unsigned int i;
    
    for (i=0;i<16;i++)
    {
        spi_data = (d & 0x8000) ? 1 : 0;
        spi_clk = 1;
        spi_clk = 0;
        d <<= 1;
    }
    spi_data = 0;    //not necessary but looks neater on scope
}
static void sendspi8 (unsigned char d) {//sends 8 bit data, MSB first, at rate dependent on micro clock unsigned int i; for (i=0;i<8;i++) { spi_data = (d & 0x80) ? 1 : 0; spi_clk = 1; spi_clk = 0; d <<= 1; } spi_data = 0; //not necessary but looks neater on scope }
static void sendspistr(const char *string) { while (*string != 0) sendspi8(*string++); }

Sending "Test String" from an STM32F4 at 168MHz SYSCLK using this bitbashing SPI method takes 7.1us per character.

Mikeselectricstuff SPI method sending "Test String"

Both of these take advantage of the fact that the clock is generated by the master and can be arbitrarily stretched without affecting the ability to decode it as long as the data line is similarly stretched. So an interrupt in the middle of the SPI send loop doesn't affect the decoding ability, which it would if you bit bashed in UART format. Pulse stretching the UART waveform is equivalent to stretching the receiver sampling clock, which I have previously analysed.

In the following screenshot, the yellow trace shows delays caused (deliberately to test this) by timer ISR processing, showing that the T character is not affected by the delay.

Mikeselectricstuff SPI method with deliberate interrupt delays

If there are no spare output pins but there is sufficient non-volatile memory such as EEPROM or NVRAM, that could be used and read back by the programmer, but then you would want to reduce your logs to the bare minimum.

Accessing Module Local Variables

Often the unit test code needs access to variables in the module under test which are local (declared static in C) to determine if the functions are behaving correctly. It can be tempting to just change the scope of those variable to global (remove the static keyword) but either you end up with more globals in production code than necessary or you #define out the static keyword in those modules for production release which is yet another code difference between tested and released versions, and you should eliminate as many of those as possible. 

An idea from Alexopoulos Ilias' talk at the EmbeddedOnlineConference 2020 can be helpful here.

  • Put all the module level variables for that module into one struct (with a typedef in the header)
  • Add a function in that module to return a pointer to the struct
  • This function is global (not static) so that it can be called from outside the module, but is #defined out in production release, along with other testing code
  • Higher level test modules can observe and change values of module level variables using that pointer and with typedef knowledge of the structure

Here's an example of this technique. The module under test is time_calc:

time_calc.h

typedef struct {
    uint8_t hrs;
    uint8_t mins;
    uint8_t secs;
} wall_time_t;

void inc_time(void);
void set_time_hms(uint8_t h, uint8_t m, uint8_t s);

#ifdef TESTBUILD
wall_time_t* testing_get_current_time(void);
#endif

time_calc.c

#include "main.h"
#include "time_calc.h"

//declare instance of local variable of type of global structure
static wall_time_t current_time;

void inc_time(void)
{
    if (++current_time.secs >= 60)
    {
        current_time.secs = 0;
        if (++current_time.mins >= 60)
        {
            current_time.mins = 0;
            if (++current_time.hrs >= 24)
                current_time.hrs = 0;
        }
    }
}

void set_time_hms(uint8_t h, uint8_t m, uint8_t s)
{
    if (h < 24 && m < 60 && s < 60)    //validity check
    {
        current_time.hrs = h;
        current_time.mins = m;
        current_time.secs = s;
    }
    //otherwise leave the old value
}

#ifdef TESTBUILD
//need access to local static structure for testing
wall_time_t* testing_get_current_time(void)
{
    return &current_time;
}

#endif

Unit test code in MS Test C++


  #include "../time_calc.h"
  #include "../time_calc.c"
    
    //...
    
    TEST_METHOD(SetTimeDistinctHMS)
        {
            uint8_t h;
            uint8_t m;
            uint8_t s;
            wall_time_t* test_time;

            h = 1;
            m = 2;
            s = 3;

            set_time_hms(h, m, s);
            test_time = testing_get_current_time();
            Assert::AreEqual((uint32_t)h, (uint32_t)test_time->hrs);
            Assert::AreEqual((uint32_t)m, (uint32_t)test_time->mins);
            Assert::AreEqual((uint32_t)s, (uint32_t)test_time->secs);
        };