Improving the embedded software testing process
From the embedded software development team at Ignys
What is test driven development?
Test driven development (TDD) is the principle of writing tests before writing your core functionality. This forces the developer to adopt a different approach to writing software by:
- Breaking down the project into functional modules or “units”.
- Thinking about the interfaces to each of these units.
- Determining the desired behaviour of these units and how they should handle errors.
- Writing tests that check this behaviour.
- Finally implementing software to implement this behaviour and ensure tests pass.
This encourages developers to write well designed and modular code as this is also necessary to facilitate writing tests for each module. By writing tests before implementation, the developer must consider much more deeply the design of each module, and its API, so that they can write appropriate tests for the module.
Although TDD may seem like it is increasing the work needed, in reality it can save time especially in the longer term. When debugging software, a failing test will highlight quickly where code logic has failed and many testing frameworks facilitate dropping into a debugger at failing code to investigate the problem. This eliminates the need to rerun a large number of steps to replicate buggy behaviour.
Critics of this approach may also point out that you have even more code to debug and maintain, how do you ensure your tests are valid? In practice this is rarely an issue, although it is entirely possible, likely even, that tests could contain bugs or bad logic especially when first drafted. It is likely this buggy behaviour will be quickly identified as the test is most likely to fail rather than coincidentally pass. Furthermore, tests are usually far simpler to implement than the functionality they are testing and therefore less likely to be incorrectly reasoned.
There are many other additional benefits to writing software in this manner:
The tests you write can serve a dual purpose of documenting how the API is intended to be used and effectively serves as example code. This can be an effective way to ensure continued documentation of the codebase and makes it easier for new developers to the codebase to get up to speed quickly.
With a full suite of tests with good code coverage, it becomes much safer to make code changes for new features, refactoring or bug fixes. The test suite will quickly highlight where one of these code changes has resulted in unexpected side-effects and broken the code logic; this form of testing is known as regression testing. If a test does fail, the developer can take steps to modify refactored code and ensure the test passes. Tests become a vital lifeline, giving confidence when releasing refactored code.
Finding edge cases in embedded software testing
Once you have a suite of tests that demonstrate your implementation behaves correctly, the effort to write extra tests to cover additional edge cases becomes minimal. In many instances it’s a case of copying the code of one test and modifying it to check different set of inputs and the expected behaviour. Some testing frameworks streamline this process specifically, by allowing the developer to write parameterised tests further reducing the need for code duplication (and fewer tests to maintain!).
Resolving bugs in embedded software testing
Test driven development also changes the way bugs should be solved. When a bug is reported, the first step should be reproduction of buggy behaviour and in many cases the reproduction steps can be converted into one or more tests!
Once your tests can reliably fail on the buggy behaviour, it becomes the same streamlined process of writing and modifying the code to ensure the test passes.
Although not a guarantee that tests will correctly determine good behaviour of your code, they should greatly improve the overall quality of software produced and maintaining that quality over time.
Though not a comprehensive list, these should give some indication in some of the ways TDD should add value development for all projects.
How do I run test driven development in embedded platforms?
Embedded platforms such as baremetal software on a microcontroller can pose particular challenges when running a project with a TDD approach. The main two issues these are:
- Highly resource constrained platforms, usually having very little RAM, flash and CPU resources – there may not be enough flash and RAM to run a full suite of tests.
- The code has to directly communicate with hardware peripherals – How do I test hardware interfaces without a complex and instrumented rig.
Despite these challenges it is still possible to reap many of the benefits of using test driven development.
Testing on resource constrained environment
You could try to write and setup all of your tests to execute on your microcontroller however in many cases this will not be possible as a full suite of tests could put consume excessive RAM and flash resources. In addition many testing frameworks are simply not designed to run in a baremetal environment often requiring functionality often only found in systems with a full general purpose operating system such as Windows or Linux.
However it is still possible to write the bulk of software using TDD despite these constraints. Assuming you are writing the bulk of your software in a high level language such as C we would instead we would suggest implementing and running all your tests hosted on your development machine. Using a high level language such as C allows us to write code that can be ported between our development machine and target microcontroller, yet still behave identically. Furthermore, this forces the developer to write modular code, specifically separating code that talks to the hardware from the testable application code which we will talk about next.
How do I test hardware interfaces?
In most microcontroller applications your software will need to communicate with various peripherals through I2C, SPI, UART etc. The approach we would suggest is to wrap up software that talks to a specific peripheral interface such as I2C into a singular module, known as a hardware abstraction layer (HAL).
Although it is possible to write unit tests for some of the HAL it is unlikely it will be possible to correctly verify the software’s implementation with tests alone. It is also unlikely you will capture the correct hardware behaviour in all circumstances. HALs instead will typically need to be verified by manually testing and checking it, for example with an oscilloscope. However, once you’re confident the HAL is working, it can, and should be left alone! If further changes are made to your HAL these should be carefully reviewed and tested. However, by properly separating, or decoupling, the functionality of the HAL modules from the rest of the code should allow you to write higher level code knowing there will be significantly less risk of breaking your HAL.
Often writing your own HAL is not necessary, most microcontroller of microcontrollers will offer a library of prewritten HALs for each of the modules that have generally been well tested. However we would always recommend manually testing the HAL modules used in your particular application are behaving as expected; even vendor HALs are not always bug free however the often can save you huge development effort.
By decoupling the HAL from rest of your application code you should be free to write unit tests for your non-HAL code and ‘mock’ the interface to your HAL. Most popular unit testing frameworks have a provision for automatic mock creation, which for example, allow you to check that your code is sending expected data over UART by calling key functions such as UART_Send_Message(“a message”); the framework will verify that the function is called with the correct arguments, often even returning specified data back to the code.
Hopefully this will give an indication of value and ways in which you can add test driven development to your workflow, even in embedded projects!
Article Authorship: Embedded Software Testing
This article was written by Stephen Lynch embedded software engineer. Stephen has a PhD from Southampton and a first-class degree in electronics engineering. He works on a vast variety of software projects.