BLOG
Hardware Is Hard: Firmware Unit Testing Makes it Easier
One issue MistyWest often sees hardware startups struggle with is firmware reliability. It’s common to develop the firmware, do some functional testing, and ship it – but if you’re developing a product that has a high maintenance cost or strict regulatory requirements, such as sensors in an underground mine or medical diagnostic tools, it’s critical to ensure that its firmware is fully functional and robust before sending it to production.
When MistyWest developed firmware for a wearable pulse oximeter for use in clinical trials, our implementation of unit testing verified that all code met requirements and quality standards before being deployed on hardware. It allowed our developers to consider edge cases and potential failure points, so we could avoid scrambling to fix major bugs at the last minute – and enjoy a smoother and more predictable project timeline.
In this article, we’re sharing how and when you should consider applying it to your next design project.
No hardware? No problem!
Unit testing is more common in software development than firmware due to firmware’s reliance on hardware. However, if your code is written to abstract the logic from the physical layer, you can mock the hardware so that the behavior of the code can be tested – eliminating the need for hardware for preliminary testing. Firmware, when abstracted from hardware, is “just code” and can be tested in the same manner as any software program.
Unit testing’s approach to designing for testability and scalability confirms that the firmware is modular and well-structured. By ensuring that individual components of the software are thoroughly tested in isolation, you can fix bugs sooner.
Source: Dzone
The design of MistWest’s firmware application strongly influences the testability of our code. By creating smaller, more specialized modules, our team can set up and write unit tests more quickly. This practice means that each new release requested by our clients can have bugs fixed and/or features added sooner by having existing tests to detect breaking changes.
Choosing the right framework for the project
The right framework will depend on the specifics of your firmware application, the operating system in use, and your development environment. MistyWest’s engineers have learned that being proficient with multiple frameworks enables us to match the level of testing with the level of risk acceptable to the product’s end user.
MistyWest recommends the following software tools and libraries:
GMOCK/GTEST
These frameworks are excellent for FreeRTOS-based firmware applications. GTest provides a robust testing environment, while GMock allows for the creation of mock objects to simulate hardware interactions and isolate components for thorough testing.
ZTest/FFF
For ZephyrOS-based applications, ZTest provides a seamless unit testing framework specifically designed for the Zephyr RTOS. The FFF (Fake Function Framework) works well for mocking functions and enabling isolated testing.
Ceedling/Unity/CMock
CppUTest/CppUMock
Ideal for C++ projects, CppUTest and CppUMock offer mocking libraries that can be used to simulate physical layers and test the logic part of the firmware. They are particularly useful if you need to maintain cross-platform compatibility.
Running Tests on Target Hardware vs. Host Machine
When it comes to firmware unit testing, one of the key decisions you need to make is whether to run your tests on the target hardware or a host machine. Each approach has its advantages and challenges, and choosing the right one can significantly impact your development process.
Running Tests on Target Hardware vs. Host Machine
Running Tests on the Target Hardware
Running tests directly on the target hardware provides the most accurate representation of how the firmware will perform in the real world. This method verifies that all hardware-specific interactions and timing issues are correctly handled.
Advantages
- Realistic Environment: Tests run in the actual operating environment of the firmware, capturing hardware-specific bugs.
- Accurate Timing: You can identify and address timing-related issues that might not appear on a host machine.
Disadvantages
- Resource Limitations: Target hardware often has limited resources, which can constrain the number and complexity of tests you can run.
- Longer Setup Times: Setting up and maintaining a testing environment on the target hardware can be time-consuming, as it may need to be developed first
Recommended Tools/Frameworks
- Ceedling: Works well with embedded systems and can be integrated with hardware-specific testing frameworks.
- OpenOCD: Useful for debugging and testing on target hardware with a JTAG interface.
- Unity: Can be used on the target with minimal overhead, providing a lightweight testing framework.
Running Tests on a Host Machine
Running tests on a host machine (such as a PC) allows for faster test execution and easier debugging, though it might not catch all hardware-specific issues.
Advantages
- Speed: Tests run significantly faster on a host machine, allowing for rapid iteration and feedback.
- Convenience: Easier setup and maintenance, with access to powerful debugging tools and frameworks.
- Scalability: You can run a larger number of tests and more complex test scenarios due to the host machine’s greater resources.
Disadvantages
- Environmental Differences: Tests might not fully replicate the behavior on the target hardware, potentially missing hardware-specific issues.
- Simulation Limitations: Some hardware interactions might be difficult to simulate accurately on a host machine.
Recommended Tools/Frameworks
- Ceedling with CMock: Excellent for running tests on a host machine, with CMock providing the ability to mock hardware interactions.
- QEMU: An emulator that can simulate the target hardware environment on the host machine, useful for more comprehensive testing.
- GoogleTest: A robust and flexible framework that can be used for testing C++ firmware on a host machine. With minor modifications, Google Test can be used to test firmware coded in C as well.
So, how do you make the right choice? This will depend on your specific needs and constraints. Here are some guidelines to help you decide:
- If your firmware involves extensive interactions with external devices and relies heavily on hardware-dependent logic rather than high-level application code, prioritize running tests on the target hardware.
- For faster development cycles and easier debugging, start with testing on a host machine and gradually move to target hardware for final validation.
- Wild card: a hybrid approach often works best. Start by developing and testing on a host machine for speed and convenience, then validate critical functionalities on the target hardware.
Using strategic decision-making around framework selection and understanding the trade-offs will help you optimize your firmware unit testing process and maintain an efficient development workflow.
Firmware unit testing reduces risk of in-field failure
Consider firmware unit testing your product if you know that patching it after deployment poses serious risks to its performance or end users.
Creating a comprehensive set of tests that covers the code base and evaluates it under both expected and unexpected conditions ensures that the firmware a) functions as intended and b) fails gracefully without affecting other parts of the system. Early validation of key features through unit testing builds confidence that the final product will perform as expected, reducing the likelihood of costly in-field failures.
The consequences of a firmware error in high-stakes industries can be severe. In mining, corrupted sensor data due to a firmware error could cause the system to respond incorrectly, leading to tens or hundreds of millions of dollars in lost output or physical harm to a miner. In a medical device, such errors could result in misdiagnosis or injury to patients.
Save this infographic for the next time you’re deciding if unit testing is right for your project
Hardware doesn’t have to be hard
It may appear at first that firmware unit testing will slow down product development, which can be a concern when you have tight timeline commitments to customers and investors. However, in industries where the stakes are high for hardware devices, it’s faster to have more testing up front to ensure you identify those buggy issues. Doing this will ultimately increase the speed of your product development, getting those de-bugged devices into the hands of users sooner.
For our client developing the pulse oximetry wearable, MistyWest’s rigorous unit testing practice has helped contribute to long-term maintainability. The test suite we created at the start of the development has served as a supplementary form of documentation that indicates how each component is intended to function. This enabled our client to continue developing in a manner that is consistent with the existing codebase – long after MistyWest’s work is done.
Developing a scalable and maintainable firmware codebase contributes to long-term project success by minimizing risk of in-field failures, increasing product life cycles, and ensuring that the firmware is not only functional at launch, but stays reliable over time.