Stephen Starkie with Peter Cheng writes:
First, I’ll give you some background; some of which is probably familiar.
At eg technology, we design and build medical devices. The software process, particularly, has many regulatory hurdles and we must comply with industry standards and various countries’ laws. We try to keep our processes as light as possible, so that our Engineers can concentrate on the difficult core tasks, whilst also conforming to best practice. We try to comply with the standards naturally and as part of our culture rather than by bolting it on at the end (which is strictly wrong anyway, according to IEC 62304). We also avoid make-work; we try to produce documents as a core part of our design process, rather than write-only tomes to tick a box.
The biggest overriding concern is, of course, safety, and depending on the classification of the device we are building, this motivates our architecture and design from the outset. As our job is hard enough, we want our tools, frameworks and library code to do as much for us as possible and we have built up a body of well tested code to help us on every project. Because safety is such a concern, traditionally we have tended to write using bare metal or at least low-level vendor-supplied driver and Hardware Abstraction Level (HAL) code. We do this to avoid SOUP (Software Of Unknown Provenance) and also so that we have as little operating system in the way, as possible. This is both good and bad; we know exactly what our code is doing, but we also have to write support for things an OS would provide, ourselves. On some projects, we have used Real-time Operating Systems (RTOS) but, particularly for class C software, we try to avoid them. We use C++ so that we can structure things well and we always work from a well thought out architecture and workflow design – usually a carefully analysed set of Finite State Machines using an asynchronous call-back pattern developed with round-robin cooperative ‘threading’ (we don’t actually tend to use real or particularly pre-emptive threads at the embedded level).
Quite often this approach can be quite restrictive and we all miss the high-level operating systems features that can be so helpful; file systems particularly! So, for some time we have been looking at various RTOSs for projects with a less restrictive classification. We have looked at SafeRTOS (the certified version of FreeRTOS), ThreadX, Mbed OS, VxWorks, CMSIS, RTX and others. Mbed OS has stood out for us for some time as fitting with our way of working. It has a C++ interface that feels very natural to us and for some time I have been sending BBC micro:bit discover kits to graduate entry joiners and asked them to port the various examples in the tutorials to C++ using Mbed. We have never quite fully taken the plunge – partly because we have a good way of working already and partly because various questions held us back that only a ‘proper’ project would answer. Recently, a ‘proper’ project arose that allowed us to get stuck in.
A “Proper” project
For the last two years we have been working on a project with a Cambridge-based start-up which is a “perfect” example of an IoT embedded medical device, for home and clinical use. It takes a measurement over a period of time and posts the results to a web-service where it applies Artificial Intelligence techniques to diagnose conditions and report to clinicians and users via a Web interface. The product will be the focus of a future case study as soon as the client goes to market, but for now it’s under wraps. We originally wrote the firmware using low-level drivers from ST on an STM32F746 (we needed the CPU power for the on-board data processing) connected to a Telit GE-866 QUAD (a quad band 2.5G GPRS modem module). We chose GPRS because the 5G technologies had not fully matured when we started and because GPRS is nearly ubiquitous. However more recently 2G networks (particularly in the US) have been shut down as 5G takes off, so we are looking to move to a dual SIM Telit ME-310 to support LTE Cat M1/NB2. The Telit modems have worked well, but we have suffered from firmware version problems. Support for certain advertised AT commands, particularly when using TLS have been broken in earlier versions. We had to write our own extensive support for AT commands for these modems which felt very coupled and something that could have been solved better by an OS with driver support. We also had to write our own Flash memory support which is much more in-depth a problem than initially thought. On top of that, there was also a requirement to support over-the-air firmware upgrades.
The client was heavily involved all the way through, and also had quite advanced technical understanding. This meant that the stars aligned to port our bare metal code to Mbed OS and use the Pelion system and embedded client to manage over-the-air firmware upgrades. We jumped at the opportunity!
Custom Target Configuration
First of all, there were some slight teething issues – the basic tutorial project (Blinky, of course!) doesn’t include the mbed_app.json file, that all projects really need and that provides all the essential configuration options – so it is quite a good idea to move on quickly to one of the others that does! We needed to work with a custom target because it is a custom PCB. We found that while the documentation is extremely good for most of Mbed, the custom target support is not as detailed and it took a bit of trial and error to get this right – we started from the closest supported development kit (the DISCO-F746NG) and modified it to fit the custom PCB.
Debugging on STM32F746NG didn't work due to an incompatibility with PyOCD. This was fixed in PyOCD for the STM32F767ZI, which we tested on a dev board and it worked well, but unfortunately it didn't get fixed for our device. So, debugging and analysis had to be done via inserting trace output into the application and Mbed OS code, and hard-faults had to be located via map files and a bit of reverse engineering of the binary (https://ghidra-sre.org/). We did find that debugging on the development kit worked very well indeed – in fact better than quite a few other popular embedded toolchains.
MBED STUDIO – MY EXPERIENCE
One of the great features of Mbed Studio is that it is cross-platform. As we were in lockdown and working from home for much of the project, we didn’t always have access to our work PCs except by Remote Desktop (now everyone is on laptops, but in the beginning at least, you just couldn’t buy them!). So, we took advantage of the cross-platform IDE and wrote on Linux Mint, MacOS and Windows. This worked quite well, though Intellisense didn't work out of the box on Linux Mint (favoured by one our engineers). This was because it is only officially supported on a couple of specific Linux distributions, and so the installer omits clangd and associated tools during installation. As Linux Mint is close enough to Ubuntu, we worked around this by tricking the installer. Admittedly, this is a fairly harsh requirement to throw at the tool – Ubuntu is more usual.
The Mbed Studio search tool to find-in-files was useful for searching the Mbed OS source code and can be toggled to search through ignored files.
Serial Comms
Once we had sorted out tools, we dived into the frameworks; these are very well documented and written almost exactly how we would, so it was instantly familiar to us. There were some minor problems, for instance the BufferedSerial class doesn't appear to have a method for determining how many bytes there are in the buffer, available for reading. The BufferedSerial::readable() method indicates if data is actually available on the UART, not in the buffer. The BufferedSerial::poll() method will indicate that there is data in the buffer waiting to be read, but not how much, but this was only determined by reading the source code. We quickly got used to delving into the source code rather than treating it as a black box and the tools and setup make this easy to do.
To add to this, one of the great things about Mbed is the community – issues like this are often picked up very quickly and the Mbed team and the community work together on pull requests to fix them.
Networking
Although Mbed OS contains TCP and TLS socket implementations, it doesn't include a HTTP library, which was essential to the project. However, there is an Arm community one available (https://os.mbed.com/teams/sandbox/code/mbed-http/) which we could use. It was developed for Mbed OS 5, and required some additional tweaks to get working with Mbed OS 6. Again, this was possible due to the source code being available.
As I mentioned earlier, we use asynchronous code and call-backs throughout our own libraries. Unfortunately, the HTTP library implementation blocked on calls so instead we introduced additional threads (which we had avoided in our bare metal code).
One particularly nice feature of building on Mbed OS, is that the process of connecting via HTTPS can be split into two separate layers: the network interface and the HTTPS layer. We built and tested the HTTPS layer on a separate development board that had an ethernet connection, which helped with rapid turnaround and debugging. Then we transferred to our custom board which just needed the GE866-QUAD modem driver in order to provide the required network interface.
Modems, Comms and PPP
Working with the GE-866 had been a little painful with pure AT commands, particularly when earlier versions of the module lacked TLS 1.2 support. We had to build a whole suite of AT command handlers and while it worked, there was a lot of code that felt like it shouldn’t be our problem. With Mbed, it isn’t; the modem is managed in PPP mode seamlessly. For the GE866-QUAD, even the transition into PPP mode worked out of the box, as it could be done from standard AT commands that Mbed OS issued. The only difficulty encountered was that the GE866-QUAD uses PAP authentication by default, but Mbed OS uses CHAP authentication by default. Although we could edit the Mbed OS source to make it use PAP (the setting to do so hadn't been made available as a configuration option) I didn't need to, once I found that the GE866-QUAD could be set to use CHAP authentication via an additional AT command.
Mbed OS provides a TLS implementation that far surpassed the implementation built into the GE866-QUAD. It is also configurable, if you need to optimise for memory size for example. A big help here was that the toolchain produces compiler warnings, if it has been configured insecurely. For example, the defaults use a fake entropy source for random number generation, but until you configure a sufficiently good entropy source it produces a compiler warning. In our case, the STM32F746NG contains a hardware TRNG peripheral, and it was particularly nice that all we had to do to get the TLS library to use that was insert a single line into the custom_target.json file.
QSPI Flash support and File Systems
I mentioned earlier that one of the OS features we miss is good filesystem support. We had written our own QSPI Flash management support and we wanted to replace this with the Mbed LittleFileSystem support. To do this, we needed the Mbed QSPI driver to work for our custom board and things got a little difficult. Oddly, rather than either fully working or not working at all, this worked a little bit. The driver would fail to initialise on first run after programming the device, but succeed every time after that. But then, mounting a LittleFileSystem on the block device initially worked, but eventually would consistently fail to work. The Mbed OS QSPI Block Device driver obviously was not fully compatible with our QSPI flash device. This was easily resolved by wrapping our existing STM32 HAL-based QSPI driver into a class that implemented Mbed OS BlockDevice interface. Once we had done that, the LittleFS worked straight away, though once again the calls were blocking so we added threading.
Over-the-Air Firmware Updates
For the client, one of the biggest reasons for moving to Mbed was to gain the firmware update abilities of the Pelion client and platform. This is an area that originally, we had planned to do ourselves – but it’s quite an in-depth problem and relying on a ‘tried and tested’ solution is definitely a good idea. There's a lot going on under the hood when using Pelion for firmware updates and as such, the documentation is quite large. You end up jumping around it a lot to try to understand how it fits together. Fortunately there is example code for the DISCO-F746NG board, which we used as a basis for our device; otherwise it might have been a lot harder to get it going. Pelion does allow for delta updates, which is a boon for over-the-air updates in a cellular-based system.
The tutorials focus mainly on using the development tools to get it working for a developer role, but for a production environment, extra work is required to ensure production certificates and production provisioning are used.
Little Niggles
There were some other minor considerations to be aware of, for instance; in Mbed OS 6, the minimal printf is used by default. Unfortunately this doesn't support padding (e.g. printf("d", 1) will output "1" instead of "01"). This is because the minimal-printf() reduces memory footprint at the cost of functionality. Alternatively ,the slightly larger standard printf() library can be reinstated, as described here. Additionally, the standard gmtime() function does not work in Mbed OS 6. It returns incorrect values. Perhaps, this is because it is inherently not thread-safe because it uses global shared memory. Unfortunately, gmtime_r() is not supported on our platform. However, the _rtc_localtime() function declared in mbedos/platform/mbed_mktime.h looks to be performing the same functionality as gmtime(), i.e. it still returns UTC, so we can use that instead. It's also worth noting that it says 'For use by the HAL only.', which means that these are not officially supported APIs, so I guess there's a risk that the implementation could change in the future to return the local rather than UTC time, but for now it does what we want.
Driver support
Mbed does also includes some other useful drivers and wrappers that we would have implemented ourselves such as basic usage of the Hardware Watchdog, retrieving the Reset Reason, and using the RTC. This worked out of the box for the STM32F746NG. It also has a driver for setting the MPU in the Cortex-M device.
Another reason we wanted to move to Mbed was for the module support that will come from being in such a community. We developed a driver for the GE-866, but right now we are waiting for PCBs with the ME-310 and expect them to work with very few changes, as the drivers are already in the Mbed source.
Would we do it again?
Overall, the experience has been quite a pleasant one – Mbed fits the way we work very well. It has good support for a lot of embedded microprocessors and plenty of development kits, so that you can get going quickly. There were some teething problems with the tools, but these have been overcome and compared with other popular systems the tools are very nice to use. We achieved all the goals we set out to do and have a code base that we are happy with. We are very likely to use Mbed on future projects where possible.
If you would like to hear more about our experience, I recently joined the team at ARM for their Mbed OS Tech Forum, an hour long YouTube live stream. Watch the recording here.
Our thanks go to Peter Ferguson, Matthew Ockerse and Teemu Takaluoma from Arm for their support throughout this project.
For more information on our software process or to chat with one of our team about your product design and development requirements, please contact us:
via email on design@egtechnology.co.uk
by giving us a call on +44 01223 813184
or by clicking here