We broke an Over-The-Air update on the ESP32 on purpose

We broke an Over-The-Air update on the ESP32 on purpose groundrun.io

14 points•adunk•11 days ago•7 comments•

7 comments

What an empty article. Testing an OTA update can survive a power interruption during the downloading is such a basic test. It doesn't need an article that long. I guess AI wrote it from a few sentence prompt?

They didn't do any interesting tests. What about brownouts, voltage ripples, timing the reset signal to see if there's any critical moments in the update process, corrupted update files, high EMC environment etc etc. That would be interesting.

I think the article is an advert for a testing platform that enables automating this sort of test. But weirdly they don't show how their platform does the automating, so it just looks like they're really pleased with themselves for doing really basic engineering.

rckoepke9 days ago

Indeed. Would be nice to have a rock-solid “correct” example to critique and learn from. And a write-up from an expert which goes into detail about all the traps for new players, what failure states / eFuse settings could be truly unrecoverable even if you have remote control of the MCU’s PWR / COM / RST pins, etc.

Years ago I had to design some robust OTA systems for both ATmega’s and ESP32’s. I was very confident that my ATmega solution could recover from absolutely any failure state that wasn’t a true “not my fault” hardware failure. At the time though, I was never quite sure if I properly covered every last edge case of the ESP32, which has significantly higher complexity of things that can go wrong from a bad OTA firmware update.

pseudohadamard8 days ago

> Testing an OTA update can survive a power interruption during the downloading is such a basic test. It doesn't need an article that long.

Tell that to the developers of containerloads of IoS (Internet of Shit) devices that can barely manage an OTA update under perfect conditions without bricking themselves.

While you're at it, also let a well-known company that I'll leave unnamed know for their Windows Update service.

rurban9 days ago

You need just enough flash ram for 2 firmwares and a proper bootloader, which knows which partition is active/verified. Simple as that. Biggest problem is getting enough flash. writing the bootloader is trivial.

pseudohadamard8 days ago

+1 to the above. And while a works-under-every-failure-case bootloader isn't quite that trivial, it's still absolutely nothing compared to telling the hardware guys that you want twice as much flash as the SoC they're using can manage.

mikewarot8 days ago

Personal anecdote: It was circa 1985 that I encountered a similar problem. I hand a hand held barcode scanner that had to upload the collected data via a proprietary serial protocol, for eventual reporting, to a PC.

The customer asked "what happens if I disconnect this right now?" during the upload. It was a corner case I hadn't considered. It only took a few days to make it bulletproof.

---

The authors don't seem to consider the case of very slow or intermittent Internet.

I'd be worried about corruption in the download, and check for that. I'd also make very sure the watchdog hardware was on and the code respected it.

It seems to me you need 3 buffers for the OTA code, not 2. There should be a way to keep the last version that runs for X seconds in addition to any new updates, where X is quite large, perhaps a whole day.

ec1096859 days ago

I was expecting something much more rigorous than a few trials resetting the controller at random points in its update process.

Read the full thread on Hacker News →

7 comments

Related stories