Open in app

Sign in

Write

Sign in

Python Utilities for ESP32

Ihor Mierkielov
7 min readSep 8, 2021

--

Automatization in embedded systems development requires a lot of software engineer’s attention. And it’s not only about compiling, installing dependencies, setting up a virtual environment, or testing.

Today, we will analyze official Espressif’s Python utilities for communicating with the ROM bootloader, operating partition tables, and working with SPIFFS and firmware.

With their help, you can speed up your development process. So, let's go!

1. esptool

Fig 2.: Just a beauty picture from this resource

This utility is the most powerful and popular one to communicate with ESP8266 & ESP32 platforms.

If you suddenly didn’t know: using this tool you can read/write/remove all binary data; You can specify appropriate flash addresses, baud speed, chip, and other important options; Moreover, you can get some embedded information about your board such as SPI flash size and MAC address!

To get helpful info and positional arguments we simply run:

$ python esptool.py -h

There are a lot of examples in the project folder. And we can erase the flash:

$ python esptool.py erase_flash

Output:

esptool.py v3.2-dev
Found 1 serial ports
Serial port /dev/ttyUSB0Connecting..........................
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision 0)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: xx:xx:xx:xx:xx:xxUploading stub...
Running stub...
Stub running...Erasing flash (this may take a while)...
Chip erase completed successfully in 7.4s
Hard resetting via RTS pin...

But much more interesting, if you familiar with Python, is calling esptool from other Python scripts, thereby providing you with a “gang programmer”, for example, if your application implies minor architectural changes in serial production.

For instance:

command = [ '--baud', '115200', 'read_flash', '0', '0x200000', firmware.bin' ]
print( 'Using command %s' % ' '.join(command) )
esptool.main(command)

Application

It is a commonly used tool in ESP’s firmware development process. When you build and flash your application to ESP, the IoT Development Framework uses esptool “in backend” by default. The same applies to others IDE (Eclipse, VS Code PlatformIO, or Arduino).

Problems that may arise

  • pyserial VS serial

My recommendation: uninstall serial module via pip and install pyserial, before installing esptool.

  • Hardware flow control

Some boards waiting for a built-in button press for 1 second to connect to UART host when using a serial connection. The esptool receives this signal and makes a further action.

2. nvs_partition_gen

Fig. 3: ESP’s Partitions Structure

Factory data, default configurations, modes, states — CSV, and why not use this?

This utility creates a bin file based on key-value pairs provided in a CSV file and also allows to decrypt an encrypted NVS binary file. Yes, key-value pairs. First of all, a few important things:

  1. You need to know NVS architecture;
  2. As Espressif said, NVS works best for storing small values. If you need to store large blobs or strings, use FAT filesystem;

So, it is not a simple CSV file: its structure contains four parameters separated by a comma.

Espressif suggests this example of such a file:

key,type,encoding,value     <-- column header
namespace_name,namespace,,  <-- First entry, "namespace"
key1,data,u8,1
key2,file,string,/path/to/file

Drop to the folder of the same name and:

$ cat sample_singlepage_blob.csv# Sample csv file
key,type,encoding,value
dummyNamespace,namespace,,
dummyU8Key,data,u8,127
dummyI8Key,data,i8,-128
dummyU16Key,data,u16,32768
# ...
# ...

And we can get our binary NVS data sample:

$ python nvs_partition_gen.py generate sample_singlepage_blob.csv sample.bin 0x3000

, where sample_singlepage_blob.csv is input data, 0x300 size of the partition in bytes, and sample.bin is the path of our output NVS binary file.

Application

  • Example 1: CSV file may consist of some calibration data for IMU/optical sensor. When the device starts up it pulls data from NVS storage.
  • Example 2: each line in the CSV file may contain the values for the variables that are unique at the factory (device ID).

Both examples have the same algorithm of actions: prepare data using nvs_partition_gen:

$ python nvs_partition_gen.py generate calibration-module.csv calibration_data_to_nvs.bin 0x3000Creating NVS binary with version: V2 - Multipage Blob Support EnabledCreated NVS binary: ===> /home/*user*/Espressif-Toolchain/nvs_partition_gen/calibration_data_to_nvs.bin

And then flash it via esptool:

$ python esptool.py --port /dev/ttyUSB0 write_flash 0x340000 calibration_data_to_nvs.binesptool.py v3.2-dev
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision 0)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: XX:XX:XX:XX:XX:XXUploading stub...
Running stub...
Stub running...
Configuring flash size...Flash will be erased from 0x00340000 to 0x00342fff...
Compressed 12288 bytes to 642...Wrote 12288 bytes (642 compressed) at 0x00340000 in 0.2 seconds (effective 575.9 kbit/s)...Hash of data verified.
Leaving...
Hard resetting via RTS pin...

Following this link, you can find some source examples on C to testing NVS data existing.

And also, Python has easy and powerful tools for processing CSV data, that you can prepare before use nvs_partition_gen, in the case when you are dealing with raw data or non-static data.

3. parttool

Fig. 4: ESP IDF SDK Configuration

Think of the partition tables as disk partitioning. When you use dd or GParted tools you have to deal with creating/deleting /resizing disk partitions and their file systems:

Fig. 5: GParted UI

And an appropriate tool for making operations on a target device related to the partition tables on ESP32 is parttool. Using it you can make a custom partition.

The parttool can be used from Python script. You need just import it as a module using IDF_PATH and path if esptool from your environment.

Also, the utility has a command-line interface. The utility is easy to use and has few parameters. Just call like this:

$ python parttool.py {command} <port> <partition-com-parameter>

Examples:

$ python parttool.py --port "/dev/ttyUSB1" erase_partition 
--partition-name=storage$ python parttool.py --port "/dev/ttyUSB1" read_partition 
--partition-type=data --partition-subtype=spiffs 
--output "spiffs.bin"

Application

This tool may help implement mechanisms for generating the factory data that will be written to some partition of your chip. The developer can add some storage area for logging (log/), some static service information (srv/), etc.

When you designing a custom partition table, it is good practice to visualize it via the partitions structure diagram. In this great article, you can find perfect examples.

4. spiffsgen & mkspiffs

Fig. 6: Default partition scheme

SPIFFS is an ultra-lightweight “filesystem”, which utilizes extra flash memory on the ESP32 chip and produces a flat structure.

Warning! SPIFFS requires extreme attention to task timing and the developer’s ability to manipulate file’s paths.

Problems that may arise

  • It is not a real-time stack. You have to consider that in a multi-threaded environment (RTOS application) each thread creates its own stack which stores local variables and chaining method execution. There is no fixed time for SPIFFS’s write operation.
  • You could not detecting a bad block (an corrupted area of storage media that is no longer reliable for storing and retrieving data).

Application

One of the distinguishing features of SPIFFS is that you can read, write, close, and delete any files : .txt, .log, .html,.img, etc. Also, if you have a multi-chip architecture in your device, you can store additional versions of their firmware.

The spiffsgen and mkspiffs utilities are tools used to generate a spiffs image from a directory. An image created with one of the two tools can also be flashed using esptool or parttool.

The spiffsgen command call looks like this:

$ python spiffsgen.py <image_size> <base_dir> <output_file>

But mkspiffs has written in C/C++, and it has the similar signature (after installation and add bin file to your .zshrc , .bashrc etc.):

$ python mkspiffs {-c <pack_dir>|-u <dest_dir>|-l|-i} [-d <0-5>] [-b [-p <number>] [-s <number>] [--] [--version] <image_file>

Which tool to use? Espressif has a comprehensive explanation.

5. idf

Fig. 7: Example of idf utility configuring

Like the esptool, this CLI tool is one of the most frequently used commands by the developers. It used to manage the building process of some large and complicated projects. To perform such hard work it manipulates:

  • cmake to configure project;
  • ninja to building a project;
  • esptool to flash the firmware.

Espressif proposes a great getting started guide that contains a brief intro of idf command.

Conclusion

One of the greatest strengths of all utilities is that they are all written in Python — incredibly easy to learn and use language. You can develop your own tool, combining the capabilities of other tools: testing interfaces, writing configuration data, or processing collected information.

After all, it is still obvious:

Today you type 10 thousand characters, tomorrows you press enter.

Esp32
Esp Idf
Python
Embedded Systems

--

--

Ihor Mierkielov
Ihor Mierkielov

Written by Ihor Mierkielov

2 Followers
4 Following

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams