This post explains the reasoning and philosophy behind the ESP8266 IoT Framework. Since the framework is evolving over time, some of this post might be outdated. Find the latest on GitHub .

No matter what function your ESP8266 might have, it is quite likely that you want to change some settings or parameters while the device is in use. Like for example the speed of a motor, or the color of a LED. The configuration manager presented here provides a method to easily define these tuneable parameters in a JSON file. The code to change parameters from the browser and store them into EEPROM memory is generated automatically.

To make this post more concrete we will look at a practical use case from my linear clock project. Imagine we have a goal tracker display that consists of a bar of LEDs. We would like this goal tracker to be easy to configure. For that we need to use a couple of parameters:

• url: this parameter will be a string containing the URL where the goal tracker can fetch a number between 0 and 100 to indicate the goal status
• interval: this parameter will store the number of minutes between each new request to update the latest goal status
• brightness: store and change the brightness of the LEDs in the display as a value between 0 and 1000
• sleep: this is a flag that when true, turns off the display at night

These parameters are application settings. They must be changeable from the web interface, but they must also be changeable from the application itself if needed. Finally, of course these parameters must be stored persistently during power cycles. The possible flows of the data are shown in the schematic below.

Parameters are stored in EEPROM, read and written by the application and through the GUI

First we will discuss what format will be used to actually store the data.

Configuration data structure in EEPROM

The most straightformward way to store the configuration parameters in the flash is as a simple C struct. This does not require any parsing, and can be used by the application straight away. For our example, the struct will look as follows.

struct configData
{
    char url[50];
	float interval;
	bool sleep;
	uint16_t brightness;
};

In order to store this structure in memory, we also need to define a set of default values:

const configData defaults PROGMEM =
{
	"https://www.example-api.com",
	0.5,
	true,
	750
};

Generating the code from JSON

The problem with defining the data structure in the C++ application is that the web interface is not aware which parameters are available. To solve this, there are a few options:

1. Also define the struct layout in the web interface code. This makes the code less portable and adjustments need to happen in multiple places.
2. The application code stores metadata describing the fields in the structure to communicate it to the web interface, wasting memory and communication space in the process.
3. Define the structure in JSON instead of in the application code. The application code will be generated automatically from the JSON on build, and the web interface will read the structure directly from the same JSON.

You might have guessed from the way I phrased these options that I choose the last one 😏. The JSON format to store the same data as in the previous section is:

[
    {
        "name": "url",
        "type": "char",
        "length": 50,
        "value": "https://www.example-api.com"
    },
    {
        "name": "interval",
        "type": "float",
        "value": 0.5
    },
    {
        "name": "sleep",
        "type": "bool",
        "value": "true"
    },    
    {
        "name": "brightness",
        "type": "uint16_t",
        "value": 750
    }
]

PlatformIO has a mechanism to execute a Python script before each build. For that I developed a simple Python script to generate the C++ code shown earlier from the JSON code shown above. The Python script is as follows:

import json
import binascii

filename = "config"
h = open("src/generated/" + filename + ".h", "w", encoding="utf8")
cpp = open("src/generated/" + filename + ".cpp", "w", encoding="utf8")

with open('html/js/configuration.json') as f:
  data = json.load(f)

#headers
h.write("#ifndef CONFIG_H\n")
h.write("#define CONFIG_H\n\n")
h.write("struct configData\n{\n")

cpp.write("#include <Arduino.h>\n")
cpp.write("#include \"config.h\"\n\n")

cpp.write("uint32_t configVersion = " 
          + str(binascii.crc32(json.dumps(data).encode())) 
          + "; //generated identifier to compare config with EEPROM\n\n")

cpp.write("const configData defaults PROGMEM =\n{\n")

#loop through variables
first = True
for item in data: 
    
    if first==True:
        first=False
    else:
        cpp.write(',\n')

    if item['type'] == 'char':
        cpp.write("\t\"" + item['value'] + "\"")
        h.write("\tchar " + item['name'] + "[" + str(item['length']) + "];\n")
    else:
        cpp.write("\t" + str(item['value']))
        h.write("\t" + item['type'] + " " + item['name'] +";\n")

#footers
h.write("};\n\nextern uint32_t configVersion;\n")
h.write("extern const configData defaults;\n\n")
h.write("#endif")

cpp.write("\n};")

h.close()
cpp.close()

This Python script generates the configData structure from the JSON file during each build

Updating parameters from the application

If you looked at the python script above you might have noticed that also a parameter is generated called configVersion. This is the CRC32 checksum of the JSON file, and is used to detect if the JSON has been updated compared to what is in EEPROM. If the value in the source code and in the flash are the same, the data from flash is loaded. If the values are different the default values are used instead and written to the flash.

The method to update the parameter values in the flash is very straightforward. Just provide the function saveRaw a pointer to a configData structure, and it will be written to the memory:

void config::saveRaw(uint8_t test[])
{
    memcpy(&data,test,sizeof(data));
    save();
}

void config::save()
{
    EEPROM.put(0, configVersion);
    EEPROM.put(4, data);
    EEPROM.commit();
}

Updating parameters from the web interface

The page for changing the configuration parameters is generated by reading in the same JSON file that is used to generate the application code.

On this page you can view and change the current value of the configuration parameters

This page receives the actual values from the application, and sends the updated values back once the save button is clicked. There are no functions implemented to update a single parameter only. The reason for that is that typically nothing is gained by writing individual members. The ESP8266 has no real EEPROM but will rather write the content to a flash block. Since Flash memory can only be erased in blocks you have to wipe the whole lot and rewrite it anyway.

Since the application code does not have any metadata on the configuration data structure, the communication between the web interface and the application is simply a direct binary representation of the configData struct.

Binary layout of the C struct in bytes

To make and parse this binary representation there is one concept you need to understand, which is padding. In short, every member of the struct needs to start on a byte that is a multiple of the length of said member. To achieve this some padding bytes are injected into the struct. In the example above, interval has a length of 4 bytes, and therefore needs to start at byte 52 rather than 50. And something similar applies to brightness.

What you can also learn from this is that in general it is a good idea to order your struct members from large member size to small, because in this case no padding will be needed. I ordered them poorly on purpose to show the concept 🙂.

Finally, as part of the web interface two functions obj2bin and bin2obj are developed in Javascript (GitHub ) that take care of the translation to and from a correctly padded binary structure. In this way the application can directly store the updated data to the flash memory without the need for any parsing.

Full Source Code

This post only contained some snippets of the code to explain the high level approach that was taken. The full implementation for the ESP8266 IoT framework is found on GitHub . The documentation for the configuration manager can be found here .