This post explains the reasoning and philosophy behind the ESP8266 IoT Framework. Since the framework is evolving over time, some of this post might be outdated. Find the latest on GitHub .

The first component I will discuss is the web server which presents the web interface to configure WiFi and other settings from the browser. This web interface is developed in React, and communicates with the ESP8266 through an API. Webpack is used to merge the GUI content into a single gzipped file, which is automatically converted into a byte array and stored in the ESP8266 PROGMEM, avoiding the need for SPIFFS.

In the schematic below, you will find a high level overview of what the web server component in the framework will be doing. It will expose two things to the outside world. First of all, a web interface that the user can interact with, and secondly an API that this web interface can interact with.

The reason for this architecture is to allow a clear split between the web interface and the ESP8266 application, making customization of the GUI easier, and allowing the web interface to be developed with a normal front-end toolchain. The general approach taken in this framework is comparable with a JAMstack website.

The architecture of the web server class is shown in blue

First I will show how the automatic integration of the web interface in the application is handled. Next I will show an example of the API and web interface implementation using the SPIFFS file manager as an example. The full code for the web server class can be found on GitHub .

Generating html.h using webpack

For this framework I wanted the web interface to be embedded into the application, so that it will be update automatically with each build. I think this is more elegant than the more common solution of storing the HTML files in the SPIFFS file memory. The best way of doing this is by storing the binary representation of the files in a byte array in PROGMEM memory.

As with many ideas, I am not the first one to do this, and I found a source of inspiration at Tinkerman . But instead of Gulp I wanted to use webpack, since I already used that for development anyway. For those of you with an electronics background, this might be unknonw territory. Basically, webpack is a method to translate development Javascript code into the final code that is served by a browser. This allows for much more flexible development methods. For our purposes here, I first need a few webpack plugins to webpack.config.js.

const { CleanWebpack } = require('clean-webpack-plugin');
const InlineSource = require('html-webpack-inline-source-plugin');
const Compression = require('compression-webpack-plugin')
const EventHooks = require('event-hooks-webpack-plugin');

The first three take care of preparing the bundle from the React web app. With these, all source Javascript and CSS will be deleted and instead inlined into the index.html file. Finally, the resulting bundle will be gzipped to significantly reduce its size. The last plugin allows to create a node.js callback function that will be executed when webpack is finished. In this callback the bundle is automatically generated into a byte array:

new EventHooksPlugin({
    done: () => {
        if (argv.mode === 'production')
        {
            var source = './dist/index.html.gz';
            var destination = './src/generated/html.h';

            var wstream = fs.createWriteStream(destination);
            wstream.on('error', function (err) {
                console.log(err);
            });

            var data = fs.readFileSync(source);
            
            wstream.write('#ifndef HTML_H\n');
            wstream.write('#define HTML_H\n\n');
            wstream.write('#include <Arduino.h>\n\n');                

            wstream.write('#define html_len ' + data.length + '\n\n');

            wstream.write('const uint8_t html[] PROGMEM = {')

            for (i = 0; i < data.length; i++) {
                if (i % 1000 == 0) wstream.write("\n");
                wstream.write('0x' + ('00' + data[i].toString(16)).slice(-2));
                if (i < data.length - 1) wstream.write(',');
            }

            wstream.write('\n};')

            wstream.write('\n\n#endif\n');

            wstream.end();

            del([source]);
            del('./dist/');
        }
    }
})

The generated bundle is saved as html.h and included in the web server class to serve it to clients. The webpack script is hooked into PlatformIO to automatically build and include the latest version of the interface into the ROM. Next, let's look at the API and the web interface itself.

The API

In order to have the API embedded in the web server class, it has to communicate with all the other parts of the application, which means these have to be included in the web server class. In the specific example of the file manager, this means that we have to include the File class.

#include <FS.h>

The two main API methods for the file manager are to list file information and to delete a file. These two web server callbacks will be shown below, starting with the function to create a file listing.

File listing

server.on(PSTR("/api/files/get"), HTTP_GET, [](AsyncWebServerRequest *request) 
{
    String JSON;
    StaticJsonDocument<1000> jsonBuffer;
    JsonArray files = jsonBuffer.createNestedArray("files");

The server.on callback is part of ESPAsyncWebServer, which is handles the low level web server implementation. This means that when the URL /api/files/get is requested, the server will call this function and start creating a response by initializing a StaticJsonDocument using the ArduinoJSON class. Next, the JSON object will be filled with the right information:

    //get file listing
    Dir dir = SPIFFS.openDir("");
    while (dir.next())
        files.add(dir.fileName().substring(1));

    //get used and total data
    FSInfo fs_info;
    SPIFFS.info(fs_info);
    jsonBuffer["used"] = String(fs_info.usedBytes);
    jsonBuffer["max"] = String(fs_info.totalBytes);

First the filenames are added to an array, and next two individual values are added to the JSON object to indicate the used bytes, and total available bytes of the storage. Finally the JSON is serialized and sent out as a response to the API request:

    serializeJson(jsonBuffer, JSON);
    request->send(200, PSTR("text/html"), JSON);
});

Removing a file

The API call for removing a file is shown below. As you can see it follows the same format, but in this case only an empty 200 request is sent to acknowledge that the request was received.

server.on(PSTR("/api/files/remove"), HTTP_GET, [](AsyncWebServerRequest *request) 
{
    SPIFFS.remove("/" + request->arg("filename"));
    request->send(200, PSTR("text/html"), "");
});

The Web Interface

Finally, this API will be used by the web interface to implement the required functionality. The page for the file manager looks as follows:

The web interface for the file manager

When the page is loaded the function fetchData is called:

function fetchData() {
    fetch('/api/files/get')
        .then((response) => {
            return response.json();
        })
        .then((data) => {
            setState(data);
        });
}

This function requests the file info object from the API and sets it as the state of the parent React component to show the file listing. As you can see there is also a delete button for each file, which links to the second API function:

<Fetch href={'/api/files/remove?filename=' + state.files[i]} onFinished={fetchData}>
    <RedButton title="Remove file"><Trash2 /></RedButton>
</Fetch>

When a response is received from the server that the file has been deleted, fetchData is called again to refresh the file listing.

Full Source Code

This post only contained some snippets of the code to explain the high level approach that was taken. The full implementation for the ESP8266 IoT framework is found on GitHub . In that repository, the documentation for the web server and API functions can be found here .