Announcing TensorFlow Lite Micro support on the ESP32

by
Tensorflow

A guest article by Vikram Dattu, Aditya Patwardhan, Kedar Sovani of Espressif Systems

Introducing ESP32: The Wi-Fi MCU

We are glad to announce TensorFlow Lite Micro support for the ESP32 chipset.

The ESP32 is a Wi-Fi/BT/BLE enabled MCU (micro-controller) that is widely used by hobbyists and makers to build cool and interesting projects that sense or modify real world data/object, and also commonly deployed in smart home appliances like light bulbs, switches, refrigerators, and air conditioners to provide connectivity.
The interesting part of ESP32 is that it’s a unique SoC that can be used right from quick prototypes to high-volume production. A wide community, numerous development kits, and plethora of tutorials/SDKs make it a great vehicle for quick prototypes with almost any vertical you might be interested in. The all-in-one package (Wi-Fi/BT/MCU) and existing high volume deployments in the field make it ideal for building end-products with.

ESP32 is already being used in a number of smart-home/connected-device projects with a variety of sensors and actuators connected to the microcontroller to sense the environment and act accordingly. With TensorFlow Lite for Microcontrollers executing on ESP32, this opens up scenarios for all kinds of use-cases that are triggered by local inference. ESP32 has 2 CPU cores and a bunch of optimizations, making it easier to run heavy TF Micro workfloads. The Wi-Fi backhaul helps to raise remote events and trigger actions based on the inferences made.

Person Detection or a Door-Bell Camera?

As an example, we have modified the person_detection example that you all might be familiar with to make it a smart door-bell camera. We use the ESP-EYE developer kit for this demonstration. Note that this example uses person detection (it detects when a face is in front of the camera), and not person identification (identifying who the person is).

The ESP-EYE dev-kit includes the ESP32 Wi-Fi/BT MCU coupled with a 2MP camera.

In Action

In our example, we will use this camera to observe and send out an email notification if we detect a person in the vicinity.

Building it for yourself

  1. Order the ESP-EYE: You can get the ESP-EYE Development Kit from your favourite distributor, or from here. You will need a USB to micro-USB cable for connecting this to your Windows/Linux/macOS host.
  2. Clone the repository: https://github.com/espressif/tensorflow/
  3. Setup your development host: Setup your development host with toolchains and utilities required to cross-build for ESP32. Follow the instructions of the ESP-IDF get started guide to set up the toolchain and the ESP-IDF itself.
  4. Generate the example: The example project can be generated with the following command: 
    make -f tensorflow/lite/micro/tools/make/Makefile TARGET=esp generate_doorbell_camera_esp_project
  5. Build the example:

    a. Go to the example project directory

    cd tensorflow/lite/micro/tools/make/gen/esp_xtensa-esp32/prj/doorbell_camera/esp-idf

    b. Clone the esp32-camera component with following command:

    $ git clone https://github.com/espressif/esp32-camera components/esp32-camera

    c. Configure the camera and the email address:

    idf.py menuconfig

    d. Enter the Camera Pins configuration and SMTP Configuration menus to select the camera details, and also the email details.

    e. Build the example:

    idf.py build
  6. Flash and Run the program: Use the following command to flash and run the program: 
    idf.py --port /dev/ttyUSB0 flash monitor
  7. Now, whenever a person’s face is detected, the program will send out an email to the configured email address.

What Next?

Now that you have tried the door bell camera example, you may try the other applications that are part of the TF Micro repository: hello_world and micro_speech.
ESP32 is pretty powerful for a microcontroller. Clocked at 240MHz, with just a single core it can do the detection well under 1 second (roughly ~700ms; additional optimizations are on the way to reduce this even further). This leaves the second core free for other tasks from your application.
The TinyML book is an excellent resource for a thorough understanding of TensorFlow Lite for Microcontrollers.Read More