Essential Edge AI for Embedded Developers

Embedded Developers, System Architects and Product Managers, who plan to deploy an AI/ML model for an application in one or more of the following constrained edge device types: 64-bit Linux based Single Board Computers (SBCs) or 32-bit microcontrollers with or without Neural Network Accelerators.

Please note that the course does not delve into details of different types of neural network architecture.  These details are covered in the  Practical Deep Learning course. Inference of large Generative AI models are not part of the course, as they are more likely to run on servers or workstations.

• The foundation of a modern MLOps workflow
• Neural Network Basics – Model training, loss functions and APIs of Keras training framework
• Data planning and pre-processing techniques for deep learning training and inference
• Architectures of Fully Connected and Convolutional Neural Networks (CNNs) models
• Applying Transfer Learning to customize models to application specific datasets
• Classifying time series data using CNNs
• Converting models between different formats (ONNX, TFLite) including explainable classical ML algorithms
• Performing model inference using different runtimes and Python / C++ API’s
• Porting a trained model to a microcontroller using TinyML (LiteRT for Microcontrollers, aka TFLite Micro)
• Compiling a model to a neural network accelerator
• Performing Object Detection using Single Shot Detection (SSD) and YOLO (You Only Look Once) models
• Hosting AI models on an IoT Device and providing inference using MQTT or REST APIs
• Deploying AI models using Docker containers

Attendees should be familiar with the basic idea of an embedded AI application. Specifically, you should have:

• Working knowledge of a programming language like Python or C/C++.
• Some background knowledge about edge devices, such as Linux based single board computers (x64 or ARM), Neural Network Accelerators or 32-bit Microcontrollers (such as a Cortex-M4 based)

Please contact Doulos directly to discuss and assess your specific experience against the pre-requisites.

Doulos training materials are renowned for being the most comprehensive and user-friendly available. Their style, content and coverage are unique in the training world and have made them sought after resources. The materials include:

• Fully indexed class notes creating a complete reference manual
• Jupyter Notebooks containing complete working code for all the neural networks presented during the training class, which you can use after the class for revision or as the basis for your own networks.
• Source code on GitHub for exercises using Linux SBC, Neural Network Accelerator and Microcontroller.
• Supplementary material on Keras APIs, OpenCV, embedded build systems, and State-of-the-art edge AI models

Introduction

Supervised learning • Inference engines at the edge • Edge AI components • Edge AI applications
Practicals:  Try out model training environments based on CPU and GPU

Neural Network Basics

Model training • Loss functions • Sequential and Functional API use and examples • Frequently used Keras APIs

Data Planning & Pre-processing

CRISP-DM Methodology & MLOps for Edge AI • Use case of audio feature pre-processing • Split audio file for training • Conversion of audio segments to Spectrogram • Training data preparation
Practicals:  Use SoX (Sound eXchange) software to trim, filter, and plot Spectrograms • View Spectrograms before training • Organize spectrograms into folders for training • Work with a subset of Environment Sound Classification (ESC) dataset

Creating Neural Network (NN) Models for Edge

Review of different kinds of neural networks • Convolutional Neural Network (CNN) using 2D and 1D convolution • Transfer Learning • Chaining NN models
Practicals:  Code (using Keras) simple Convolutional neural networks using small datasets • Write code to use pretrained models (such as MobileNet) for transfer learning

Edge AI Hardware

Constrained Inference platforms • Inference server •  Linux based SBC • 32 bit Microcontroller • NN Accelerator • GPU • FPGA
Practicals:  Take picture, play and record audio using SBC • Read sensor data using serial port from SBC and Microcontroller • Structure accelerometer data for supervised model training

Model Formats

Model Formats • Open Neural Network Exchange Format (ONNX)   •  TensorFlow Lite (TFLite) • Viewing Model Graph • Model Format Conversion • Model Quantization
Practicals:  Train and convert Scikit-Learn (ML) model to ONNX • Train and convert Keras Models to ONNX and TFLite

Model Inference

Inferencing steps • Input and Output Tensor Shape • ONNX runtime, TFLite Interpreter Python methods • TFLite C++ Classes
Practicals: Convert Scikit-learn model to ONNX format.  Perform Inference of NN model using ONNX/TFLite formats in Python environment, Inference of TFLite and ONNX models using C++ classes

TinyML

TinyML Implementation frameworks • TFLite Micro • Converting Keras Model • Setting up and using TFLite Micro Interpreter
Practicals:  Quantize Fully Connected TFLite Model • Convert Model to C array for storing in MPU • Read sensor value and execute model using TinyML (TFLite Micro) on Cortex-M4 device

Neural Network Accelerator

Neural Network Accelerator platforms  • Model compiler and workload partitioning • Executing Model on Accelerator • Working with multiple accelerators
Practicals:  Quantize and compile CNN Model for NN Accelerator • Run compiled model on accelerator and compare execution time with SBC

Object Detection

MobileNet architecture •Object Detection using MobileNet based SSD • Object Detector Output • Decoding and boxing detected output • Object Detection using You Only Look Once (YOLO) model
Practicals: Use object detection models to locate objects of interest. Employ post-processing on model output for Non-Maximum Suppression (NMS)

Monitor Model Inference and Performance

Share Model Inference using MQTT and REST API • Monitor Model Drift  
Practicals: Communicate model inference output using MQTT and Flask webserver

Deploy Model to Edge Device

Enumerate Model Inference Dependencies • Create Dockerfile for Model Inference Container • Push Model Inference Container to Edge Device
Practicals: Create ONNX/TFLite inference Dockerfiles • Use Dockerfile to create Docker container and test it on Edge device

Edge AI use cases

Edge AI application planning template • Discussion on creating Edge AI solution for different use cases

Supplementary content:

Appendix A. OpenCV

Classical computer vision algorithms • OpenCV pre-processing and post-processing functions • AI Model inferencing using OpenCV 

Appendix B. Build System

C++ application compilation • Use of CMake to build application • Use of precompiled shared object (.so) files of model runtime and OpenCV

Appendix C. Time Series

Terminology – univariate, multivariate, regression, classification • Pre-processing using Pandas • Time Series as supervised learning problem • Keras Time Series Generator API • Time Series classification using CNN • Time Series Database
Practicals: Set up time series dataset as supervised learning task • Perform CNN based fingerprint analysis of sections of time series data