Enhancing artificial neural networks for smarter applications on low-cost resource-constrained microcontrollers
Data
2024-03-21
Autores
Nascimento, Alexandre 
Orientadores
Basgalupp, Márcio
Tipo
Tese de doutorado
Título da Revista
ISSN da Revista
Título de Volume
Resumo
A convergência da Inteligência Artificial (IA) com a Internet das Coisas (IoT), AIoT, torna os dispositivos mais capazes em áreas como saúde, cidades inteligentes e agricultura, desempenhando um papel crucial na promoção de práticas sustentáveis e na melhoria dos esforços de conservação ambiental. No entanto, a incorporação da IA em nós computacionais de borda da IoT apresenta desafios, como a necessidade de microcontroladores superiores, que aumentam os custos e impedem soluções em larga escala. Esta tese propõe novas abordagens para lidar com três lacunas principais no uso de redes neurais artificiais (ANN, do inglês Artificial Neural Networks) em dispositivos de borda baseados em microcontroladores de baixo custo. A primeira lacuna é que as estratégias existentes para executar ANN em dispositivos de borda frequentemente resultam em acurácia reduzida e ainda exigem microcontroladores poderosos. A segunda lacuna é que os grandes conjuntos de dados necessários para treinar ANN aumentam o custo de retreino e tornam qualquer iniciativa de retreino em dispositivos de borda inviável. Finalmente, a execução de aplicacões utilizando ANN em dispositivos de borda é limitada devido à sua capacidade computacional restrita. Para abordar essas lacunas, esta tese propõe novas estratégias de treino e sintonia fina de ANN para melhorar a precisão de pequenas ANNs. As estratégias de treino propostas se baseiam na troca de otimizadores de treinamento e de pequenos sub conjuntos de dados derivados do conjunto original. Com isso, se foi possível alcançar qualidade semelhante à obtida com conjuto de dados maiores, mas utilizando conjuntos de dados menores. Por exemplo, em um experimento, foi possível atingir 80% de acurácia com apenas 52% dos dados utilizados num treino convencional. Além disso, propõe-se uma técnica para fazer a sintonia fina da ANN e adaptá-la à ANN para executar em microcontroladores de baixo-custo (ex: 8 bits), permitindo desempenho às vezes superior em comparação com benchmarks em processadores mais poderosos. Neste trabalho, um total de 16 hipóteses foram formuladas e, para testá-las, um total de 7,392,380 ANNs foram treinadas e avaliadas em 9 datasets distintos. Os resultados contribuem para o desenvolvimento de ANNs mais leves e melhores para dispositivos de borda menos poderosos, bem como criam uma agenda de pesquisa sobre a mistura de otimizadores para o treinamento de ANNs. Este trabalho estabelece o primeiro benchmark de IA para microcontroladores de 8 bits compatíveis com a plataforma Arduino.
The convergence of Artificial Intelligence (AI) with the Internet of Things (IoT), AIoT, makes devices more capable in areas such as healthcare, smart cities, and agriculture, playing a crucial role in driving sustainable practices and improving environmental conservation efforts. However, incorporating AI into IoT edge nodes presents challenges, such as superior microcontroller requirements, which increase costs and prevent large-scale solutions. This thesis proposes new approaches to address three main gaps in using artificial neural networks (ANNs) on edge devices based on low-cost microcontrollers. The first gap is that existing strategies for running ANN on edge devices often result in reduced accuracy and still require powerful microcontrollers. The second shortcoming is that the large datasets needed to train ANN increase the cost of retraining and thus preclude any retraining initiatives on edge devices. Finally, executing ANN applications on edge devices is limited due to their resource-constrained computational power. To address these shortcomings, this thesis proposes novel ANN training and tuning strategies to improve the accuracy of small ANNs. The proposed training approaches are based on swapping optimizers during training and using small subsets of data derived from the original set. It was possible to achieve quality similar to that obtained with a larger dataset but using smaller datasets. For example, 80% accuracy can be achieved with 52% of the data required on a traditional approach. Furthermore, a technique was proposed to fine-tune the ANN to run on low-cost microcontrollers (e.g., 8-bits) that could perform better than benchmarks executed on more powerful processors. For example, the fine-tuning helped to obtain networks using less than 3.5 Kbytes of RAM for execution that reached an accuracy superior to the benchmarks executed on better processors. In this work, 16 hypotheses were formulated, and to test them, 7,392,380 ANN were trained and evaluated using 9 distinct datasets. The findings suggest potential implications for developing lighter and better ANNs compatible with less powerful edge devices and highlight a research agenda on mixing optimizers for ANN training. This work also establishes the first AI benchmark for Arduino platform-compatible 8-bit microcontrollers.
The convergence of Artificial Intelligence (AI) with the Internet of Things (IoT), AIoT, makes devices more capable in areas such as healthcare, smart cities, and agriculture, playing a crucial role in driving sustainable practices and improving environmental conservation efforts. However, incorporating AI into IoT edge nodes presents challenges, such as superior microcontroller requirements, which increase costs and prevent large-scale solutions. This thesis proposes new approaches to address three main gaps in using artificial neural networks (ANNs) on edge devices based on low-cost microcontrollers. The first gap is that existing strategies for running ANN on edge devices often result in reduced accuracy and still require powerful microcontrollers. The second shortcoming is that the large datasets needed to train ANN increase the cost of retraining and thus preclude any retraining initiatives on edge devices. Finally, executing ANN applications on edge devices is limited due to their resource-constrained computational power. To address these shortcomings, this thesis proposes novel ANN training and tuning strategies to improve the accuracy of small ANNs. The proposed training approaches are based on swapping optimizers during training and using small subsets of data derived from the original set. It was possible to achieve quality similar to that obtained with a larger dataset but using smaller datasets. For example, 80% accuracy can be achieved with 52% of the data required on a traditional approach. Furthermore, a technique was proposed to fine-tune the ANN to run on low-cost microcontrollers (e.g., 8-bits) that could perform better than benchmarks executed on more powerful processors. For example, the fine-tuning helped to obtain networks using less than 3.5 Kbytes of RAM for execution that reached an accuracy superior to the benchmarks executed on better processors. In this work, 16 hypotheses were formulated, and to test them, 7,392,380 ANN were trained and evaluated using 9 distinct datasets. The findings suggest potential implications for developing lighter and better ANNs compatible with less powerful edge devices and highlight a research agenda on mixing optimizers for ANN training. This work also establishes the first AI benchmark for Arduino platform-compatible 8-bit microcontrollers.