FINAL EOSS-bibop-zephyr-port.pdf

1、#EMBEDDEDOSSUMMITSzymon Duchniewicz,AvanadeJakub Duchniewicz,Tietoevry WillmishJDuchniewicz Porting an AI Powered Wearable Health Monitor to Zephyr on Open Hardware#EMBEDDEDOSSUMMITSzymon Duchniewicz,AvanadeJakub Duchniewicz,Tietoevry WillmishJDuchniewicz Porting an AI Powered Wearable Health Monito

2、r to Zephyr on Open Hardware#EMBEDDEDOSSUMMITSzymon Duchniewicz,AvanadeJakub Duchniewicz,Tietoevry WillmishJDuchniewicz Porting an AI Powered Wearable Health Monitor to Zephyr on Open Hardware,but actually adding more Zephyr support for EOS S3,patching OpenOCD and debugging the bootloader#EMBEDDEDOS

3、SUMMITWhat will we cover?A war story:-goal-steps-hurdles-solutions and hints-future workTopics covered:-Porting HW to Zephyr-EOS S3 internals-TF Lite deployment-Debuggers/Flashers etc-Some medical background#EMBEDDEDOSSUMMIT whoami x2 Currently developing 5G NR(L1/L2/SDK)and leading a talented team

4、at Tietoevry for Undisclosed Client:),ex Samsung,ex GSoCGraduate of MSc in Embedded Systems at KTH Royal Institute of TechnologyIn my free time doing a ton of sports:wind/kite/surf/skiing/climbing/unicycle-maybe not now-ACLR rehab Open Technology Engineer at Avanade,co-chairing the Carbon Aware SDK

5、project,exploring the embedded systems world in OSSFinal year MEng Computer Science student at University College London.I dabble in the areas of game dev,embedded systems and IoT,cloud engineering and am crazy about windsurfing!Insert UCL logo hereJakubSzymon#EMBEDDEDOSSUMMITMotivation Why choose t

6、his problem?“we-all-know-what”overwhelmed healthcare systems worldwideWith cheap hardware we can diagnose and report cardiovascular diseases and abnormalities(e.g.arrhythmia),elevated blood pressure,hypertension etc.Fun to learn new things trying to solve important problems#EMBEDDEDOSSUMMITOriginal

7、plan/NOTE:this is not the CURRENT architectureOriginally on closed-source hardware Arduino Nano 33 IoT as part of element14s Design for a Cause 2021 CompetitionInferencing of sensor data in cloud ML using AWS LambdaOriginal project blogs can be found here-element14 communityAlso on Jakubs blog:here#

8、EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Here comes open source!(and open HW)New target hardware QuickLogic QTPlusRuns on EOS S3 SoC combining an eFPGA and Arm Cortex M4 MCUCan deploy TensorFlow Lite models on the Cortex M4https:/ diagram of the SoC,from AntMicroLow-power SoC suitable for ed

9、gedeployment-our problem seems perfect for itSensor Fusion and management(Flexible Fusion Engine-FFE)eFPGA for model deploymentI2S Always-on PDM microphone#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Plan vs reality.#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Board flashingInitial boards

10、came bricked-could not flash anythingTurned out the bootloader was borkedNo access to debugging HWJLink flashing support exists!our boards#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Board flashing-OpenOCDhttps:/ not work https:/ RTOS and QuickLogic EOS S3Zephyr supportSensor Manager-Custom ult

11、ra low power DSP(where is the Programming Manual?)Flexible Fusion Engine:-Additional packet processingand data flow managementSensor Manager-Custom ultra low power DSP(where is the Programming Manual?)Flexible Fusion Engine:-Additional packet processingDirect Wishbone I2CcommunicationApparently,SM i

12、s only useful in ultralow power applicationsWhat we did to improve boards/SoCs support on Zephyr?Read a lot of QuickLogics QORC SDK code and AntMicros work on the in-tree Zephyr supportAdded I2C and Wishbone bus support to HAL(based on FreeRTOS implementation)Added I2C driver for EOS S3 in ZephyrUpd

13、ating SoC+Board DeviceTree to reflect onboard sensors and I2C bus A blocker when porting HAL to Zephyr:FFE Clock enablingChanges to Zephyr Fork:soc/arm/quicklogic_eos_s3/soc.c Bug in the I2C HAL Used a busy loop to wait for FFE WishBone Bus status reg to change to readyChanges to HAL Module Fork:HAL

14、/src/eoss3_hal_i2c.c Adding an I2C driver using HAL for Zephyr Driver development tutorial from last years Zephyr Project Developer summit For a non-kernel dev and a newbie developer-tricky but manageable Modify driver CMakeLists.txt Add Kconfig file for your driver,modify main driver Kconfig file A

15、dd dts binding in dts/bindings/i2c in.yaml format Add actual source code for the driver Update relevant SoC DeviceTree to include the new peripheral DT_DRV_COMPAT must match Kconfig and dts binding!Helpful:read other drivers/ask on Discord!MAX30102 support-as simple as adding a DTS definition!And en

16、abling Configs(in prj.conf)for I2C and the configuring sensor Had to switch channels for IR and RED(swapped on sensor)Display support:SSD1306-as simple as adding a DTS definition!Good sample from Zephyr And enabling Configs(in prj.conf)for I2C,enabling Display and dynamic memory allocation,Character

17、 framebufferWorth noting that USB2UART requires FPGA code loaded-will not work out of the box-use hardware UART instead.If you dont have a USB2UART adapter feel free to use another board-Szymon used ESP32#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3AI on EOS S3Initially eFPGA was our target-eFP

18、GA targeted for small power/low logic operations-not powerful enough for ML just 2 32x32 or 4 16x16 multipliersIf not eFPGA then M4 for inference!What to choose:SensiML vs TFLite for Microcontrollershttps:/ suite for data capture,labelling,aggregation,processing and model deployment for inferenceToo

19、 much for our needs-we simply want to deploy a model and run inference!Difficult to do only that.room for improvement?https:/ Lite turned out to be quite simple:-convert the TF model to TFLite model-run quantization-load the model on MCU-preprocess the data-convert floats to int8_t-run inference(ARM

20、 CMSIS-NN under the hood)-int8_t to floathttps:/blog.tensorflow.org/2020/04/how-tensorflow-lite-helps-you-from-prototype-to-product.htmlhttps:/ we will be able to get our hands on one of Core-V MCUs from OpenHW Group,we would love to experiment more with ML on FPGA alongside an RTOShttps:/www.openhw

21、group.org/core-v-devkits/https:/ RTOS and QuickLogic EOS S3Blood Pressure prediction using photoplethysmographyNon-invasive method:-photoplethysmograph-using MAX30102 sensor-can get SpO2 and Heart Rate(also Blood Pressure:)Biological signals are complex.https:/ extracted:-peak ratio(systolic to dias

22、tolic)-cycle duration-various deltasInstead of putting entire period of samples to model,putting just 6 featuresStill needs data filtering and preprocessingbefore can be put into the modelModel trained using the public dataset-ground truth taken from ABP(invasive).Model is not State of the Art-CNNs

23、beat it but are much heavier,so YMMVAccuracy(Mean Average Error):SBP:16.5 DBP:8.3#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Some results?Quick Thing+,MAX30102,SSD1306 ESP32 for UART2USB debugging:)#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Things left out Submitting PRs to Zephyr,openO

24、CD and Quicklogics HAL If we configured user button as GPIO_input with Zephyr it stopped working Bootloader might be broken(maybe run Zephyr on bare metal instead?)Flasher hangs non-deterministically(TinyFPGA)-tried flashing 250 kB program and the board bricked Actually using the eFPGA,SM+FFE,maybe

25、Audio I2S system Emulating the system in RenodeAcknowledgements:Shoutout to Chris Friedt,Henrik Brix Andersen and other members of the Zephyr community for the help!Hoping for further help and PR guidance when upstreaming Paul Fertser from OpenOCD-for guidance with debugging Robert Dawson from Quick

26、logic for continuous support Folks from Antmicro for paving the way to EOS S3 support in Zephyr And other people:)Further reading:EOS S3 Technical Reference Manual Tutorial:Mastering Zephyr Driver DevelopmentBP Dataset:https:/archive-beta.ics.uci.edu/dataset/340/cuff+less+blood+pressure+estimation#E

27、MBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3TakeawaysTakeaways:When choosing a board to develop,check for peripheral support in Zephyr DOCUMENT YOUR PROGRESS!You will thank yourself later How we approached adding I2C driver for Zephyr Zephyr integrates smoothly with TFLite,although FPGA AI deplo

28、yment was not tested(yet.)When drivers are in place Zephyr just works:)#EMBEDDEDOSSUMMITZephyr RTOS and QuickLogic EOS S3Thank you.Questions? JDDuchniewiczJInsert UCL logo hereJakubSWillmishszymon-duchWillmiszRepository links:Project repo:https:/ Updated board module fork:https:/ Zephyr fork with EOSS3 I2C drivers:https:/ OpenOCD fork(with WIP EOSS3 patch):https:/ Original Blood Pressure from PPG notebook:https:/