Mastering_DevOps_for_Embedded_Systems.pdf

1、 2024 ArmChristopher Seidl5 June 2024Mastering DevOps/MLOpsfor Embedded Systemswith Arm Innovations2 2024 ArmAgendaImproving development flows for Embedded and IoTDevOps with cloud-based Continuous Integration(CI)MLOps development flow for implementing machine learning on edge devicesCI usage exampl

2、esCloud-native tools3 2024 ArmTwo Decades of Microcontroller Innovation32-bit ComputeTrustZone SecurityHelium ML/DSP200420162020Birth of the Low Cost,Low Power 32-bit MicrocontrollerIntroducing TrustZone advanced security for Cortex-MDelivering Helium architecture technology for ML&DSP2023Today:Deli

3、vering high-efficiency ML&DSP2009Delivering DSP ExtensionsFocus on ultra-low cost/power/area2010Addressing higher performance ML&DSP applications20224 2024 ArmImprove development workflows for embedded and IoTNew tools available to developersVersion ControlSoftware DevelopmentContinuous TestingSoftw

4、are DeploymentMachine Learning(ML)Software as a Service(SaaS)Cloud StorageVirtual Machine(VM)Geographic DistributionData Analytics5 2024 ArmSoftware components and toolsDevelopment ToolsIoT ApplicationPSA Certified InterfaceHardware AbstractionSecurity Firmware Network ConnectivityRTOSML ModelSecuri

5、ty BoundaryIDE with ExtensionsModel TrainingModel DeploymentData CapturingDEVOPSIoT stacks with standard software components built MLOps6 2024 Arm Version 6Consistent software framework for billions of devicesSystem-on-chipArm Cortex processor and peripheralsCoreSight debug logicDebuggerCMSIS-Driver

6、Middleware I/FCMSIS-NNMachine learningCMSIS-DSPCompute libraryCMSIS-SVDPeripheral descriptionCMSIS-DAPDebug accessCMSIS-RTOS2Real-time executionCMSIS-CoreProcessor core and peripheral accessMPU,SAUCMSIS-ZoneSystem partitioningCMSIS-ViewEvent recorderOn-chip memoryApplication codeCMSIS-ToolboxCommand

7、-lineproject buildCMSIS-PackSoftware packaging and deliveryCMSIS-StreamOptimized data streamingfor ML and DSPSoftware components for the Arm Cortex processor targetTools for optimizing software development RetargetingSpecifications7 2024 ArmCortex-M Processor Portfolio Instruction Set EvolutionTOP/s

8、Armv8.1-MArmv8-MArmv7-MArmv6-MCortex-M33Cortex-M23*Cortex-M35PCortex-M7Cortex-M4Cortex-M3*Cortex-M0+Cortex-M0Cortex-M85Cortex-M55Cortex-M52Helium vector instruction setDSP/SIMD instructionsFloating-Point Unit(FPU)TrustZone SecurityCo-processor interface150 new scalar and vector instructionLow overhe

9、ad loopsPredicationArithmetic support for 8-bit fixed and 16-bit float Gather load,scatter storeComplex math*Cortex-M23 does not include co-processor interface,DSP/SIMD and FPU*Cortex-M3 does not include DSP/SIMD and FPUEase of developmentEnergy-efficient computeIncreased throughputSmarter devices8

10、2024 ArmvGenerative AIAIBroadest Range of ML-optimized Processing SolutionsData throughputSensor fusionKeyword detectionSpeech recognition Object classificationAnomaly detectionReal-time recognitionBiometric awarenessObject detectionGesture detectionTOP/sML Edge workloads run on:Classic Cortex-MCort

11、ex-M with HeliumCortex-M with Ethos-UCortex-A with Ethos-U859 2024 ArmEthos-U:Unlocking the Full Potential of Neural NetworksAccelerating implementation of higher performance AI enabled systems Ethos-U5532-256 MACsEthos-U65256-512 MACsEthos-U55/Ethos-U65MatrixMACWeightsEthos-U85128-2048 MACsEthos-U8

12、5Transformer supportX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XX X X X X X XMatrixWeightsMAC9Matrix10 2024 ArmEthos-U software flow on Cortex-M systemsRuntime executable file on deviceAccelerates kernels on Ethos-

13、UDriver handles the communicationThe remaining layers are executed on Cortex-MCMSIS-NN optimized kernels if availableFallback on the TFL reference kernelsLinked platform runtimeTFLu RuntimeRef.KernelsCMSIS-NNOptimized KernelsCortex-M CPUTA R G E T /D E V I C EH O S T (O F F L I N E)TF FrameworkVela

14、CompilerTF QuantizationTFL flat fileNN OptimizerEthos-UDriverEthos-U NPUTrain network in TensorFlowModel conditioning techniques:Collaborative clustering,pruning and QAT to improve model performance on Ethos-U while preserving its accuracy.Quantize it to Int8 TFL flatbuffer file(.tflite file)NN Opti

15、mizer identifies graphs to run on Ethos-UOptimizes,schedules and allocates these graphsLossless compression,reducing size of tflite file11 2024 ArmThe AI Software Ecosystem is Converging on ArmAccelerating software development and navigating emerging frameworksDevicesAI FrameworksSoftware EcosystemD

16、evelopers,Data Scientists,Academia 2024 ArmDevOps with cloud-based Continuous Integration(CI)Test and verification with Arm Virtual Hardware(AVH)13 2024 ArmWhat is test-driven development,DevOps,and CI/CD?Test-driven development implements the“test-first”programming concepts of extreme programming.D

17、evOps combines software development(Dev)and IT operations(Ops).Continuous integration/continuous delivery(CI/CD)services provide the Ops part.Combined,these techniques enable agile software development with the ability to make code changes quicker!Source https:/commons.wikimedia.org/wiki/File:Devops

18、-toolchain.svgSoftware development(Dev)IT operations(Ops)14 2024 ArmTypes of software testingBetter quality code faster,conforming to safety standards 15 2024 ArmChallenges of hardware-based CI/CD testingHardware constrains test automationScalability is difficultComplexity and high maintenance costs

19、Limited ability to stimulate I/OSource:commons.wikimedia.org/w/index.php?curid=48764103Source:ArmSource:ArmSource: PXI Systems16 2024 ArmModern DevOps development PipelineCode RepositoryCode updateStart CIBuildTestResultsArm Virtual Hardware Cloud ServiceReturn CI resultsObtain CI resultsLocal devel

20、opmentComplex debug17 2024 ArmArm Virtual Hardware(AVH) Fast Models forCorstone and Cortex-M CPUsCortex-M TrustZone SIMD Helium Ethos-U65/U55microNPUMemory Secure/Non-secure DMAPeripherals GPIO UART,SPI,I2C EthernetVirtual I/O Data values Streaming BSD-SocketDebug Interface MDK,DS GDB Event Recorder

21、Developer ResourcesAWS and GitHubArm Virtual Hardware I/O drivers Test scripts CI/CD integration Usage examples Test report tools AVH Fast Models C/C+Compiler Build utilities,.18 2024 ArmEssentially the same event logs are generated across the different deployments.This ensures correctness.Event Log

22、 fileBoardLayerUser Application CodeApplication software from virtual to physical hardwareProvide evidence of correctness on Arm Virtual Hardware and physical hardwareVirtualLayerEvent Log fileUser Application CodeDevice SDK with configurationUser Application CodeTargetLayerEvent Log fileMore Driver

23、sReal I/O via test equipmentReal I/O via user peripheralsUnit&integration testing on virtual hardware or physical hardware boardsDeployment and system testing on production hardwareSimulated I/O via Python scripts and stimuli filesHW DriversDevice SDK with configurationHW DriversVirtual DriversIoT/M

24、L Software PlatformIoT/ML Software PlatformIoT/ML Software PlatformMore Software19 2024 Arm CMSIS-ToolboxFoundation tools for CLI and IDEThe csolution tool is designed to manage:Multiple sub-projects that are related(boot,application,etc.).Complex system setups for different targetsTarget:Portable,r

25、eproducible builds with dependencies that work across different hosts,i.e.to the AVH CI/CD cloud service.Supports all major host operating systemsMulti-compiler support(Arm,Clang,GCC,IAR)Developed in collaboration with:target-types:#different targets-type:Virtual Target device:ARM:SSE-300-MPS3#Arm V

26、irtual Hardware device-type:Board board:IMXRT1050-EVKB#NXP iMX.RT1050 board-type:Hardware device:MIMXRT1052DVL6B#iMX.RT1052 devicebuild-types:#build-types-type:Release optimize:speed-type:Debug debug:on optimize:debugprojects:#projects part of the solution-project:App/MyApplication.cproject.yml-proj

27、ect:Secure/TFM.cproject.ymlSupports multiple build-types and target-types20 2024 ArmAutomated delivery of Arm toolsTool deployment to MDK(VS Code),CI SaaS,and MLOps Arm tools artifactory provides access to all tools.Microsoft vcpkg enables uniform tool installation across various host systems.Config

28、uration file controls consistent tools setup.If part of a repository,it ensures consistent setup.MLOps and CI systems can access tools directly.Example for Docker setup: Code)MLOps System(i.e.Qeexo)SaaS CI System(i.e.GitHub)vcpkg enabledDirect access/vcpkg enabledtools artifactoryGitRepositoryTools

29、installed on demandvcpkgconfigurationML ModelUser CodeDirect accessConsistent,project-specific tools setup 2024 ArmMLOps Development FlowDeploy and maintainmachine learning applications22 2024 ArmTalk about MLEnergy Efficient AI T on Edge Devices Grow Opportunity Applied to New Emerging MarketsMedic

30、al diagnoses Natural voice-controlled devicesVision based automationSustainable agricultureetc.23 2024 ArmEEMBC AudioMark=DSP+NN Workload(baseline Cortex-M4)Helium and Ethos-U Benefits:more computations vector processing(MVE)and micro NPU +new DSP/ML kernels +stream-based PoCFoundation Tools&Softwar

31、e:simplify development-CMSIS-DSP/NN libraries,Python wrapper,Proof of concepts-stream-based processing techniques-Open software and tools platformML on Edge Devices=compute+libraries+toolsArm support for DSP/ML24 2024 ArmMLOps:deploy and maintain Machine Learning(ML)models Combines machine learning

32、data analytics with continuous development(DevOps)ML models are tested and developed in isolated systems.MLOps is an iterative process to transition the ML model to production systems.Adding ML is an evolutionary process.Evaluation and validation require the model to run on target hardware.Training

33、DataValidation DataTest DataData LabelingData PreparationModel Training*Model Evaluation*System ValidationModel Deployment*Data CollectionData Analysis*Supported by Arm Virtual Hardware and SDS Framework25 2024 ArmMachine Learning(ML)Requires Real-World DataData collection requires frequently inputs

34、 of the final target systemSource:https:/ DataValidation DataTest DataDecision:ABData LabelingData PreparationModel Training*Model Evaluation*System ValidationModel Deployment*Data CollectionData Analysis*Supported by Arm Virtual Hardware and SDS Framework26 2024 ArmMaterial for ML DevelopersML Deve

35、lopers Guide for Cortex-M Processors and Ethos-UEmbedded Developers that use microcontrollers and/or Ethos-UMLOps system architects that integrate the various development toolsData scientists that develop new ML models and need performanceinformationFoundation Tool Components for MLOps SystemsSetup

36、of a Docker container for MLOps systemsCreate trained ML models with different compilers for target processorsCompare performance(inference time)using Arm Virtual HardwareSynchronous Data Stream(SDS)FrameworkFlexible data stream management for sensor and audio data interfacesProvides methods to reco

37、rd real-world data for analysis and developmentPlayback real-world data for algorithm validation using Arm Virtual Hardware27 2024 ArmRecord real-world data with Synchronous Data Streaming(SDS)Simplify development of embedded applications that utilize DSP or ML algorithms with sensor/audio inputMCU

38、DeviceSoftware on Physical BoardMicrocontroller HardwareSDS Recorder InterfaceSDS Recorder connects viadifferent channelsAlgorithm under DevelopmentCapture physical sensor(real-world)data using the original Microcontroller target hardwareSDS Data Files*.gyroscope.sdsAudioInterfaceMicrophone InputSDS

39、 Data Files*.microphone.sdszyxTraining DataValidation DataTest DataData conversionto ML trainingplatform can beautomated.The SDS framework provides methods to record real-world data for analysis and development.Input to Digital Signal Processing(DSP)development tools such as filter designersInput to

40、 Machine Learning(ML)model classification,training,and performance optimization.SDS framework is an open-source project: Sensor InterfaceGyroscopeSensor28 2024 ArmAVH VSISoftware on Arm Virtual HardwareSoftware on Physical BoardMCU DeviceSDS enables playback of real-world data for algorithm testingC

41、ombined with AVH it enables repeatable test automation in CI systems and MLOps cloud servicesMEMS Sensor InterfaceGyroscopeSensorMicrocontroller HardwareSDS Recorder InterfaceSDS Recorder connects viadifferent channelsAlgorithm under DevelopmentSDS Data Files*.gyroscope.sdsAudioInterfaceMicrophone I

42、nputSDS Data Files*.microphone.sdszyxVSI Sensor InterfaceVirtual Streaming Interface#1Arm Virtual Hardware(AVH)Algorithm under DevelopmentVSI AudioInterfaceVirtual Streaming Interface#2SDS Data Files*.gyroscope.sdsSDS Data Files*.microphone.sdsSame algorithm is verified on precise processor simulati

43、on model29 2024 ArmCore Compute Capabilities for a Modern Development FlowAccess to DSP/ML capabilities without specialized tools simplifies developmentExample:Cortex-M Processor with Helium running a predictive maintenance workloadSignal conditioningFiltering,sensor fusion,Kalman filterFeature extr

44、actionSpectral dataML classificationClassical MLDeep learning(NN)InOutCMSIS-DSP and CMSIS-NNOptimized compute librariesCMSIS-StreamOptimized data streamingbetween compute nodes31 2024 ArmExample:MLOps workflow with Qeexo AutoMLCAPTUREDATADEPLOYTO TARGETDATA IMPORTTO MLOPSML MODEL TRAININGAdd SDS fra

45、mework to target applicationDefine sensors channels and capturing frequencyCreate metadata files to describe sensor dataCapture SDS data filesVerify SDS data files usinga viewerConvert SDS data files and label dataUpload data files to MLOps system,for example Qeexo AutoMLData cleaning and preprocess

46、ingFeature extractionML model selectionParameter optimizationModel ValidationModel conversion anddownloadIntegrate ML model library in target projectValidate ML model usingArm Virtual HardwareFinal system test in target hardware 2024 ArmCI Usage Example33 2024 ArmMachine learningTraditional signal p

47、rocessingVerification of complex applicationsVirtual interfaces combined with Python scripts for effective test automationConceptual smart speakerNoisecancellationAutomatic gain controlSpeechfeatureextractionNeural networkAudioFrontendDigitalmicrophonearrayKeyword detectionoutputsArm Virtual Hardwar

48、e BenefitsVerify correctness of software algorithms on Arm processorsPerformance comparison of different algorithms Beam forming34 2024 ArmVirtualTargetMCU DeviceAVH Example:Testing of Tensorflow M with 2n blocksAudio buffer handlingVirtualInterfaceHAL DriverAudio Drivervia I2SMicrophoneArm Virtual

49、HardwareMicrocontroller HardwareUses an audio driver with:Support for multiple channels.Support for multiple data formats.Data transfer in blocks(maximizes simulation speed reduces interaction with Python).Callback event at each block transfer.Audio DriverVSIVirtual Streaming Interface(VSI)Audio fil

50、etest.wavAudio FrontendData#0Data#1Data#2Data#3Data#n-1Block#0Data#0Data#1Data#2Data#3Data#n-1Block#1Data#0Data#1Data#2Data#3Data#n-1Block#2Data#0Data#1Data#2Data#3Data#n-1Block#m35 2024 ArmCMSIS-View:Get execution details with event annotationshttps:/ void loop():EventStartCv(0,current_time,previou

51、s_time);TfLiteStatus feature_status=feature_provider-PopulateFeatureData(error_reporter,previous_time,current_time,)EventStopCv(0,feature_status,how_many_new_slices);:EventStartCv(1,current_time,how_many_new_slices);TfLiteStatus invoke_status=interpreter-Invoke();EventStopCv(1,invoke_status,0U);:Eve

52、ntStartCv(2,current_time,0U);TfLiteStatus process_status=recognizer-ProcessLatestResults(output,current_time,&found_command,&score,&is_new_command);EventStopCv(2,process_status,score);36 2024 ArmSoftware on Arm Virtual HardwareSoftware on Physical BoardDebuggerProcessing of Event Recorder DataMicroc

53、ontroller HardwareEvent RecorderArm Virtual Hardware(AVH)ApplicationEvent RecorderDebug AdapterApplicationEventListCommand Line UtilityCTF File*.EventRecorder.logCTF File*.EventRecorder.log37 2024 ArmA CI/MLOps Examplehttps:/ 2024 ArmSummary39 2024 ArmDeveloping in a cloud-native worldCloud-native t

54、ools and services help to reduce time-to-market,improve code quality,and to create new business cases.Dont be afraid of the cloud!You dont have to switch everything at once starting the journey is key.Arm and its eco-system support you with joint initiatives and common standards40 2024 ArmWhere to g

55、o nexthttps:/ Sign up for Keil Studio Cloud a browser-based IDEAccess to supported boards and deviceshttps:/arm-software.github.io/CMSIS_6/latest/General/index.htmlDeveloper resourcesLinks to all components(incl.CMSIS-View)Migration documentationhttps:/ set of foundation tools and software component

56、s to enable MLOps systems and the overall development flow for machine learning applicationshttps:/ Documentation about SDS FrameworkUtilities for data recording,viewing,and data conversionExample project that shows usageThank YouDankeGraciasGrazie謝謝AsanteMerciKiitos 2024 ArmThe Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited(or its subsidiaries)in the US and/or elsewhere.All rights reserved.All other marks featured may be trademarks of their respective 2024 Arm