1、End-to-end Autonomous Driving Industry Report,2024-2025D End-to-end intelligent driving research:How Li Auto becomes a leader from an intelligent driving followerThere are two types of end-to-end autonomous driving:global(one-stage)and segmented(two-stage)types.The former has a clear concept,and muc

2、h lower R&D cost than the latter,because it does not require any manually annotated data sets but relies on multimodal foundation models developed by Google,META,Alibaba and OpenAI.Standing on the shoulders of these technology giants,the performance of global end-to-end autonomous driving is much be

3、tter than segmented end-to-end autonomous driving,but at extremely high deployment cost.Segmented end-to-end autonomous driving still uses the traditional CNN backbone network to extract features for perception,and adopts end-to-end path planning.Although its performance is not as good as global end

4、-to-end autonomous driving,it has lower deployment cost.However,the deployment cost of segmented end-to-end autonomous driving is still very high compared with the current mainstream traditional“BEV+OCC+decision tree” UniADAs a representative of global end-to-end autonomous driving,Waymo EMMA direct

5、ly inputs videos without a backbone network but with a multimodal foundation model as the core.UniAD is a representative of segmented end-to-end autonomous End-to-end autonomous driving researches are mainly divided into two categoriesBased on whether feedback can be obtained,end-to-end autonomous d

6、riving researches are mainly divided into two categories:the research is conducted in simulators such as CARLA,and the next planned instructions can be actually performed;the research based on collected real data,mainly imitation learning,referring to UniAD.End-to-end autonomous driving currently fe

7、atures an open loop,so it is impossible to truly see the effects of the execution of ones own predicted instructions.Without feedback,the evaluation of open-loop autonomous driving is very limited.The two indicators commonly used in documents include L2 distance and collision rate.L2 distance:The L2

8、 distance between the predicted trajectory and the true trajectory is calculated to judge the quality of the predicted trajectory.Collision rate:The probability of collision between the predicted trajectory and other objects is calculated to evaluate the safety of the predicted trajectory.The most a

9、ttractive thing about end-to-end autonomous driving is the potential for performance improvement.The earliest end-to-end solution is UniAD.A paper at the end of 2022 revealed that the L2 distance was as long as 1.03 meters.It was greatly reduced to 0.55 meters at the end of 2023 and further to 0.22

10、meters in late 2024.Horizon Robotics is one of the most active companies in the end-to-end field,and its technology development also shows the overall evolution of the end-to-end route.After UniAD came out,Horizon Robotics immediately proposed VAD whose concept is similar to that of UniAD with much

11、better performance.Then,Horizon Robotics turned to global end-to-end autonomous driving.Its first result was HE-Driver,which had a relatively large number of parameters.The following Senna has a smaller number of parameters and is also one of the best-performing end-to-end Senna DriveVLMThe core of

12、some end-to-end systems is still BEVFormer which uses vehicle CAN bus information by default,including explicit information related to the vehicles speed,acceleration and steering angle,exerting a significant impact on path planning.These end-to-end systems still require supervised training,so massi

13、ve manual annotations are indispensable,which makes the data cost very high.Furthermore,since the concept of GPT is borrowed,why not use LLM directly?In this case,Li Auto proposed DriveVLM.As the figure below shows,the pipeline of DriveVLM from Li Auto mainly involves design of the three major modul

14、es:scenario description,scenario analysis,and hierarchical The scenario description module of DriveVLMThe scenario description module of DriveVLM is composed of environment description and key object recognition.Environment description focuses on common driving environments such as weather and road

15、conditions.Key object recognition is to find key objects that have a greater impact on current driving decision.Environment description includes the following four parts:weather,time,road type,and lane line.Differing from the traditional autonomous driving perception module that detects all objects,

16、DriveVLM focuses on recognizing key objects in the current driving scenario that are most likely to affect autonomous driving decision,because detecting all objects will consume enormous computing power.Thanks to the pre-training of the massive autonomous driving data accumulated by Li Auto and the

17、open source foundation model,VLM can better detect key long-tail objects,such as road debris or unusual animals,than traditional 3D object detectors.For each key object,DriveVLM will output its semantic category(c)and the corresponding 2D object box(b)respectively.Pre-training comes from the field o

18、f NLP foundation models,because NLP uses very little annotated data and is very expensive.Pre-training first uses massive unannotated data for training to find language structure features,and then takes prompts as labels to solve specific downstream tasks by fine-tuning.DriveVLM completely abandons

19、the traditional algorithm BEVFormer as the core but adopts large multimodal models.Li Autos DriveVLM leverages Alibabas foundation model Qwen-VL with up to 9.7 billion parameters,448*448 input resolution,and NVIDIA Orin for inference How does Li Auto transform from a high-level intelligent driving f

20、ollower into a leader?How does Li Auto transform from a high-level intelligent driving follower into a leader?At the beginning of 2023,Li Auto was still a laggard in the NOA arena.It began to devote itself to R&D of high-level autonomous driving in 2023,accomplished multiple NOA version upgrades in

21、2024,and launched all-scenario autonomous driving from parking space to parking space in late November 2024,thus becoming a leader in mass production of high-level intelligent driving(NOA).Reviewing the development history of Li Autos end-to-end intelligent driving,in addition to the data from its o

22、wn hundreds of thousands of users,it also partnered with a number of partners on R&D of end-to-end models.DriveVLM is the result of the cooperation between Li Auto and Tsinghua University.In addition to DriveVLM,Li Auto also launched STR2 with Shanghai Qi Zhi Institute,Fudan University,etc.,proposed

23、 DriveDreamer4D with GigaStudio,the Institute of Automation of Chinese Academy of Sciences,and unveiled MoE with Tsinghua University.Mixture of Experts(MoE)Architecture In order to solve the problem of too many parameters and too much calculation in foundation models,Li Auto has cooperated with Tsin

24、ghua University to adopt MoE Architecture.Mixture of Experts(MoE)is an integrated learning method that combines multiple specialized sub-models(i.e.experts)to form a complete model.Each expert makes contributions in the field in which it is good at.The mechanism that determines which expert particip

25、ates in answering a specific question is called a gated network.Each expert model can focus on solving a specific sub-problem,and the overall model can achieve better performance in complex tasks.MoE is suitable for processing considerable datasets and can effectively cope with the challenges of mas

26、sive data and complex features.Thats because it can handle different sub-tasks in parallel,make full use of computing resources,and improve the training and reasoning efficiency of Mixture of Experts(MoE)Architecture STR2 Path PlannerSTR2 is a motion planning solution based on Vision Transformer(ViT

27、)and MoE.It was developed by Li Auto and researchers from Shanghai Qi Zhi Research Institute,Fudan University and other universities and institutions.STR2 is designed specifically for the autonomous driving field to improve generalization capabilities in complex and rare traffic conditions.STR2 is a

28、n advanced motion planner that enables deep learning and effective planning of complex traffic environments by combining a Vision Transformer(ViT)encoder and MoE causal transformer architecture.The core idea of STR2 is to wield MoE to handle modality collapse and reward balance through expert routin

29、g during training,thereby improving the models generalization capabilities in unknown or rare DriveDreamer4D World ModelIn late October 2024,GigaStudio teamed up with the Institute of Automation of Chinese Academy of Sciences,Li Auto,Peking University,Technical University of Munich and other units t

30、o propose DriveDreamer4D.DriveDreamer4D uses a world model as a data engine to synthesize new trajectory videos(e.g.,lane change)based on real-world driving data.DriveDreamer4D can also provide rich and diverse perspective data(lane change,acceleration and deceleration,etc.)for driving scenarios to

31、increase closed-loop simulation capabilities in dynamic driving scenarios.The overall structure diagram is shown in the figure.The novel trajectory generation module(NTGM)adjusts the original trajectory actions,such as steering angle and speed,to generate new trajectories.These new trajectories prov

32、ide a new perspective for extracting structured information(e.g.,vehicle 3D boxes and background lane line details).*Subsequently,based on the video generation capabilities of the world model and the structured information obtained by updating the trajectories,videos of new trajectories can be synth

33、esized.Finally,the original trajectory videos are combined with the new trajectory videos to optimize the 4DGS Table of Content(1)1.Foundation of End-to-end Autonomous Driving Technology1.1 Terminology and Concept of End-to-end Autonomous Driving1.2 Introduction to and Status Quo of End-to-end Auton

34、omous DrivingBackground of End-to-end Autonomous Driving Reason for End-to-end Autonomous Driving:Business ValueDifference between End-to-end Architecture and Traditional Architecture(1)Difference between End-to-end Architecture and Traditional Architecture(2)End-to-end Architecture EvolutionProgres

35、s in End-to-end Intelligent Driving(1)Progress in End-to-end Intelligent Driving(2)Comparison between One-stage and Two-stage End-to-end Autonomous DrivingMainstream One-stage/Segmented End-to-end System Performance ParametersSignificance of Introducing Multi-modal models to End-to-end Autonomous Dr

36、ivingProblems and Solutions for End-to-end Mass Production(1)Problems and Solutions for End-to-end Mass Production(2)Progress and Challenges in End-to-end Systems 1.3 Classic End-to-end Autonomous Driving CasesSenseTime UniADTechnical Principle and Architecture of SenseTime UniAD Technical Principle

37、 and Architecture of Horizon Robotics VAD Technical Principle and Architecture of Horizon Robotics VADv2 VADv2 TrainingTechnical Principle and Architecture of DriveVLM Li Auto Adopts MoE MoE and STR2E2E-AD Model:SGADSE2E Active Learning Case:ActiveAD End-to-end Autonomous Driving System Based on Fou

38、ndation Models 1.4 Foundation Models 1.4.1 Introduction Core of End-to-end System-Foundation Models Foundation Models(1)-Large Language Models:Examples of Applications in Autonomous DrivingFoundation Models(2)-Vision Foundation(1)Foundation Models(2)-Vision Foundation(2)Foundation Models(2)-Vision F

39、oundation(3)Foundation Models(2)-Vision Foundation(4)Foundation Models(3)-Multimodal Foundation Models(1)Foundation Models(3)-Multimodal Foundation Models(2)1.4.2 Foundation Models:Multimodal Foundation ModelsDevelopment of and Introduction to Multimodal Foundation ModelsMultimodal Foundation Models

40、 VS Single-modal Foundation Models(1)Multimodal Foundation Models VS Single-modal Foundation Models(2)Technology Panorama of Multimodal Foundation Models Multimodal Information Representation1.4.3 Foundation Models:Multimodal Large Language ModelsMultimodal Large Language Models(MLLMs)Architecture a

41、nd Core Components of MLLMsMLLMs-Mainstream ModelsApplication of MLLMs in Autonomous Driving1.5 VLM&VLA Application of Vision-Language Models(VLMs)Development History of VLMsArchitecture of VLMsApplication Principle of VLMs in End-to-end Autonomous DrivingApplication of VLMs in End-to-end Autonomous

42、 D Table of Content(2)VLMVLAVLA ModelsVLA PrincipleClassification of VLA ModelsCore Functions of End-to-end Multimodal Model for Autonomous Driving(EMMA)1.6 World ModelsDefinition and ApplicationBasic ArchitectureGeneration of Virtual Training DataTeslas World ModelNvidiaInfinityDrive:Breaking Time

43、Limits in Driving World Models1.7 Comparison between E2E-AD Motion Planning ModelsComparison between Several Classical Models in Industry and Academia Tesla:Perception and Decision Full Stack Integrated ModelMomenta:End-to-end Planning Architecture Based on BEV SpaceHorizon Robotics 2023:End-to-end

44、Planning Architecture Based on BEV SpaceDriveIRL:End-to-end Planning Architecture Based on BEV SpaceGenAD:Generative End-to-end Model1.8 Embodied Language Models(ELMs)ELMs Accelerate the Implementation of End-to-end SolutionsApplication ScenariosLimitations and Positive Impacts2 Technology Roadmap a

45、nd Development Trends of End-to-end Autonomous Driving2.1 Technology Trends of End-to-end Autonomous Driving Trend 1Trend 2Trend 3Trend 4Trend 5Trend 6Trend 72.2 Market Trends of End-to-end Autonomous DrivingLayout of Mainstream End-to-end System SolutionsComparison of End-to-end System Solution Lay

46、out between Tier 1 Suppliers(1)Comparison of End-to-end System Solution Layout between Tier 1 Suppliers(2)Comparison of End-to-end System Solution Layout between Other Autonomous Driving Companies Comparison of End-to-end System Solution Layout between OEMs(1)Comparison of End-to-end System Solution

47、 Layout between OEMs(2)Comparison of NOA and End-to-end Implementation Schedules between Sub-brands of Domestic Mainstream OEMs(1)Comparison of NOA and End-to-end Implementation Schedules between Sub-brands of Domestic Mainstream OEMs(2)Comparison of NOA and End-to-end Implementation Schedules betwe

48、en Sub-brands of Domestic Mainstream OEMs(3)Comparison of NOA and End-to-end Implementation Schedules between Sub-brands of Domestic Mainstream OEMs(4)2.3 End-to-end Autonomous Driving Team BuildingImpacts of End-to-end Foundation Models on Organizational Structure(1)Impacts of End-to-end Foundation

49、 Models on Organizational Structure(2) Table of Content(3)End-to-end Autonomous Driving Team Building of Domestic OEMs(1)End-to-end Autonomous Driving Team Building of Domestic OEMs(2)End-to-end Autonomous Driving Team Building of Domestic OEMs(3)End-to-end Autonomous Driving Team Building of Domest

50、ic OEMs(4)End-to-end Autonomous Driving Team Building of Domestic OEMs(5)End-to-end Autonomous Driving Team Building of Domestic OEMs(6)End-to-end Autonomous Driving Team Building of Domestic OEMs(7)Team Building of End-to-end Autonomous Driving Suppliers(1)Team Building of End-to-end Autonomous Dri

51、ving Suppliers(2)Team Building of End-to-end Autonomous Driving Suppliers(3)Team Building of End-to-end Autonomous Driving Suppliers(4)3.End-to-end Autonomous Driving Suppliers3.1 MOMENTA ProfileOne-stage End-to-end Solutions(1)One-stage End-to-end Solutions(2)End-to-end Planning ArchitectureOne-sta

52、ge End-to-end Mass Production Empowers the Large-scale Implementation of NOA in Mapless CitiesHigh-level Intelligent Driving and End-to-end Mass Production Customers3.2 DeepRoute.aiProduct Layout and Strategic DeploymentEnd-to-end Layout Difference between End-to-end Solutions and Traditional Soluti

53、onsImplementation Progress in End-to-end Solutions End-to-end VLA Model AnalysisDesignated End-to-end Mass Production Projects and VLA Model FeaturesHierarchical Prompt TokensEnd-to-end Training SolutionsApplication Value of DINOv2 in the Field of Computer VisionAutonomous Driving VQA Task Evaluatio

54、n Data SetsScore Comparison between HoP and Huawei 3.3 Huawei Development History of Huaweis Intelligent Automotive Solution Business UnitEnd-to-end Concept and Perception Algorithm of ADS ADS 3.0(1)ADS 3.0(2):End-to-endADS 3.0(3):ASD 3.0 VS.ASD 2.0End-to-end Solution Application Cases of ADS 3.0(1)

55、End-to-end Solution Application Cases of ADS 3.0(2)End-to-end Solution Application Cases of ADS 3.0(3)End-to-end Autonomous Driving Solutions of Multimodal LLMs End-to-end TestingVQA TasksArchitecture of DriveGPT4End-to-end Training Solution ExamplesThe Training of DriveGPT4 Is Divided Into Two Stag

56、esComparison between DriveGPT4 and GPT4V3.4 Horizon Robotics ProfileMain PartnersEnd-to-end Super Drive and Its AdvantagesArchitecture and Technical Principle of Super Drive Journey 6 and Horizon SuperDrive?All-scenario Intelligent Driving SolutionSenna Intelligent Driving System(Foundation Model+En

57、d-to-end) Table of Content(4)Core Technology and Training Method of SennaCore Module of Senna3.5 Zhuoyu Technology Profile R&D and ProductionTwo-stage End-to-end ParsingOne-stage Explainable End-to-end ParsingEnd-to-end Mass Production Customers3.6 NVIDIA ProfileAutonomous driving solutionDRIVE Thor

58、 Basic Platform for Autonomous Driving Next-generation Automotive Computing PlatformLatest End-to-end Autonomous Driving Framework:Hydra-MDP Self-developed Model Architecture3.7 Bosch Intelligent Driving China Strategic Layout(1)Based on the End-to-end Development Trend,Bosch Intelligent Driving ini

59、tiates the Organizational Structure ReformIntelligent Driving Algorithm Evolution Planning 3.8 Baidu Profile of ApolloStrategic Layout in the Field of Intelligent DrivingTwo-stage End-to-endProduction Models Based on Two-stage End-to-end Technology Architecture Baidu Auto Cloud 3.0 Enables End-to-en

60、d Systems from Three Aspects3.9 SenseAutoProfile UniAD End-to-end SolutionDriveAGI:The Next-generation Autonomous Driving Foundation Model and Its AdvantagesDiFSD:SenseAutos End-to-end Autonomous Driving System That Simulates Human Driving Behavior DiFSD:Technical Interpretation3.10 QCraftProfile Dr

61、iven-by-QCraft High-level Intelligent Driving SolutionEnd-to-end LayoutAdvantages of End-to-end Layout3.11 WayveProfileAdvantages of AV 2.0GAIA-1 World Model-ArchitectureGAIA-1 World Model-Token GAIA-1 World Model-Generation Effect LINGO-2 3.12 Waymo End-to-end Multimodal Model for Autonomous Drivin

62、g(EMMA)EMMA Analysis:Multimodal I Table of Content(5)EMMA Analysis:Defining Driving Tasks as Visual Q&AEMMA Analysis:Introducing Thinking Chain Reasoning to Enhance InterpretabilityLimitations of EMMA3.13 GigaStudioIntroductionDriveDreamerDriveDreamer 2DriveDreamer4D 3.14 LightWheel AIProfileCore Te

63、chnologyCore Technology StackData Annotation and Synthetic Data4.End-to-end Autonomous Driving Layout of OEMs4.1 Xpengs End-to-end Intelligent Driving LayoutEnd-to-end System(1):ArchitectureEnd-to-end System(2):Intelligent Driving ModelEnd-to-end System(3):AI+XNGPEnd-to-End System(4):Organizational

64、TransformationData Collection,Annotation and Training4.2 Li Autos End-to-end Intelligent Driving LayoutEnd-to-end Solutions(1)End-to-end Solutions(2)End-to-end Solutions(3)End-to-end Solutions(4)End-to-end Solutions(5)End-to-end Solutions(6)End-to-end Solutions:L3 Autonomous DrivingEnd-to-end Soluti

65、ons:Building of a Complete Foundation ModelTechnical Layout:Data Closed Loop4.3 Teslas End-to-end Intelligent Driving LayoutInterpretation of the 2024 AI ConferenceDevelopment History of AD Algorithms End-to-end Process 2023-2024 Development History of AD Algorithms(1)Development History of AD Algor

66、ithms(2)Development History of AD Algorithms(3)Development History of AD Algorithms(4)Development History of AD Algorithms(5)Tesla:Core Elements of the Full-stack Perception and Decision Integrated Model End-to-end AlgorithmsWorld ModelsData EnginesDojo Supercomputing Center4.4 Zerons End-to-end Int

67、elligent Driving LayoutProfileEnd-to-end Autonomous Driving System Based on Foundation Models(1)End-to-end Autonomous Driving System Based on Foundation Models(2)-Data TrainingAdvantages of End-to-end Driving System 4.5 Geely&ZEEKRs End-to-end Intelligent Driving LayoutGeelys ADAS Technology Layout:

68、Geely Xingrui Intelligent Computing Center(1) Table of Content(6)Geelys ADAS Technology Layout:Geely Xingrui Intelligent Computing Center(2)Geelys ADAS Technology Layout:Geely Xingrui Intelligent Computing Center(3)Xingrui AI foundation modelApplication of Geelys Intelligent Driving Foundation Model

69、 TechnologyZEEKRs End-to-end System:Two-stage SolutionZEEKR Officially Released End-to-end PlusZEEKRs End-to-end PlusExamples of Models with ZEEKRs End-to-end System4.6 Xiaomi Autos End-to-end Intelligent Driving LayoutProfile End-to-end Technology Enables All-scenario Intelligent Driving from Parki

70、ng Spaces to Parking SpacesRoad Foundation Models Build HD Maps through Road TopologyNew-generation HAD Accesses End-to-end SystemEnd-to-end Technology Route4.7 NIOs End-to-end Intelligent Driving LayoutIntelligent Driving R&D Team Reorganization with an Organizational Structure Oriented Towards End

71、-to-end System From Modeling to End-to-end,World Models Are the Next World Model End-to-end SystemIntelligent Driving Architecture:NADArch 2.0End-to-end R&D Tool ChainImagination,Reconstruction and Group Intelligence of World Models NSimSoftware and Hardware Synergy Capabilities Continue to Strength

72、en,Moving towards the End-to-end System Era 4.8 Changan Automobiles End-to-end Intelligent Driving LayoutBrand LayoutEnd-to-end System(1)End-to-end System(2)Production Models with End-to-end System 4.9 Mercedes-Benzs End-to-end Intelligent Driving LayoutBrand New Vision-only Solutions without Maps,L

73、2+All-scenario High-level Intelligent Driving Functions Brand New Self-developed MB.OS Cooperation with Momenta4.10 Cherys End-to-end Intelligent Driving LayoutProfile of ZDRIVE.AICherys End-to-end System Development P ContactBeijing HeadquartersTEL:13718845418Email:Chengdu BranchTEL:028-68738514FAX:028-86930659Website:ResearchInChinaWeChat:Zuosiqiche