代理、推理和硬件.pdf

1、Agents,Inference and Hardware Andrew Ng LandingAI All rights reserved.Technical Trends In AIOn-device AI.Instead of running an LLM in the cloud,run it on your own laptop,phone or industrial PC.Image/Video analysis.LLMs brought us the text processing revolution.The visual processing revolution is com

2、ing not just generation,but analysis.This will affect,manufacturing,life sciences,self-driving,retail,etc.AI Agentic Workflows.Given an instruction(“research topic X for me”)software that can carry out a sequence of steps to generate a result.LandingAI All rights reserved.Please type out an essay on

3、 topic X from start to finish in one go,without using backspace.LLM-Based AgentsNon-agentic workflow(zero-shot)Write an essay outline on topic X Do you need any web research?Write a first draft.Consider what parts need revision or more research.Revise your draft.Agentic workflowThinking/ResearchRevi

4、seStartFinish LandingAI All rights reserved.Zero-shotReflectionTool UsePlanningMultiagentGPT-4GPT-3.5Zero-shot(67%)Zero-shot(48%)40%50%60%70%80%90%100%Coding benchmark(HumanEval)Thanks to Joaquin Dominguez and John Santerre(DeepLearning.AI)for help with analysis.LandingAI All rights reserved.Heres c

5、ode intended for task:def do_task(x):.Check the code carefully for correctness,style and efficiency,and give constructive criticism for how to improve it.Theres a bug on line 5.Fix it by Reflection with LLMs Related work:Self-Refine:Iterative Refinement with Self-Feedback,Madaan et al.(2023)Reflexio

6、n:Language Agents with Verbal Reinforcement Learning,Shinn et al.,(2023)Please write code for taskdef do_task(x):.It failed Unit Test 3.Try changing def do_task_v2(x):def do_task_v3(x):Coder Agent(LLM)LandingAI All rights reserved.def do_task_v2(x):Heres code intended for task:def do_task(x):.Check

7、the code carefully for correctness,style and efficiency,and give constructive criticism for how to improve it.Theres a bug on line 5.Fix it by Recommended reading:Self-Refine:Iterative Refinement with Self-Feedback,Madaan et al.(2023)Reflexion:Language Agents with Verbal Reinforcement Learning,Shinn

8、 et al.,(2023)Please write code for taskdef do_task(x):.It failed Unit Test 3.Try changing def do_task_v3(x):Coder Agent(LLM)Critic Agent(LLM)Reflection with LLMs LandingAI All rights reserved.Agentic Reasoning Design Patterns1.Reflection2.Multi-agent collaboration 3.Tool use(API calls)4.Planning(de

9、cide on steps for task)LandingAI All rights reserved.The importance of inference 1.Agentic workloads use many more tokens than zero-shot prompting.2.Fast,low cost token generation will be a huge performance driver.Open weight models(like Llama 3.1)also make it easier for providers to compete directl

10、y on inference price and speed.3.Training remains important,but I hope our community will also invest significantly in inference.(E.g.,SambaNova,Cerebras,Groq).Also see benchmarks at artificialanalysis.ai LandingAI All rights reserved.Watch this video and tell me if any surfer is within 10m of a sha

11、rkLMM-Based AgentsNon-agentic workflow(zero-shot)1.Detect the sharks(bounding boxes)2.Detect the surfers(bounding boxes)3.Compute distances between sharks&surfers 4.Determine if any are 10m5.Iterate through Steps 1-4 for all frames in videoAgentic workflowPlanning/Testing CodingStartFinishVideo cred

12、it:Crazy Nature youtu.be/2_HfdtljBKk?si=3rKt7StDATeuFYjSDemo:Mercedes Logo LandingAI All rights reserved.Tool use:Computer Vision Models12Source:Huggingface.coImage Classification+214.3%Y/Y4.3k 13.6kObject Detection+384.2%Y/Y514 2.5kImage Segmentation+186.4%Y/Y200 842Zero-Shot Image Classification+222.9%Y/Y144 465Resources-Try VisionAgent for free at va.landing.ai-Open source code on githubLandingAI team here todaySteve AckleyIan RysdaleFireside Chat Andrew NgExecutive Chairman,LandingAIBaskar SridharanVP of AI/ML services&infrastructure at AWS at