深度探討技術：退化AI？如果 GenAI 已經達到極限那么它的下一次進化是什么？.pdf

1、QATAR COMPUTING RESEARCH INSTITUTE1Dr.Sabri BoughorbelDegenerativeAI?If GenAI Has Already Hit its Limit,What is its Next Evolution?QATAR COMPUTING RESEARCH INSTITUTE2Outline2 Review of AI progress Challenges on the road of AI Promising directionshttps:/nishu- COMPUTING RESEARCH INSTITUTE3GenAI Under

2、 the Hood3QATAR COMPUTING RESEARCH INSTITUTE4AI Progress4 Compute expanding at 4x per year Fastest technological expansions in recent historyo Mobile adoption(2x/year)o Solar energy capacity(1.5x/year)o Human genome sequencing(3.3x/year)o Moores Law(1.3x/year)QATAR COMPUTING RESEARCH INSTITUTE5AI Pr

3、ogress5 AI Performance is predictableo Scaling of model,data and compute Would this lead to capability explosion?o Task automationo Fast economy growthQATAR COMPUTING RESEARCH INSTITUTE6Monetization of AI 6 Nvidia:o 100B annual revenue projected for Nvidia from data center OpenAI:o$1B revenue in Aug

4、 2023o$2B revenue in Feb 2024 o doubling every 6 months Microsoft:o$5B estimation of incremental AI revenueQATAR COMPUTING RESEARCH INSTITUTE7Models are Reaching a ceiling?7.Ilya Sutskever,co-founder of AI labs Safe Superintelligence(SSI)and OpenAI,told Reuters recently that results from scaling up

5、pre-training-the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures have plateaued.Nov 11,2024 QATAR COMPUTING RESEARCH INSTITUTE8Challenges8 Data Wall Scaling Training Runs QATAR COMPUTING RESEARCH INSTITUTE9Will We Run Out of Data

6、?9https:/epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-dataQATAR COMPUTING RESEARCH INSTITUTE10Will we run out of data?10 Data from other modalities Enterprise data Synthetic dataQATAR COMPUTING RESEARCH INSTITUTE11Synthetic Data Could Lead to Degenerative AI11

7、Gibney,Elizabeth.AI models fed AI-generated data quickly spew nonsense.Nature 632.8023(2024):18-19.Shumailov,Ilia,et al.AI models collapse when trained on recursively generated data.Nature 631.8022(2024):755-759.QATAR COMPUTING RESEARCH INSTITUTE12Scaling Training Runs122030 compute projectionhttps:

8、/epoch.ai/blog/can-ai-scaling-continue-through-2030https:/situational-awareness.ai/QATAR COMPUTING RESEARCH INSTITUTE1313Scaling Training RunsQATAR COMPUTING RESEARCH INSTITUTE14AI Capabilities Can be Improved Without Expensive Retraining14QATAR COMPUTING RESEARCH INSTITUTE15AI Capability Unleashing

9、15Compute Equivalent Gainhttps:/epoch.ai/blog/ai-capabilities-can-be-significantly-improved-without-expensive-retrainingQATAR COMPUTING RESEARCH INSTITUTE1616AI Capability UnleashingQATAR COMPUTING RESEARCH INSTITUTE17AI Capability Unleashing17Test-time computehttps:/ COMPUTING RESEARCH INSTITUTE18Conclusion18 Current AI Progress is dependent on scaling of data and compute Signs of saturation in pretraining performance New algorithmic techniques in inference time leading to significant improvement