AI 模型基于信息的軟件優化.pdf

1、AI Hardware&SystemsaiandsystemsInformation-BasedSoftware Optimization of AI ModelsGerald Friedland,Principal Scientist AWS|Adjunct Faculty,UC BerkeleyDisclaimer:The views and opinions expressed in this presentation are my own and do not necessarily reflect the official policy or position of Amazon o

2、r its affiliates.https:/ Hardware&SystemsaiandsystemsBook on this Topic 200 downloads per day(e-book)Spanish version soon.Jupyter Notebooks with algorithms as code:https:/ Hardware&SystemsaiandsystemsHow much hardware(money,cpu time,memory,IO)do I need to budget for my AI model?State of the Art:No a

3、nswer.For example,ImageNet models vary significantly:AlexNet:238MB model,2.27Bn OpsDarkNet:28MB model,0.96Bn OpsVGG-16:528 MB,30.94Bn OpsSource:https:/ QuestionAI Hardware&SystemsaiandsystemsThe Scientific MethodData Science:The Science of Automating the Scientific MethodAI Hardware&Systemsaiandsyst

4、emsThe Scientific Method:Traditional=2AI Hardware&Systemsaiandsystems=2The Scientific Method:NewAI Hardware&SystemsaiandsystemsAssume(binary classifier)Question:How many state transitions does M need to model the training data?,()0,1Conceptual FrameworkAI Hardware&SystemsaiandsystemsEvery row gets i

5、ts own arrow:Lookup Table/DictionaryBest accuracy.No Generalization.State Machine Models Table:OverfitAI Hardware&SystemsaiandsystemsState Machine Models Table:Over generalOne arrow:Maximum generalization.Best guess accuracy.AI Hardware&SystemsaiandsystemsOptimization Problem:How many state transiti

6、ons does M need to model the training data for maximum accuracy and best generalization?Machine LearningAI Hardware&SystemsaiandsystemsCS explains Machine Learning Intellectual Capacity:The number of unique target functions a machine learner is able to represent(as a function of the number of model

7、parameters).Memory Equivalent Capacity(MEC):A machine learners intellectual capacity is memory-equivalent to N bits when the machine learner is able to represent all 2N binary labeling functions of N uniformly random inputs.At MEC or higher,M is able to memorize all possible state transitions from t

8、he input to the output.AI Hardware&SystemsaiandsystemsWhat is the MEC for?-Binary classification in a table:=#rows in bits -1 Boolean function:2v bits(v=#variables)-A game of Tic-Tac-Toe:log2 9!=18.469 bits-A game of Chess:log2 10120=400 bits(Shannon Number)-A 10-gram in an English Language model:lo

9、g2 1714761010=173.88 bits -Human Genome:6.15 10 bits-Human Brain:1015 bitsExample Capacity RequirementsAI Hardware&SystemsaiandsystemsCapacities of Manipulated TitanicMEC is proportional to the difficulty/cost of a task.Data drift/Regime Changes can be detected by estimating the Memory Equivalent Ca

10、pacity of the input data before training a model.TASKMEC ESTIMATE(BEFORE TRAINING)VALIDATION ACCURACY(AFTER TRAINING)OG78 bits82.86%all female113 bits74.14%same class,same price81 bits81.30%no family83 bits81.93%super equality139 bits68.97%only insignificant columns375 bits61.05%theoretical baseline

11、(equilibrium)H(class distribution)*800 instances/2=384 bits61.50%AI Hardware&SystemsaiandsystemsAn Energy Threshold!What is a Neuron?AI Hardware&Systemsaiandsystems1)The output of a neuron is maximally 1 bit.2)The maximum memory capacity of a neuron is the number of parameters(including bias)in bits

12、.(MacKay 2003)3)The maximum memory capacity of neurons in a neural network is additive,except 4.(MacKay 2003 speculative,Friedland and Krell 2017)4)The maximum memory capacity of a layer of neurons depending on a previous layer of neurons is limited by the maximum output(in bits)of the previous laye

13、r.(Data Processing Inequality,Tishby 2012)Capacity of Neural NetworksAI Hardware&Systemsaiandsystems2*3 bits+min(2,2*3)bits+min(2,3)bits=10 bitsw1w2w3w4b1b2w5w6w7w8b3b4b5w9w10How many bits of capacity?AI Hardware&SystemsaiandsystemsCapacity Progression:Take 5%,10%,20%,40%,80%,100%of training data and measure MEC for decision tree:Is there convergence or is it memorizing?Data SufficiencyAI Hardware&SystemsaiandsystemsData Sufficiency:TitanicAI Hardware&SystemsaiandsystemsBlessing of High DimensionalityAI Hardware&SystemsaiandsystemsQuestions?Contact:https:/ you!