使用 DATABRICKS SDK FOR GO 生成應用程序.pdf

1、2024 Databricks Inc.All rights reservedBUILDING BUILDING DATABRICKS DATABRICKS INTEGRATIONS INTEGRATIONS WITH GOLANGWITH GOLANGSerge SmertinSerge Smertin12024 Databricks Inc.All rights reserved2CAN BE EVERYTHING:CAN BE EVERYTHING:FROM INFRASTRUCTURE FROM INFRASTRUCTURE TO APPLIED LLM USETO APPLIED L

2、LM USE2024 Databricks Inc.All rights reserved2022 Databricks Inc.All rights reserved3About SergeAbout SergeAt Databricks since 2019Author of Databricks Terraform Provider(written in Go)Author of Databricks SDK for Go(and Python,Java,)Initiated replatforming of Databricks CLI into Go(from Python)Driv

3、ing Databricks Labs With love-hate attitude towards GoLang since 20202024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved4-RELEASE NOTES RELEASE NOTES WITH LLMSWITH LLMS-CLEANUP OF CLEANUP OF DATABRICKS DATABRICKS WORKSPACESWORKSPACES2024 Databricks Inc.All rights reserved2

4、024 Databricks Inc.All rights reserved5-50 50 REPOSITORIESREPOSITORIES2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved62024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights res

5、ervedUNRELEASED UNRELEASED COMMITSCOMMITSGIT DIFF GIT DIFF PER COMMITPER COMMITSPLIT DIFF SPLIT DIFF PER FILEPER FILESUMMARIZE SUMMARIZE FILE CHANGEFILE CHANGESUMMARIZE SUMMARIZE ALL CHANGESALL CHANGESWRITE WRITE CHANGELOG.MDCHANGELOG.MDWRITE WRITE ANNOUNCEMENTANNOUNCEMENT1 12 23 34 45 56 67 72024 D

6、atabricks Inc.All rights reserved9var fileDiffTemplate=MessageTemplate(var fileDiffTemplate=MessageTemplate(Here is the commit message terminated by Here is the commit message terminated by-for the context:for the context:.Message.Message-Do not hallucinate.You are Staff Software Engineer,and you ar

7、e reviewing one file at a time in a unified Do not hallucinate.You are Staff Software Engineer,and you are reviewing one file at a time in a unified diff format.Do not use phrases like In this diff,In this pull request,or In this file.Do not diff format.Do not use phrases like In this diff,In this p

8、ull request,or In this file.Do not mention file names,because they are not relevant for the feature description.mention file names,because they are not relevant for the feature description.If new methods are added,explain what these methods are doing.If existing functionality is changed,If new metho

9、ds are added,explain what these methods are doing.If existing functionality is changed,explain the scope of these changes.Please summarize the input as a single paragraph of text written in explain the scope of these changes.Please summarize the input as a single paragraph of text written in America

10、n English.Your target audience is software engineers,who adopt your project.If the prompt contains American English.Your target audience is software engineers,who adopt your project.If the prompt contains ordered or unordered lists,rewrite the entire response as a paragraph of text.ordered or unorde

11、red lists,rewrite the entire response as a paragraph of text.)func(lln*llNotes)Commit(ctx context.Context,commit*github.RepositoryCommit)(History,error)func(lln*llNotes)Commit(ctx context.Context,commit*github.RepositoryCommit)(History,error)err:=err:=lln.http.Do(ctx,GET,lln.http.Do(ctx,GET,fmt.Spri

12、ntf(https:/ httpErr*httpclient.HttpErrorvar httpErr*httpclient.HttpErrorif errors.As(err,&httpErr)&httpErr.StatusCode=404 if errors.As(err,&httpErr)&httpErr.StatusCode=404 return Historyreturn HistoryAssistantMessage(fmt.Sprintf(Commit%s was not found,commit.SHA),AssistantMessage(fmt.Sprintf(Commit%

13、s was not found,commit.SHA),nil,nil tokens:=strings.Split(commit.Commit.Message,)tokens:=strings.Split(commit.Commit.Message,)if len(tokens)15_000 if len(tokens)15_000 commit.Commit.Message=strings.Join(tokens:15_000,)commit.Commit.Message=strings.Join(tokens:15_000,)return return lln.explainDiff(ct

14、x,Historylln.explainDiff(ctx,HistoryfileDiffTemplate.AsSystem(commit.Commit),fileDiffTemplate.AsSystem(commit.Commit),&buf),&buf)2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved11func(lln*llNotes)Talk(ctx context.Context,h History)

15、(History,error)func(lln*llNotes)Talk(ctx context.Context,h History)(History,error)logger.Debugf(ctx,Talking with AI:logger.Debugf(ctx,Talking with AI:n%s,h.Excerpt(80)n%s,h.Excerpt(80)response,err:=lln.response,err:=lln.w.ServingEndpoints.Queryw.ServingEndpoints.Query(ctx,(ctx,serving.QueryEndpointI

16、nputserving.QueryEndpointInputName:lln.model,Name:lln.model,Messages:h.Messages(),Messages:h.Messages(),MaxTokens:lln.cfg.MaxTokens,MaxTokens:lln.cfg.MaxTokens,)if err!=nil if err!=nil return nil,fmt.Errorf(llm:%w,err)return nil,fmt.Errorf(llm:%w,err)for _,v:=range response.Choices for _,v:=range re

17、sponse.Choices h=h.With(AssistantMessage(v.Message.Content)h=h.With(AssistantMessage(v.Message.Content)return h,nilreturn h,nil 2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved122024 Databricks Inc.All rights reserved13NO LANGCHAIN.NO LANGCHAIN.PURE DATABRICKS PURE DATA

18、BRICKS REST API.REST API.ONE BINARY AS ONE BINARY AS RESULT.RESULT.2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved1 hour1 hourExplaining release Explaining release and writing and writing announcementsannouncements7 minutes7 minutesRemoving LLM Removing LLM hallucinati

19、ons and hallucinations and other minor editsother minor edits after two weeks of part after two weeks of part-time effort time effort in between other work and meetings I in between other work and meetings I was able to reduce release time:was able to reduce release time:2024 Databricks Inc.All righ

20、ts reserved2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved16HOW TO WIPE HOW TO WIPE THOUSANDS OF THOUSANDS OF TEST JOBS/TEST JOBS/USERS AT SCALEUSERS AT SCALE2024 Databricks Inc.All rights reserved17“static fixtures”are created by

21、 Terraform“dynamic fixtures”that require a cleanupDatabricks Labs Watchdog2024 Databricks Inc.All rights reserved20MAINTAIN TEST MAINTAIN TEST ENVIRONMENTS ENVIRONMENTS CLEAN AND SECURECLEAN AND SECURE2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedLIST ALL LIST ALL WOR

22、KSPACESWORKSPACESLIST 30 OBJECT LIST 30 OBJECT TYPESTYPESVERIFY VERIFY EXPECTED EXPECTED CONFIGSCONFIGSWAIT FOR ANY WAIT FOR ANY CI JOBS TO CI JOBS TO FINISHFINISHRANDOMIZED RANDOMIZED PARALLEL PARALLEL DELETEDELETE1 12 23 34 45 52024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved222024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved23