2019年歷時三年的超大型軟件系統自動化與devops之路.pdf

《2019年歷時三年的超大型軟件系統自動化與devops之路.pdf》由會員分享，可在線閱讀，更多相關《2019年歷時三年的超大型軟件系統自動化與devops之路.pdf（78頁珍藏版）》請在三個皮匠報告上搜索。

1、歷時三年的超大型軟件系統DevOps之路DevOps是一種從開發到運維的快速的相互反饋的文化，并沒有具體的標準模式，但從自動化入手，最容易取得成效。本次議題主要分享實戰經驗，可以快速應用到新的產品研發中去。Session 1:DevOps 相關概念00:00 00:15Session 2:搭建DevOps 生產線00:15 02:30Session 3:問答交流02:30 03:0003:00 閉館:繼續問答交流微信/郵件：繼續問答交流Session1:DevOps Pipeline?2019/9/234傳統瀑布模型 基線綠線為時間軸紅線為基線化的文檔或文件交付的產品交付的產品系統需求需求軟件

2、需求軟件需求設計規格說明設計規格說明需求分析軟 件 設 計編碼/單元測試軟件集成測試驗 收 測 試代代 碼碼待交付的產品待交付的產品維 護2019/9/235產品研發管理文檔管理產品信息變更管理需求管理代碼版本缺陷管理客戶反饋測試用例Branch model:Git flow$git flowusage:git flow initInitialize a new git repofeature Manage your feature branches.bugfix Manage your bugfix branches.release Manage your release branches.

3、hotfix Manage your hotfix branches.support Manage your support branches.version Shows version information.config Manage your git-flow configuration.log Show log deviating from base branch.Git commands wrapper:to implement the processes and integrate other tools1.Change jira ticket status2.Tag naming

4、 convention3.Branch naming convention4.Auto Rebase.Waterfall way of workingAgile&Lean way of workingProduct readyfor deliveryTrouble report inflowTeam learningWATERFALL VS AGILE&Lean WAYS OF WORKINGSystem test progressTRANSPARENCY TASK BOARD DevOps is a set of software development practices that com

5、bine software development(Dev)and information-technology operations(Ops)to shorten the systems-development life cycle while delivering features,fixes,and updates frequently in close alignment with business objectives.DevOps DefinitionDevOps Definition Academics and practitioners have not developed a

6、 unique definition for the term DevOps.“Academic defining DevOps as a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production,while ensuring high quality.ToolchainsAs DevOps is intended to be a cross-functional mode of w

7、orking,those that practice the methodology use different sets of toolsreferred to as toolchainsrather than a single one.18These toolchains are expected to fit into one or more of the following categories,reflective of key aspects of the development and delivery process:19unreliable source?20unreliab

8、le source?Coding code development and review,source code management tools,code mergingBuilding continuous integration tools,build statusTesting continuous testing tools that provide quick and timely feedback on business risksPackaging artifact repository,application pre-deployment stagingReleasing c

9、hange management,release approvals,release automationConfiguring infrastructure configuration and management,infrastructure as code toolsMonitoring applications performance monitoring,end-user experienceSome categories are more essential in a DevOps toolchain than others;especially continuous integr

10、ation(e.g.Jenkins,Gitlab,Bitbucket pipelines)and infrastructure as code(e.g.,Terraform,Ansible,Puppet).21unreliable source?22unreliable source?Forsgren et al.found that IT performance is strongly correlated with DevOps practices like source code management and continuous delivery.13Relationship to A

11、gile Agile and DevOps serve complementary roles:several standard DevOps practices such as automated build and test,continuous integration,and continuous delivery originated in the Agile world,which dates(informally)to the 1990s,and formally to 2001.23Agile can be viewed as addressing communication g

12、aps between customers and developers,while DevOps addresses gaps between developers and IT operations/infrastructure.24Also,DevOps has focus on the deployment of developed software,whether it is developed via Agile or other methodologies.23.Relationship to Continuous delivery Continuous delivery and

13、 DevOps have common goals and are often used in conjunction,but there are subtle differences.2728 While continuous delivery is focused on automating the processes in software delivery,DevOps also focuses on the organizational change to support great collaboration between the many functions involved.

14、27 DevOps and continuous delivery share a common background in agile methods and lean thinking:small and frequent changes with focused value to the end customer.29unreliable source?Lean management and continuous delivery are fundamental to delivering value faster,in a sustainable way.14Continuous de

15、livery focuses on making sure the software is always in a releasable state throughout its lifecycle.13This makes it cheaper and less risky to deliver the software.13 Improved collaboration and communication both between and within organizational teams can help achieve faster time to market,with redu

16、ced risks.3031 move faster and be more agile without sacrificing security or reliability.DevOps is a cultural perspective on how everyone should be engaged in working the right way.DevOps is a culture,not a role!The whole company needs to be doing DevOps for it to work.DevOps is about continual lear

17、ning and improvement rather than an end state.DevOps:BenefitsDevOps:BenefitsDevOps:AutomationAgile-CI-CD-DevOpsAgile-CI-CD-DevOpsAgile-CI-CD-DevOpsDev+OpsAgile vs.DevOps怎么做到的搭建前，制定目標與量化指標：易于維護過程透明，白盒化開發人員易于使用簡單化架構易于擴充與修改配置便于支持與維護搭建過程中，采用敏捷方式，分期迭代部署出現錯誤時可以及時回溯到歷史版本不同任務之間要有清晰的接口可移植性促進而不是阻礙開發盡量使用工具的原生命

18、令日志記錄便于分析改進Session 2:DevOps PipelineDevOps toolchainDevOps toolchainDevOps toolchainBitBucket Integrations Bitbucket is more than just Git code management.Bitbucket gives teams one place to plan projects,collaborate on code,test,and deploy.https:/bitbucket.org/product/Jira SoftwareImprove efficiency

19、 by connecting branches,commits,and pull requests to Jira Software issues.Trigger webhooksIntegrate easily with systems you already use in your day-to-day using Bitbucket webhooks.git Git Large File SupportSourcetree supports Git LFS,allowing teams to track large assets in one single place.Git-flow

20、out of the boxSmart branching with Sourcetree and Git-flow,keeping repositories clean and development efficient.SubmodulesSubmodules make life easier when managing projects,their dependencies,and other project groupings.Local commit searchSearch for commits,file changes,and branches right within Sou

21、rcetree.Interactive rebaseGet clean and clear commits with Sourcetrees interactive rebase tool.Remote repository managerSourcetree allows you to search for and clone remote repositories within its simple user interface.Pipelines We accelerate release cycles from once per month to twice per week,allo

22、wing QA and product teams to test early and get customer feedback quickly“Manage your entire workflow in one toolEmpower your team with a fully visible pipelineJira The#1 software development tool used by agile teams The best software teams ship early and often.Choose a workflow,or make your own Int

23、egrate with the tools you already useJenkins 構建偉大，無所不能 Jenkins是開源CI&CD軟件領導者，提供超過1000個插件來支持構建、部署、自動化，滿足任何項目的需要。https:/jenkins.io/zh/doc/Jira SW WorkflowTransitionTriggersStart progress(To Do In Progress)Branch created(Bitbucket Server)Commit created(Bitbucket Server)Start review(In Progress In Review

24、)Pull request created(Bitbucket Server)Pull request reopened(Bitbucket Server)Review started(Crucible)Restart progress(In Review In Progress)Pull request declined(Bitbucket Server)Review rejected(Crucible)Review abandoned(Crucible)Done(In Review Done)Pull request merged(Bitbucket Server)Review close

25、d(Crucible)One Jenkinsfile on all branches,with default values.Code static analysis:sonar/coverity軟件模塊化、獨立化1.dependencies from artifactory server2.try not use local file system快速編譯是最核心部分，如何增量編譯和并行編譯自動完成絕大部分的評審，盡量減少人工評審工作量自動完成持續集成流水化作業：Jenkins從simple job到pipeline到JenkinsFile，持續集成環境也要和代碼一起進行版本管理使用虛擬機和

26、容器來提升效率如何提高構建速度：Gradle,Electrical Accelerator,ccache,RAID0,Makefile測試也分布式完成構建結果的管理和提升以Docker Images的方式發布持續交付中自動更新產品數據庫ALL in Git version control:Server Env Configurations Jenkins Job Configurations Source CodeJenkins Job Builder or Jenkins DSL Plugin:Jenkins Job Configurations對比http:/ JenkinsProduct

27、ion Jenkins的的界面修改界面修改JobJob配置配置Server Env ConfigurationsJenkins Job ConfigurationsJenkins Job Inside ScriptsTwo Instances:StagingProductionServer Env Configurations in AnsibleJenkins Job Configurations into DSL script用JobDSL腳本生成Jenkins JobIn Jenkins,create a DSL seed job,copy*.groovy into the worksp

28、ace.Then build the seed job.JobDSLWhat?CI team decided to introduce JobDSL plugin.It allows us to describe our Jenkins jobs as Groovy scripts and store them as source code in the repository of the project.Why?As it has been already stated,it was a huge effort to maintain and supervise the jobs.By on

29、ly using the web interface of Jenkins,its almost impossible to track the changes of the jobs(Job config history plugin can help you with some basic information,but its still not a solution).Pipeline jobs are used by many people in the PT,communication of the changes in the jobs cannot be done via ma

30、il(if only mail was always sent out about the changes).Not to mention repeating/common part of the jobs,etc.The introduction of JobDSL plugin and Groovy scripts will reduce maintenance costs and decrease time to deliver new CI features.How can I learn more about this plugin?Start here:https:/ is a G

31、etting started section.Also note the Pages section on the right side of the wiki page.It contains a lot of useful links to pages like:FAQ-https:/ reference-https:/ reference-https:/ Configure Block-https:/ playground(https:/job- also useful place in case you want to test your Groovy scripts.編譯加速 EA:

32、One Pager Report Significant improvements have been observed Up to 12 times faster builds for full clean build(200Mins vs.17Mins)Total Investment:240,000 SEK per year 50 EA Agents 5 Agent Servers RIO Payback Period:0.4 months050100150200250full clean buildmakeemakeEA ArchitectureBuild Cluster(10-100

33、 CPUs)ClusterManagerDeveloper MachinesProduction MachinesNetworkAgentAgentElectric File SystemNodeKernelUser-levelBuild with ElectricMakeinstead of GNU MakeGerrit/Git RepositoryelectriccommanderelectricacceleratorelectricinsightAutomateAccelerateAnalyzeBuild Acceleration:Build History File55Each blo

34、ck is a job running on an agent for some durationWhite space isthe enemyCluster is idle while blocking job runs(serialization or stall)Y-Axis is AgentsX-Axis is TimeEA Conclusion 6 Hours to 90 Minutes by build avoidance.Build out 3rdparty platform from nightly build.Reuse 3rdparty platform in a new

35、workspace.90 Minutes to 10 Minutes by build acceleration.Research on EA and deploy it in build environment.EA requires specific configuration for a given project.The tuning of EA is a continuous activity.Other considerations.Create workspace on local disk will reduce build time by 50%.Revise makefil

36、e to invoke local compiler first.編譯加速-虛擬內存盤 RAMDISK原理和用途 通過軟件將一部分內存（RAM）模擬為硬盤來使用的一種技術。實現及軟件編輯 DOS系統：XMSDSK Windows系統：VSuite Ramdisk Linux系統：直接格式化并掛載/dev/ramX 即可效果：提速一倍ccache Compiler Cache Wrapper for gcc Works fine for C,C+,Objective C or Objective C+Hash on source-code+header Local or Shared cache

37、 Can be combined with distcc Toolchain/crosscompiler may not support ccache.Precompiled headers Sometimes a single cpp-file needs hundreds of headerfiles Precompiled header flatten out headerfiles that includes many other headerfiles You can only have definitions in headerfiles,no implementationJenk

38、ins Builds Promotion and LockBuild Keeper PluginBuild Promotion PluginTo Keep a build for ever添加內容Code Change Auto-Review in GerritRun Gerrit Sanity Test on All Code Changes如果任何一項檢查沒有通過把你的代碼存入Github通過gerrithub來reviewhttps:/review.gerrithub.io/static/intro.htmlSetup Gerrit in JenkinsJenkins SystemJen

39、kins JobRain MINGW64/src/iptnms-p/.git/hooks(GIT_DIR!)applypatch-msg.sample*pre-applypatch.sample*pre-push.sample*commit-msg*pre-commit.sample*pre-rebase.sample*commit-msg.sample*prepare-commit-msg*update.sample*post-update.sample*prepare-commit-msg.sample*關聯代碼修改與User Story和Bug Fix關聯代碼修改與User Story和

40、Bug Fixgit commit$cat.gitmodulessubmodule fuel/librarypath=fuel/libraryurl=././ecs/mirantis/openstack/fuel-librarybranch=cee/ecs-mossubmodule fuel/ostfpath=fuel/ostfurl=././ecs/mirantis/openstack/fuel-ostfbranch=cee/ecs-mosGit Submodules Auto UpliftJenkins Slave auto-manager Auto add a new slave lab

41、el for a new branch pipeline Keep the next build on the same slave Remove slave label when branch obsoletes Delete its workspace on slave after 20 daysArtifactory+Docker:包文件管理與發布Artifactory will empower your experience,especially with the following integrations.Working with Docker:http:/ with Git:ht

42、tp:/ with Jenkins:http:/wiki.jfrog.org/confluence/display/RTF/Jenkins+%28Hudson%29+Artifactory+Plug-inAnt+Ivy、Maven和Gradle這些構建工具都可以自動下載Artifactory里的構件（artifact），此外Jenkins、Bamboo等CI工具也可以通過構建工具將生成的構件（artifact）部署到Artifactory上Or Nexus配置ArtifactoryManage Jenkins-Configure System-ArtifactoryJob-Configure-

43、Generic-Artifactory IntegrationThen in Artifactory,the artifacts published.Uploading using Jenkins-ArtifactoryAll deliveries from components to Release Management shall be done by uploading the binary packages into Artifactory.This includes installation packages,installation scripts,change logs and/

44、or release notes,etc.Any file type is supported(zip,gz,tgz,etc.)Uploading using CLI-ArtifactoryGet ARM-Uploader and use it in your CLI.To get usage description,run:java-jar artifactoryUploader.jar-hUsage example:Suppose you want to upload a file named package.tar as version 1.0.0 of CEE delivery to

45、ARM.You would execute:java-jar artifactoryUploader.jar-repo proj-ecs-dev-local-p cee-r 1.0.0-usr -pwd -file/path/to/my/package/package.tarNOTE!Be sure to use your encrypted password(#Artifactory|see this section)Uploading using Web GUI-Artifactory1.Log in to ARM with your account2.Click Deploy and choose the file that you want to deploy 3.Enter the details as show in the picture below.4.Refer to Naming your binaries for more details on file naming采用容器和虛擬機使用VirtualBox代替OpenStackQ&A