41.國際論壇-Avanish Tiwari-分布式敏捷和開發運維——實踐者方法（30頁）.pdf

1、Distributed Agile and DevOpsPractitioner approach Avanish TiwariAvanish TiwariIndustry Principal and Head of DeliveryIndustry Principal and Head of DeliveryInfosys Technologies(China)Company LimitedOrganizational Context 組織背景3復雜的環境更迅速的變化更多的工作復用和改造更快、更優、更智能Time allocation for operation team運維團隊的時間分配4

2、Source:Study from Deepak Patil Microsoft Global Foundation Services in 2006,via James HamiltonDeployment management and fix deployment related incident take up nearly half of their time(47%)in typical large and medium-sized operations teams運維團隊有將近一半（47%）的時間花在了與部署有關的工作中：1.日常新增功能的部署2.修復與部署有關的問題DevOps

3、Philosophy centers around“Enterprise Agility”DevOps注重于“企業敏捷性”Speed to ValueChange ImaginedPrioritizeChange DeliveredAccelerateAdoptShift leftLean&agile cultureProblem-findingEmpathy-based stakeholder engagementInsights-based learningContinuous feedbackContinuous deliveryTime to ValueCustomer Centric

4、ityCost EffectivenessService Reliability and QualityIntegratedtoolingOrganizational Readiness&Maturity Assessment a glimpse組織準備狀態和成熟度評估Key Activities Define DevOps goals Define scope Identify key stakeholders Conduct a workshop/surveys with identified stakeholders Understand the current initiatives,

5、priorities,challenges andexpectations IT value stream analysis Portfolio analysis,DevOpsreadiness and maturity assessment,analysis and gap report DevOps technology assessment and tooling gap analysis Create assessment report,playback the findings with the stakeholders and incorporate the feedbackDev

6、Ops readiness and maturity assessmentImportant DevOps Metrics DevOps 重要指標Focus AreaMetricFormulaProposed UsageRelease frequencyDeploys/DayTotal#of deployments/#of days To understand increased collaboration&keep a track of time based improvementReleases/Week Total#of releases/week(by environments)Imp

7、rovement in production release agility over the period of timeAgilityChange Lead TimeTime required to release a change in production from the time the development begins on the change requestTo know the trend in lead time&help remove any bottlenecks if a degradation is observed QualityCode/Test Cove

8、rage%of code covered through the test cycle,(Unit test)%functionality covered through test cycle(regression test)To know the testing effectiveness through the developed codeRelease Efficiency/repeatability%of successful releases over a period of timeHelps measure effectiveness of CD setup&observe tr

9、end over a period of timeProduction Defect MatrixAverage#of defects released into production Helps measure effectiveness of quality control through continuous delivery setupLead time provisioningTime taken for provisioningTo know the trend in lead time&help remove any bottlenecks if a degradation is

10、 observed PerformanceAvailability%availability of serversDetermine stability of environments across enterpriseOutage Outage per day/week/monthTo measure availability against SLACapacity%capacity availability&utilization Gauge adequate capacity planning&useReliabilityMean Time to RecoverTime required

11、 to recover from outages Improvement in MTTR over the timeEfficiencyCost or effort per releaseInitial effort/cost per release compared to post DevOpsTo understand how efficient release process is IllustrativeWhat is Agile?What should be the investment?How to implement agile in our business/technolog

12、y context?How to enable teams and business stakeholders?When to choose agile for a project?How to replicate success of small projects to large size projects?Why certain agile projects fail?Why we do not get desired agile benefits?How to perform multi-location execution?How can I improve on my Agile

13、adoption?How Agile and waterfall co-exist in a portfolio?How to standardize the process across organization?While IT uses Agile,we really dont get frequent production releasesHow to measure/report agile projects?How can I scale up Agile adoption?Entrant,practitioner or expert,organizations still hav

14、e concerns to be addressed新手、從業者或專家、組織仍有待解決的問題Deploy to multipleenvironments 多環境部署Gated deployments 原型部署Artefact repository based deployments 基于版本控制系統Container based deployments 容器部署Service testing 服務測試Performance testing 性能測試Virtualization 測試結果可視化Functional testing 功能測試Security testing 安全測試CONTINUO

15、US DEPLOYMENT持續部署CONTINUOUS TESTING持續測試完善的DevOps管理平臺Automated code reviews,Unit testing,Coverage,Obfuscation,Security scan 自動化代碼測試Auto setup of Open Source tools 開源工具的自動化安裝CONTINUOUS INTEGRATION持續集成預測和挖掘的能力Predication&Insights9DevOps Platform DevOps平臺的能力Increased Adaptability to ChangeIncreased Spee

16、dImproved Visibility Quicker DecisionsFail Early,Fail CheapReduced WastageDrives Innovation and CompetitionCutting CornersFitting more in less budgetJust replacement for WaterfallEasy to implement:No need to train or coachOnly coding and nothing elseEscape from processesBut Agile does not guarantee

17、success.It only increases the likelihood of success 敏捷不保證成功。它只增加成功的可能性Agile is notAgile deliversAgile has gained wide acceptance for its showcased benefits and quick success with small projects敏捷已經被廣泛接受，因為它展示了小項目的快速成功Traditional Approach towards Setting up CI-CD需要從傳統的工作方式走向持續集成-持續交付Create workflowsS

18、et UpTools orchestrationMaintenanceCI-CDComplexities in integration of various tools整合多種工具的復雜度Lack of tooling and skills 工具和技能缺乏Effort intensive需要巨大的工作量來測試，部署和交付Process&compliance overheads-approvals and role based governance審批管控，流程和協作的管理工作量流程Process人員People技術Technology11Redundant tooling ecosystem冗

19、余的工具和功能，割裂的支撐系統Reduced visibility&governance缺乏可視化和管控Delay in Time to Value set up,integrate&adopt交付的延遲Significant maintenance overheads巨大的維護和管理成本Disparate CI-CD orchestrations完全不同的集成和上線節奏Challenges get amplified in a larger IT landscape在多團隊的組織中，運營挑戰會成倍上升Multiple teams re-inventing the wheel多團隊相互協同影響

20、Problem areas 發生問題的領域Set UpTools orchestrationMaintenanceCI-CDCreate workflowsSet UpTools orchestrationMaintenanceCI-CDCreate workflowsSet UpTools orchestrationMaintenanceCI-CDCreate workflowsMultiple Application teams 12Agile is more relevant today for dynamic market conditions but its time to chan

21、ge the agile way今天，敏捷與動態市場情況更加相關，但現在是時候改變敏捷方式了Agile is the way for IT to remain relevant to BusinessChallenging market conditions make Business demand more from ITReduced time-to-market Low upfront investmentEarly realization of business valueMonitoring delivered valueagainst fundingIncreased Adapta

22、bility to ChangeIncreased SpeedImproved VisibilityFail Early,Fail CheapReduced WastageImproved QualityBut,co-located,small team focused agile is not sufficientLosing cost advantage and talent availability Inability to rapidly scale up agile execution Not able to generate business value rapidlyChangi

23、ng Agile with changing timesCo-located ToDistributedProjects To EnterpriseIT goalsToBusiness valueChanging Agile and its key considerations改變敏捷方式及其關鍵點 Communication Collaboration Maintain agile spirit Scalability Faster adoption Organization challenges Reduced cycle time Time to Market Productivity

24、gain Low cost Improved quality Customer satisfactionChanging Agile with changing timesCo-located ToDistributedProjects To EnterpriseIT goalsToBusiness value1231.Changing Agile from Local to Global 敏捷從本土轉向全球Traditional Agile High cost Small teams with scalability challenge Rapid development of featur

25、es by group of smart developers in random fashionGlobal Agile Global Agile with state of art infrastructure for global communication and collaboration Distributed agile practices Global Agile:a structured,risk controlled,measurable delivery methodology.Expertise:agile trained talent pool across glob

26、eHow do we do it?Robust Process and IPsCollaboration Talent Enablement Virtual ScrumBoot CampAgile Scrum Process TrainingAgile Mindset TrainingAgile AwarenessScrum Master and Team View152.Shift focus from rapid development to quicker business value realization關注點從快速開發轉變為更快的商業價值實現16Traditional Agile

27、Agile is a project development methodologyGlobal Agile Realization of business value is a key to agile success.Rapidly developing code which is released less frequently leads to delay in realizing business value.Moving from agile development to agile delivery is a solution.Building automated continu

28、ous delivery pipelines leading to increased release frequency ensures frequent business value delivery.How do we do it?Building Engineering Ecosystems for continuous deliveryalliances beneficial to our customers 與客戶共贏Progression from Agile development Agile Delivery DevOps敏捷開發 敏捷交付 DevOps3.Change Ag

29、ile from projects to enterprise-wide:structured,repeatable 敏捷從項目延伸到到企業范圍：采用結構化的、可重復的方式17Traditional Agile Consultants add theory but execution is a different reality.Scaling agile is doing more and more agile projectsGlobal Agile Theory to practical Consulting capabilities coupled with large scale p

30、roven delivery capability ensures end to end responsibility of clients agile journey Structured scaling from projects to portfolios to enterprise coupled with organization change management How do we do it?Largest pool of consultants and coaches providing servicesEnterprise Scalability FrameworkAgil

31、e Maturity Assessment FrameworkAgile DeciderScrum Adoption FrameworkRisk Management Framework BVACXO dashboardITPA-IT process enablerAgile Readiness/Maturity AssessmentEnterprise Agile Transformation Rapid Agile Adoption Agile CoachingAgile TrainingTools and products evaluationAgileCoaches and consu

32、ltants.and moreAgile journey by adopting Global Agile 全球敏捷之旅Global Agile Ecosystem全球敏捷生態系統Coaching&Talent DevelopmentProcessRole basedTechnologyToolsSoft skillsMindsetCultureCapabilityPracticesToolsGovernanceGlobal AgileCase Reference 案例參考Customer Delivery ModelUSA-HeadquartersOffshore Development C

33、enterThe program follows a Global Scrum Delivery model from 2 different geographical locations USA Customer Onsite locationOffshore Development Center in IndiaContextThe Engagement is deals with Partner eco-system of the Large Hi-Tech customer.This engagement has few applications that offers several

34、 tools and platforms to its partners.Provides enrollment,Re-enrollment and program/benefits access for ISV partners;manages annual enrollment+profile,Serves as Partner Data Master.ScopeDelivering key business functionalitiesFeature Development-EnhancementsSustain EngineeringTestingBuild EngineeringX

35、-Geo DeliveryCost Savings and Time Advantages(Follow-the-Sun)were the key drivers for adopting a X-Geo delivery by the customer.10 feature crews and 2 Sustain crews with more than 90%of personnel presence.Program Context Global Agile delivery in scale規?；蛎艚萁桓禟ey Challenges 關鍵挑戰Huge pressure due to

36、a recent failure of another vendor team to create a new platform replacing the current.Low CSAT in using current platform.High volume of support ticketsBacklog from business pending to be delivered for partners through features;Acceptance criteria not defined well/not completeUrgent demand to ramp u

37、p new teams in IT(India/USA)Accumulated bugs,technical debts(automation,performance)in the current platform.CICD was not implemented and deployments took longer than expectedAgile transformation of the new teams.Transparent metrics to measure output through VSO adoption.Slower time to market,with hu

38、ge number of integration points and dependencies to be regressed every release.Agile teams working across geographies to align to a solution.Low morale due to aggressive timelinesTraditional culture on understanding on roles and responsibilities of a SWE.Client problem-Need for transformed agile tea

39、ms in scale.客戶需要規?；拿艚輬F隊Client expectations客戶期望Increased CSATReduced bug trendImproved demand management and traceability from portfolio through to Agile delivery teams Improved throughput and productivity from the teams.Faster time to market22SAFe Principles and Agile practicesOrg wide approval fro

40、m customer to adopt SAFe including business stakeholders with commitment to business results.Created optimally sized,cross-functional feature crews(Scrum Master,Product Owner,SWEs).Defined roles,artifacts,and agile team performance metricsIntroduced Portfolio,Program,and Team levels making up the Ag

41、ile Release TrainIntroduced backlogs with Epics,Features and stories in VSO and maintained hygiene of updates.Introduced Normalized Story points and a common estimation template across teams.Set up cadence(2 week sprints)to synchronize and align with the crews.Built an effective“Sprint ahead”practic

42、e to ascertain delivery of sprint commitment.At the program level had a 12 week planning increment event.Created parallel teams to address automation(Build,Deploy,Regression and Load tests).Created a systems team to help set up CI-CD to gate check-ins,deployments and code promotions and manage envir

43、onments.1Agile Teams structure(90%team located in offshore)敏捷團隊結構（90%為離岸成員）Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 1(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 2(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBug

44、sFeature Crew 7(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 8(8)Stabilization Crew(20)(Note:Functions can be optimized into feature crews as they get more self-organized)Regression(5)Release Manager(RTE)Product Owners(8)Sustain Crew(10)Build(5)Perf Testing(5)Svc En

45、gineering(5)Bug Triage Support ProcessRoot cause,bug fixes,Improve Live site monitoring through Power BI Dashboards.Data workaroundsAdhoc business change/data requestsModularize Regression test suitesClear Automation debts(Build,Deploy,Regression,Perf tests)Enable CI-CD Monitor DeploymentsEnvironmen

46、t Support and MaintenanceMonitor System health,system performance and availability.Onsite 現場Offshore 離岸Zero-defect Feature DevelopmentUnit tests,Acceptance TestsFixes defects from FortifyFixes defects from regression suite.Fixes defects from Load testsIntegrate Acceptance tests into regression suite

47、Self report on Sprint performance metricsAccountable for project success or failure.Product Owner manages the Product Backlogs,stories,acceptance criteria.Owns what is desired and why is desiredInterfaces with Release manager,SMEs,Business stakeholders to engage with scrum team.Release calendarFacil

48、itate Program Increment workshopsTracks and mitigates release level issues/risks.Program level status reportWorkshop GoalQ2 program Increment PlanAlignmentTypical Quarterly Program Increment planning workshop典型的季度產品增量規劃季度研討會Sprint 1Iteration 1Sprint GoalsIteration 2Sprint GoalsUAT&DeployDemo&RetroSp

49、rint 2Sprint 3Sprint 4Iteration 1Sprint GoalsIteration 2Sprint GoalsDemo&RetroSprint 5Sprint 6Sprint 7EpicsEpic,User StoriesUAT&DeployIteration 1Sprint GoalsIteration 2Sprint GoalsDemo&RetroUAT&DeployFeature PM manages the Product Backlogs.Owns what is desired and why is desiredSprint aheadSprint ah

50、eadSprint aheadEstimateEstimateEstimateCode completion and unit testingCode review and reworkCode check-inAutomation test scriptingEstimate Sprint BacklogDefine Sprint scopeCommit to sprintRegression Pass,Demo&RetroRegression pass,Demo&RetroRegression pass,Demo&RetroSprint aheadEstimateSprint aheadE

51、stimateSprint aheadEstimateNext Program IncrementRelease 1Release 2Release 3Scrum Master:Keeper of scrum process,facilitatorTeam:Owns how and how quickly work is deliveredProgram IncrementRelease 2Release 3Technology,tools andFrameworks1.Visual Studio Online2.ASP.NET MVC/WEB API3.JQUERY4.Fortify sca

52、ns5.Style Cop4.MS unit (Unit tests)5.Selenium web driver(Feature tests)6.PowerShell scripting1.Release manager2.App Insights(MSIT data link)3.SQL 2014,SSIS1.Azure Web apps2.SQL Azure3.Azure active directoryWeekly status reporting depicting all the below pictorially to all business stake holders ona.

53、Release/Sprint calendar with monthly release cadence b.Progress of the features,risk/issues.e.Weekly support tickets trend old/new,application availability,highlight and lowlights26PlatformImprovementsImproved Telemetry needs that helped in faster resolution of bugsAddress environment gaps to be on

54、par with production for early visibility of defects.Addressed Regression tests,Loads test automation and Build and deployment automation.Continuous Integration using Release Manager to boost release confidenceGated Check-ins with unit tests Daily Build with integrated BVTs,automated security scans.C

55、ontinuous deployment pipeline gated with the right categories of tests(Feature/Regression)Built POWERBI dashboards with comprehensive set of production scenarios for live site monitoring to address problems proactively to avoid negative CSAT.Implemented few self-serve tools to fix data bugs so susta

56、in crew could focus on business features and also improve CSAT.2自管理成員的敏捷性能指標示例566702468S32S33S34S35Story points delivered per person/SprintVelocity TrendTrack the team effort completed for each sprint.Estimate the backlog effort that team can handle in future sprints if the team composition and spri

57、nt duration stay constant.Sprint Burn down chartTrack remaining work in a sprint backlog,understand how quickly team has completed tasks,and predict when the team will achieve the goal or goals of the sprint.Productivity TrendThis will help the Scrum Master to track the Productivity improvement with

58、 each iteration for each member in the team Bug TrendTrack the teams progress toward resolving and closing bugs by viewing the Bug Status report.This report shows the cumulative bug count based on the bug state,priority,and severityReduced Bug TrendReduced bugs from 300 to 90 through Telemetry,Power

59、 BI reportsEarly visibility to defects with improved test suites as a service for feature,regression and load tests.Followed TDD where Acceptance tests were automated using GHERKIN and Specflow.Regular Cadence with Service Engg and Helpdesk teams to control new tickets.Improved ThroughputCrews veloc

60、ity improved from sprint to sprint;Productivity Improvement from 750 SP to 1400 SP in 2 quarters with same team sizeSprint Ahead Feature PM has realized in improved Sprinting Ahead,saving 2 dev days per Sprint team.Faster Time to MarketAutomated the build and deployment process by incorporating CI/C

61、D builds(80%effort reduction in deployment cycle in various environments,60%reduction of downtime)Enhanced the regression automation suite and performance suite to reduce the regression effort.(40%reduction in Regression Testing cycle time)Reduced UAT timeline to the bare minimum by ensuring zero es

62、caped defects.Increased CSATDemo/Review the features with Business groups for each release has enhanced the partner experience Enabling Service Engg,Helpdesk on new features for smooth support and quality experience to partners.Self serve tools to enable RSCs to resolve data issues with Quick TAT(without engineering team interference)Stabilizing the eco-system with more modern engineering practices.Questions?Thank You