Continuous Integration Match Backend to Frntedn

Software Production Management Agile/Continuous Integration Backend infrastructure to realize Agile software development methodologies November 18, 2008 Jeff Price Account Manager Dave Finnegan Systems Engineer www. electric-cloud. com

What am I going to tell you today? Four practical steps to support Agile / Continuous Integration Impacts on the Build/Integration team Best practices you can adopt right now Commercially-available tools to overcome these impacts 02 December 2020 Slide 2

Practical steps for Agile / CI One : Enable developers to build small pieces of functionality Two: Automate the full integration build Three: Perform an integration build for each SCM commit Four: Minimize broken builds and maximize integrated testing of new features 02 December 2020 Slide 3

Impacts on Build/Integration One : Enable developers to build small pieces of functionality Problem : Developers work on isolated components; Integration becomes a trouble spot Two: Automate the full integration build Three: Perform an integration build for each SCM commit Four: Minimize broken builds and maximize integrated testing of new features 02 December 2020 Slide 4

Impacts on Build/Integration One : Enable developers to build small pieces of functionality Problem : Developers work on isolated components; Integration becomes a trouble spot Two: Automate the full integration build Problem : DIY build scripts are fragile and expensive Three: Perform an integration build for each SCM commit Four: Minimize broken builds and maximize integrated testing of new features 02 December 2020 Slide 5

Impacts on Build/Integration One : Enable developers to build small pieces of functionality Problem : Developers work on isolated components; Integration becomes a trouble spot Two: Automate the full integration build Problem : DIY build scripts are fragile and expensive Three: Perform an integration build for each SCM commit Problem : Number of builds and total build time explodes Four: Minimize broken builds and maximize integrated testing of new features 02 December 2020 Slide 6

Impacts on Build/Integration One : Enable developers to build small pieces of functionality Problem : Developers work on isolated components; Integration becomes a trouble spot Two: Automate the full integration build Problem : DIY build scripts are fragile and expensive Three: Perform an integration build for each SCM commit Problem : Number of builds and total build time explodes Four: Minimize broken builds and maximize integrated testing of new features Problem : keeping the main line buildable and clean 02 December 2020 Slide 7

Overcoming Impacts Problem : Developers work on isolated components; Integration becomes a trouble spot Problem : DIY build scripts are fragile and expensive Problem : Number of builds and total build time explodes Problem : keeping the main line buildable and clean 02 December 2020 Slide 8

Give the developers access to the build! Put full builds in the hands of developers Set up an internal web page that allows a developer to fire off a build The developer should be able to select: Which branch to build… … or the location of his personal source tree Whether to run unit tests or not Select platforms The build runs and the engineer gets an email with the result 02 December 2020 Slide 9

The Problems for the Build team… Problem : Developers work on isolated components; Integration becomes a trouble spot Problem : DIY build scripts are fragile and expensive Problem : Number of builds and total build time explodes Problem : keeping the main line buildable and clean 02 December 2020 Slide 10

Script your build the way you write code! Design generic, modular build elements which can be reused Make use of parameterization Develop or apply Continuous Integration systems (start with cron) Standardize on a single set of build tools Pick a scripting language and stick with it Enforce SCM for build scripts Code review for build scripting Know when to stop fiddling 02 December 2020 Slide 11

The Problems for the Build team… Problem : Developers work on isolated components; Integration becomes a trouble spot Problem : DIY build scripts are fragile and expensive Problem : Number of builds and total build time explodes Problem : keeping the main line buildable and clean 02 December 2020 Slide 12

Make the build fast, automatic, and visible Achieve fast builds That means better build structure Fast build hardware, special software for parallel builds gmake -j may get you 2 -3 x improvements Schedule builds through automation cron for CI builds Web front-ends and request systems for developer builds Investigate better build reporting Collect and publish build result information Build status Collect metrics: errors, warnings, tests, etc Include changelogs Build tracking repository 02 December 2020 Slide 13

The Problems for the Build team… Problem : Developers work on isolated components; Integration becomes a trouble spot Problem : DIY build scripts are fragile and expensive Problem : Number of builds and total build time explodes Problem : keeping the main line buildable and clean 02 December 2020 Slide 14

Guard the mainline and create policies Private Branches Give developers private branches, or multi-stage CI streams Integration Builds Require developers to run integration builds before merging SCM Triggers Use SCM hooks to fire an integration build for every checkin Automate Checkin Reverts Integrate SCM and CI to revert checkins which break the build Communicate Status Use notification, dashboards, or lava lamps to communicate mainline status 02 December 2020 Slide 15

Going Beyond DIY 02 December 2020 Slide 16

The Build Manager 02 December 2020 Slide 17

Improving on in-house I need fast builds That means better build structure Fast build machines, special software for parallel builds gmake -j doesn't give the speedup up I need I'm doing 200 builds a day, I need better tools cron just won't cut it I need resource management tools I need developers to build in the production environment I need reporting tools Everyone wants live build feedback Perl + Excel isn't enough I need this because builds are now the heartbeat of software engineering 02 December 2020 Slide 18

Electric Cloud SPM Solutions Electric. Insight Visualize Builds Electric. Accelerator 10 -20 x Faster Builds Electric. Commander Automate Production Processes SCM Tools Perforce, Subversion, Clear. Case, Accu. Rev, etc 02 December 2020 Build Package Test Deploy Build Tools Package Tools Test Tools Deploy Tools Make, Ant, Maven, etc. Installers, etc. Unit Test Tools, Code Coverage Misc. Slide 19

Improving on in-house I need fast builds That means better build structure Fast build machines, special software for parallel builds gmake -j doesn't give the speedup up I need I'm doing 200 builds a day, I need better tools cron just won't cut it I need real management tools I need developers to build in the production environment I need reporting tools Everyone wants live build feedback Perl + Excel isn't enough I need this because builds are now the heartbeat of software engineering 02 December 2020 Slide 20

Electric. Accelerator Problem it solves: Build cycle is too long Very long builds Complex matrix of builds and targets Benefits: 10 -20 x faster builds How it works: Run builds in parallel using a cluster of standard hardware Automatically discover and track dependencies Plug-compatible replacement for GNU make, Microsoft nmake, Visual Studio, Ant No process changes required Works with Electric. Commander or stand-alone 02 December 2020 Based on proven, patented dependency Slide 21 management technology

How Do You Get There? Requirements for real build acceleration: ü Fast and accurate builds via parallelism • Requires understanding of dependencies ü Robust and scalable • Support high build volume and avoid resource failure ü Work within existing environment • Must not require costly re-write ü Visibility and reporting • Provide status reports and tools for diagnostics 02 December 2020 Slide 22

Managing Dependencies for Parallel Builds have inherent parallelism Should be able run pieces concurrently Large SMP Machines (gmake –j) Distributed builds (distcc) Other attempts yield small results due to dependencies: Source Code Object Files Libraries Executables 01010 10101 01010 10101 01010 10101 01010 10101 01010 10101 01010 10101 Incomplete or unknown Can't be expressed between Makefiles Result: broken builds 02 December 2020 Release Slide 23

Solution for Fast, Accurate Builds Deduce dependencies on-the-fly: Watch all file accesses: these indicate dependencies Automatically detect out-of-order steps Desired class. cpp read class. obj app. exe 10101010 10101010 101010 10101 write Compile source Actual read write Link application class. cpp read write 10101010 1010101 class. obj app. exe 10101010 10101010 101010 10101 old! 02 December 2020 class. obj read write Compile and link steps run in parallel Slide 24

Solution for Fast, Accurate Builds Deduce dependencies on-the-fly: Watch all file accesses: these indicate dependencies Automatically detect and correct out-of-order steps Save discovered dependencies for future builds Result: high concurrency is safe Desired class. cpp read class. obj app. exe 10101010 10101010 101010 10101 write Compile source Actual read write Link application class. cpp read app. exe 10101010 10101010 101010 10101 write read write Rerun link class. obj app. exe 10101010 10101010 101010 10101 old! 02 December 2020 class. obj read write Discard Slide 25

Electric. Accelerator Architecture Network Production Machines Build with Electric. Make instead of GNU Make, Microsoft NMAKE, Visual Studio Cluster Manager Agent Developer Machines 02 December 2020 Agent … User-level Kernel Build Cluster (10 -1000 CPUs) Electric File System Node Slide 26

File Caching for Network Optimization Make Machine Electric Make Network bandwidth concentrates at make machine Network Node Agent Electric File System File Cache Peer-to-peer file transfer File caching on nodes cuts 90 -95% of network traffic Just-in-time compression cuts traffic 2. 5 -3 x P 2 P file transfers offload 20 -25% of outbound traffic 02 December 2020 Slide 27

Linear Scalability 16. 4 x 11. 3 x 9. 4 x Samba My. SQL Gtk 02 December 2020 Local 16 m 23 m 15 m 20 CPUs 58 s 124 s 95 s Speedup 16. 4 x 11. 3 x 9. 4 x Slide 28

Works Seamlessly in Environment Works seamlessly with existing infrastructure GNU make, Nmake, Visual Studio No learning a new interface/tool Use existing Makefiles and scripts Identical log files Works with grid system Use dedicated cluster of servers or central data center Fully integrated with the Platform LSF grid computing solution Provide build resources just when needed Minimal changes required for end users 02 December 2020 Slide 29

Improving on in-house I need fast builds That means better build structure Fast build machines, special software for parallel builds gmake -j doesn't give the speedup up I need I'm doing 200 builds a day, I need better tools cron just won't cut it I need real management tools I need developers to build in the production environment I need reporting tools Everyone wants live build feedback Perl + Excel isn't enough I need this because builds are now the heartbeat of software engineering 02 December 2020 Slide 30

Electric. Commander Problem it solves: Manual, inflexible build and test infrastructure Web-based platform for managing distributed processes Faster, more efficient build and release processes More efficient resource utilization Better visibility and predictability Can standardize throughout an organization Job Procedure Setup Step Job Schedule/Trigger Step Snapshot Step Compile Step Unit Step Test Step Report Step Step Job Step Step Step Pool Linux server 02 December 2020 Linux server Resources Windows server Test harness Slide 31

Managing Global Teams SW DEVELOPERS Munich SW DEVELOPERS San Francisco ENGINEERING MGR Boston BUILD TEAM San Francisco OUTSOURCED QA Bangalore Specific Access and Permissions Based on Role Slide 32 32 Anywhere in the World 02 December 2020

Managing Resources VMware Lab Manager ENGINEERING MGR SW DEVELOPERS BUILD TEAM Dynamic Provisioning VMware Lab Manager SW DEVELOPERS 02 December 2020 Build Servers OUTSOURCED QA Test Servers Production Servers Virtual Servers Slide 33 33

Integrate Tools and Processes Deployment Tools SW DEVELOPERS SCM Tests Tools Unit Build Tools. BUILD TEAM ENGINEERING MGR Build Tools Unit Tests Deployment Tools SCM Tools 02 December 2020 OUTSOURCED QA Build Servers Test Servers Production Servers Virtual Servers Slide 34

Tie it all Together SW DEVELOPERS ENGINEERING MGR BUILD TEAM Build Tools Unit Tests Deployment Tools SCM Tools 02 December 2020 OUTSOURCED QA Build Servers Test Servers Production Servers Virtual Servers Slide 35 35

Electric. Commander Basics Container for processes and results Job Project Procedure When to run a procedure Setup Step Process to run automatically Snapshot Step Job Job Step Step Schedule/Trigger Step Step Compile Step Step Command or script Step Unit Test Step Step Report Step Pool Linux server 02 December 2020 Linux server Resources Linux server Windows server Test harness Records results of running a procedure Slide 36 36

Continuous Integration QA Build Success electriccommander Check-In Developer runs local build and smoke tests Developer checks tested code into SCM system Integration build at frequent intervals or upon check-in SCM System Production Servers Integration Build 02 December 2020 Slide 37

Frequent Problem: Continuously Broken Builds Developer builds/tests on local system, checks in code Wait X Check-In electriccommander SCM System Back Out X Wait Production Servers Integration Build Integration build started, breaks ("it worked on my machine") Team impacted while check-in is backed out or build fixed Build Failure 02 December 2020 Slide 38

Solution: Pre-Flight Builds and Tests Developers build and test across all targets/platforms electriccommander Success Test Servers Production Servers Developers SCM System Ensures successful integration build Developers can check in changes with confidence Broken builds less likely to affect the entire team Build or Test Failure 02 December 2020 Slide 39

Preflight Workflow 1. Developer invokes preflight through Visual Studio, Eclipse or command line if {x<y} … Developer changes Electric. Commander IDE Preflight Result if {x<y} 1. 2. Preflight result displayed in IDE … if {x<y} … func {y}; 1. Developer changes and source snapshot uploaded to Electric. Commander 2. Electric. Commander overlays the changes on top of the snapshot 3. Electric. Commander runs a build as if the developer had checked-in the code Source snapshot SCM func {x}; func {y}; Autocommit checks in successful changes to SCM (optional) +

Improving on in-house I need fast builds That means better build structure Fast build machines, special software for parallel builds gmake -j doesn't give the speedup up I need I'm doing 200 builds a day, I need better tools cron just won't cut it I need real management tools I need developers to build in the production environment I need reporting tools Everyone wants live build feedback Perl + Excel isn't enough I need this because builds are now the heartbeat of software engineering 02 December 2020 Slide 41

Pinpoint Data for Reporting Extract data from log files, create reports Job Step tests: 217 Log File Patterns xyzzy cl. *warning *error* xxxx Postprocessor errors: 2 Properties Problem: Electric. Commander solution: • • Large, complex log files Only a small amount of data matters Log files get deleted • 02 December 2020 Extract key data from log: • Tests executed • Errors • Diagnostic messages Save in database Slide 42 42

Visibility: Management Reports 02 December 2020 Slide 43

Improving on in-house I need fast builds That means better build structure Fast build machines, special software for parallel builds gmake -j doesn't give the speedup up I need I'm doing 200 builds a day, I need better tools cron just won't cut it I need real management tools I need developers to build in the production environment I need reporting tools Everyone wants live build feedback Perl + Excel isn't enough I need this because builds are now the heartbeat of software engineering 02 December 2020 Slide 44

Builds are too expensive to ignore! Multi-Platform Gaming Company Problem: Challenging, complicated build-test matrix (Xbox, PS 3, PC) Slow builds kept QA waiting and Developers avoiding p. m. checkins Initial Continuous Integration approach led to 300 broken builds per month Solution: Electric. Accelerator + Electric. Commander Build acceleration led to more frequent builds and tests Build time from 2. 5 hours to 12 minutes Developer "preflight" builds reduced broken builds by 90% Preflight build and test run on 3 platforms prior to check-in 10% due to developers not doing preflight 02 December 2020 Slide 45

Customer Examples Networking 02 December 2020 Semiconductor Cellular ISV Other Slide 46 46

What we have covered today Four practical steps which are required to support Agile Impacts of these steps on the Build/Integration team Best practices you can adopt right now to handle these impacts Commercially-available tools which overcome these impacts 02 December 2020 Slide 47

Thanks for attending For more information: Visit our website: www. electric-cloud. com E-mail: [email protected] com Phone: 408. 419. 4300 02 December 2020 Slide 48

Questions and Answers Backend infrastructure to realize Agile software development methodologies www. electric-cloud. com

Supplemental Slides Build Your Own www. electric-cloud. com

Build Your Own? Investment Seems pretty straightforward… Automated build and test for one build tool, one machine Scheduled jobs Features Trigger builds after check-ins 02 December 2020 Slide 51

Build Your Own? Investment Just a few finishing touches Automated build and test for one build tool, one machine Scheduled jobs Web-based access to results Features Trigger builds after check-ins 02 December 2020 Notifications via e-mail/RSS Database of build results Slide 52

Build Your Own? Investment Multiple servers, concurrency Notifications via e-mail/RSS Automated build and test for one build tool, one machine Scheduled jobs Trigger builds after check-ins 02 December 2020 Run multiple builds simultaneously Web-based access to results Database of build results Resource pooling, load balancing Multiple jobs run Features on one resource Resource selection criteria simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Slide 53

Build Your Own? Investment Better monitoring and control Monitor system status Notifications via e-mail/RSS Automated build and test for one build tool, one machine Scheduled jobs Trigger builds after check-ins 02 December 2020 Run multiple builds simultaneously Cancel jobs On-demand job invocation via Web-based access to results Database of build results Resource pooling, load balancing View partial results of builds in progress Multiple jobs run Features on one resource Resource selection criteria simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Slide 54

Build Your Own? Investment Better reporting Monitor system status Notifications via e-mail/RSS Automated build and test for one build tool, one machine Scheduled jobs Trigger builds after check-ins 02 December 2020 Run multiple builds simultaneously Cancel jobs On-demand job invocation via Web-based access to results Database of build results Resource pooling, load balancing View partial results of builds in progress Multiple jobs run Features on one resource Resource selection criteria Customizable reports Trend reports Cross-product reports simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Tools for extracting data from logs Annotate builds after completion Resource utilization reports Slide 55

Build Your Own? Investment Better error handling Monitor system status Notifications via e-mail/RSS Automated build and test for one build tool, one machine Scheduled jobs Trigger builds after check-ins 02 December 2020 Run multiple builds simultaneously Retry steps after errors/ failures Detect resource failures Cancel jobs On-demand job invocation via Web-based access to results Database of build results Resource pooling, load balancing View partial results of builds in progress Multiple jobs run Features on one resource Resource selection criteria Customizable reports Logging and error reporting Timeouts for runaway job steps Trend reports Cross-product reports simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Tools for extracting data from logs Annotate builds after completion Resource utilization reports Slide 56

Build Your Own? Investment Developer access to production builds Shared access by multiple teams LDAP authentication Priorities Role-based access control Notifications via e-mail/RSS Automated build and test for one build tool, one machine Scheduled jobs Trigger builds after check-ins 02 December 2020 Monitor system status Resource pooling, load balancing View partial results of builds in progress Run multiple builds simultaneously Retry steps after errors/ failures Detect resource failures Cancel jobs On-demand job invocation via Web-based access to results Database of build results User impersonation, password management Handle multiple independent groups Multiple jobs run Features on one resource Resource selection criteria Customizable reports Logging and error reporting Timeouts for runaway job steps Trend reports Cross-product reports simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Tools for extracting data from logs Annotate builds after completion Resource utilization reports Slide 57

Build Your Own? Investment Developer access to production builds Shared access by multiple teams LDAP authentication Notifications via e-mail/RSS Scheduled jobs Trigger builds after check-ins Database of build results 02 December 2020 Parameters Monitor system status Resource pooling, load balancing Retry steps after errors/ failures Detect resource failures Cancel jobs On-demand job invocation via Web Multiple jobs run Features on one resource Control environment for jobs Conditional steps View partial results of builds in progress Run multiple builds simultaneously Web-based access to results Modular, composable process steps Extensibility Priorities Role-based access control Automated build and test for one build tool, one machine User impersonation, password management More programming features Resource selection criteria Customizable reports Logging and error reporting Timeouts for runaway job steps Trend reports Cross-product reports simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Tools for extracting data from logs Annotate builds after completion Resource utilization reports Slide 58

Build Your Own? It Never Ends Investment Developer access to production builds Shared access by multiple teams LDAP authentication Notifications via e-mail/RSS Scheduled jobs Trigger builds after check-ins 02 December 2020 Web interface for editing processes Command-line interface Portable across hardware/OS platforms Modular, composable process steps Extensibility Parameters Monitor system status Resource pooling, load balancing Retry steps after errors/ failures Detect resource failures Cancel jobs On-demand job invocation via Web Multiple jobs run Features on one resource Control environment for jobs Conditional steps View partial results of builds in progress Run multiple builds simultaneously Web-based access to results Database of build results User impersonation, password management SCM integration, bill of materials Priorities Role-based access control Automated build and test for one build tool, one machine Support multiple languages and build tools Resource selection criteria Customizable reports Logging and error reporting Timeouts for runaway job steps Trend reports Cross-product reports simultaneously Multiple servers, remote invocation Run job steps in parallel Single job can use multiple resources Tools for extracting data from logs Annotate builds after completion Resource utilization reports Slide 59

Questions and Answers www. electric-cloud. com

stumpfrente.blogspot.com

Source: https://slidetodoc.com/software-production-management-agilecontinuous-integration-backend-infrastructure-to/

Share this post