Scrum And DevOps

Joshua Partogi — April 8, 2018

— April 8, 2018

Hello great people of the world. Welcome back to Professional Scrum Developer (PSD) blog series with yours truly. This time we’re going talk about how to use Scrum And DevOps. I am interested to discuss this topic because it’s quite common I get a question from someone in the agile community, “Should I use Scrum or DevOps?”. This question deserves an attention because of the motivation to choose DevOps over Scrum. People want to go with DevOps over Scrum because they want to be agile but they can’t change their organisation. Many people in the agile community whom I’ve interacted with think that DevOps is only about toolings for continuous delivery. We’re going to learn why DevOps is not only about tooling for delivery pipeline. Unlike my previous article about Scrum and eXtreme Programming, this time we’re going to learn how to integrate Scrum And DevOps starting from DevOps lense.

Scrum is just a framework

Scrum is just a simple framework for product development that is based on values. Scrum is highly focused on what is happening during the Sprint. Scrum is an additive framework, what that means is Scrum will only tell you the minimum sets on what you need to have so that you can claim you are using Scrum. Scrum will not tell you how your process will look like within the Sprint. So with this premise, there is nothing wrong with adding additional practices that will enhance the flow of your value delivery on top of the Scrum Framework.

Lean Thinking, Systems Thinking and Value Stream Mapping

DevOps starts from Systems Thinking and view the whole value stream in the system rather than only zooming into development process only. Systems Thinking views how every interconnected element in a system affects one another. In a complex system like a corporate, elements do not work in isolation. Making one change in an element is going to impact another element in the system. Value stream how processing customer requests into a tangible outcome flow from one element to another element in the whole system. Whenever there is a request, there is a value stream. Besides Systems Thinking, DevOps is based on Lean Thinking. Lean Thinking is about reducing waste in the value stream. Waste are non-value added activities.

Lean Thinking, Systems Thinking and mapping the whole value stream is important and works with Scrum. In fact, Scrum is based on Lean Thinking. This is what I do before starting Scrum in a large corporate, view the whole system holistically and map the whole value stream in the system. Kanban is a good tool to visualise the whole value stream in the system.

When we’re using Scrum And DevOps, all of the activities in the value stream, from customer request to release to production happens within a Sprint. This does not mean a Sprint is a mini-waterfall where deployment only happens at the end of the Sprint and all of the analysis happens at the beginning of the Sprint. Using Scrum with Kanban helps to get out of using Sprint as mini-waterfall. In this article, I am not going to talk about Scrum and Kanban.

Scrum Team applying DevOps: The Composition

Scrum Teams applying DevOps Three Ways will have a different way of working to Scrum who are not adopting DevOps. Not only their way of working is different, the team composition is different.

The Development Team consists of professionals who do the work of delivering a potentially releasable Increment of “Done” product at the end of each Sprint.
– Scrum Guide

Scrum says that the Development Team consists of professionals who delivers the potentially releasable increments. As DevOps views the whole value stream and uses Systems Thinking, the professionals in Scrum Team adopting DevOps are everyone who processes the Product Backlog Item in the whole value stream from end-to-end. This includes everyone, but not limited to, marketing people, analysts, designers, developers, operations people, sysadmins, data scientists and site-reliability engineers. They all work together as one unit to deliver value to their customers.

DevOps Three Ways

The DevOps Three Ways are the set of underpinning principles that make up DevOps based on Lean Thinking and Systems Thinking. DevOps Three Ways work with Scrum as the Three Ways is not about specific tools and practices, that are often more emphasised during any discussion about DevOps in the communities. Scrum Teams applying DevOps Three Ways will have a different way of working with Scrum who is not applying it. None of the DevOps Three Ways are in conflict with the Scrum Framework and Scrum Values. The practices I elaborate here are not a complete list of practices to implement the DevOps Three Ways.

1. Optimise Flow

The First Way in the DevOps Three Ways is Optimise Flow. In DevOps, we are concerned about optimising the flow of single Product Backlog Item since the first time customer requested it until the customer get the requested PBI in forms of tangible item (i.e feature) in the production environment. Anything that gets in the way to make the PBI flows smoothly in the value stream may be a bottleneck and should be removed. Many people in the agile communities believe that flow contradicts with Scrum’s Sprint model because:

1. You plan for the whole PBI for the Sprint during the Sprint Planning.
2. You can only deliver to production once after the Sprint Review.

We will see why nothing in Scrum in contradicts with flow-based model.

In Scrum, the Scrum Master is the role responsible to remove anything that impedes the flow of value delivery. When the Scrum Team decides to adopt flow-based model, the Scrum Master coaches the whole Scrum Team on how Scrum works with flow-based model.

1. Sprint Planning

… enough work is planned during Sprint Planning for the Development Team to forecast what it believes it can do in the upcoming Sprint. Work planned for the first days of the Sprint by the Development Team is decomposed by the end of this meeting, often to units of one day or less.
– Scrum Guide

Scrum Team adopting flow-based model will have a different nature of Sprint Planning. Nothing in Scrum Guide states that the Scrum Team fixed the plan during Sprint Planning. The Sprint Planning is more focused and committed with the Sprint Goal. The Sprint Planning is an opportunity to get everyone’s heads down to look at the same goal. During Sprint Planning enough work is forecasted for the very few days of a Sprint. More work may emerge later on during the Sprint as long as it does not endanger the Sprint Goal. Having Kanban systems helps the team to manage the work that emerges later during the Sprint.

2. Continuous Delivery to Production

The heart of Scrum is a Sprint, a time-box of one month or less during which a “Done”, useable, and potentially releasable product Increment is created.
– Scrum Guide

Nothing in Scrum that says that you can only deliver to production after Sprint Review. At minimum, you need to have a “potentially” releasable product increment at the end of the Sprint. One thing that I would like highlight here is the Sprint is not a release cadence but a planning cadence. So there is nothing wrong to release the product to production environment more than once in a Sprint. Scrum Team adopting flow-based model will deliver more than once in a Sprint. In a previous article, we’ve learned that Scrum Team adopting eXtreme Programming (XP) delivers to production every night.

3. Flow Based Daily Scrum

The structure of the meeting is set by the Development Team and can be conducted in different ways if it focuses on progress toward the Sprint Goal.

Scrum Team adopting flow-based model will use the Daily Scrum to inspect its progress towards meeting the Sprint Goal. Of course new Sprint Backlog may be added as the team learn more about their progress. The Scrum Master will work to remove anything that is impeding flow that is discovered during Daily Scrum.

4. Definition of Done

Scrum Team adopting DevOps will have a more rigour Definition of Done compared to Scrum Team not adopting DevOps. Some practices they may have in their Definition of Done that will optimise flow are:

• Infrastructure as Code so that the team ensures they have production-like environment at every stages in the value stream.
• Automated Testing that will improve flow from Unit Testing, Integration Testing to Acceptance Testing.
• Continuous Integration.
• Continuous and Automated Deployment.
• Decouple deployments from release using techniques like canary release or feature toggling.
• Architect for low-risk releases like using Microservices.

Some of the practice listed here are already implemented by Scrum Teams implementing eXtreme Programming (XP). The list above are not necessarily a complete list of practices to optimise flow as I believe there will be more practices that may be discovered by the community in the future.

2. Amplify Feedback

The Second Way in the DevOps Three Ways is Amplify Feedback. Scrum is all about feedback. At its core, Scrum has the Sprint and the Daily Scrum as built-in feedback loops. Scrum Team adopting DevOps will have a different nature of implementing the feedback loops.

1. Sprint Review

Scrum Team adopting DevOps will have a different way of running Sprint Review because the product is already in the production environment. So instead of giving feedback about the product increment, the Scrum Team will look at a single dashboard of metrics that covers business level metrics, application level metrics, infrastructure level metrics and deployment pipeline metrics. By looking at a holistic view instead of isolated view metrics, the whole business and the Scrum Team is able to give a sound judgment about the performance of the product in the market and gives feedback on what strategy they should apply in the next Sprint.

2. Hypothesis-Driven Development and A/B Testing

Scrum Team adopting DevOps will implement Hypothesis-Driven Development. For them “Done” is not about feature complete because feature completed does not mean the product is successful in the market. For them “Done” is when the feature gained traction in the market.The whole Scrum Team works together to ensure that every feature delivered is valuable for customers.

3. You ship it, you manage it

Developers in a Scrum Team adopting DevOps has a higher responsibility and need to learn about maintaining the application in production. To amplify feedback and reduce wasteful activities, developers in a Scrum Team adopting DevOps get to maintain the application they develop. The practice is, you ship it, you get to manage it. In some organisation this goes as far as giving time-shifts to developers to attend to production incidents and outages support call.

4. Pair programming and code review

From my previous article we have learned that Scrum Team implementing eXtreme Programming religiously will already have done pair programming. We have learned that pair programming is not about two programmers doing programming together on one computer but more about live feedback on how the code will actually work in production.

5. Daily Scrum as feedback loop

Daily Scrum is all about feedback. Daily Scrum is not about a status report. In a complex unpredictable environment, feedback is very important. Having feedback loops nested within another feedback loop is essential in a complex environment. The feedback in Daily Scrum will amplify the feedback for the Sprint. The team will learn how they are progressing towards the Sprint Goal.

The list above are not necessarily a complete list of practices to amplify feedback as I believe there will be more practices that may be discovered by the community in the future.

3. Maximise Learning and Experimentation

The Third Way in the DevOps Three Ways is Maximise Learning and Experimentation. The heart of Scrum is about continuous learning because Scrum is based on empiricism. The purpose of having Sprints is to maximise learnings and to improve how Scrum Teams will operate and deliver value in the next Sprint. Sadly, many organisations without a clear understanding of Scrum implement Sprint as a mini-waterfall and fix the scope for the Sprint.

1. Psychologically-safe and blameless environment

Great Scrum Master will focus and invest more time in injecting the Scrum Core Values into the Scrum Team rather than the Scrum mechanics. They know that the Scrum core values will have contribute to create a psychologically-safe and a blameless environment.

In DevOps, everyone works together to ensure that the product is valuable and meets the business goal. In a psychologically-safe and blameless environment there should be no politics and silos. Everyone in the value stream looks at the same holistic product metrics regardless of their role. A blameless culture is important so that everyone in the value stream be collaborative and not just throwing work over each one’s shoulder. Any incentives that makes everyone only cares about their metrics should also be removed. This is more about organisation refactoring practice than it is about technical practice.

2. On-demand retrospectives

The Scrum Master encourages the Scrum Team to improve, within the Scrum process framework, its development process and practices to make it more effective and enjoyable for the next Sprint.
– Scrum Guide

One misconception about Scrum is that retrospectives or learnings only happen at the end of the Sprint. In a Scrum Team adopting flow-based model, learning needs to happen just-in-time also. As we’ve learned that Scrum is an additive process. Everything in Scrum is a minimum set. Sprint Retrospectives provides a focused time to reflect back and to learn about what is happening in the Sprint. But that does not mean that Scrum does not allow you to learn multiple times in a Sprint. Many Scrum Teams have learnings during the Daily Scrum as they see Daily Scrum is about inspecting and adapting.

3. Slack time for improvements

To ensure continuous improvement, it includes at least one high priority process improvement identified in the previous Retrospective meeting.
– Scrum Guide

Many managers or organisations still see Sprints as a mini-waterfall and only about delivering features. People working in this organisations come to think that Scrum does not provide time for improvements. This is not true. In fact Scrum Guide states that at least one item in the Sprint Backlog must contain improvements identified during the previous retrospectives. Scrum Team adopting DevOps will go further by deliberately providing slack time to innovate as much as 20% time (or more) to improve the value of delivery in the value stream.

The list above is not necessarily a complete list of practices to maximise learnings as I believe there will be more practices that may be discovered by the community in the future.

Closing: DevOps is more about Organisation Culture

As you can see, DevOps is not about tools and automation in the delivery pipeline. In fact as we have learned that is only one third of DevOps. Overall DevOps is about Collaboration & Collective Ownership, Focus of the flow of value delivery and Learning and experimentation culture. But sadly, many tooling vendors position DevOps about tools and process for delivery pipeline (the vendors that I’ve witnessed in my market focus on tools but your experience may be different to mine). This will get the management excited because many managers whom I’ve met think that buying and installing the tools without changing their organisation will make their company agile.

From this article we’ve seen that Scrum and DevOps actually share more in common than most realise. Just like how Scrum is more about values and principles, the DevOps Three Ways is also about values and principles.

Hopefully this article along with the visual helps you understand how Scrum and DevOps work together and how they do not contradict each other. Looking forward to hearing you exploit Scrum and DevOps to deliver great products to your customers. If your team want to learn how to use Scrum and DevOps check out Professional Scrum Developer course.

Thank you to Jesse Houwing, Fredrik Wendt and everyone in the Professional Scrum Developer Trainer community for assisting me on the creation of this article.

Business & Finance Articles on Business 2 Community

(85)