Agile Cheetah just added a YouTube channel to our list of platforms. This is another way for us to share knowledge about Agile, DevOps, and organizational agility.
You can subscribe to the channel directly from the watermark link in lower right-hand corner of the video below. Visit the channel and don’t forget to like the video and click on the notification bell to ensure you don’t miss future videos!
The Never-Ending Agile Grind
Table of Contents
Agile has become the way we build software. Across industries, government institutions, and non-profits, it is the default software development approach. Debates over whether Agile should supplant traditional project management approaches to software development (i.e., Waterfall) have long been passé. The focus now is on accelerating the product development lifecycle through DevOps.
However, all is not well in “Agileland”. Many organizations are not enjoying the benefits promised by the Agile gurus. Many software developers are fed up with Agile. If Agile is so great, why are so many people, especially developers, souring on it? The bottom line upfront (BLUF) is that what passes for Agile in many places is anything but Agile.
This article is the first of a four-part series that describes the most common pain points experienced by software developers and other technical contributors engaged in Agile software development, the root causes, and potential solutions. The pain points covered in the series are:
Software development is a demanding intellectual activity. It requires study, perseverance, and focus. The work requires time to think and experiment. Some degree of failure is necessary as long as it yields valuable insights and teams use those insights to continually improve the product and how they work.
The experience of too many developers on Agile teams, however, is working in “feature factory” environments and “death march” projects. They feel pressured and rushed to deliver code to the point of sacrificing quality.
Sprints become Sisyphean ordeals where overburdened teams are under constant pressure to deliver working code as quickly as possible while also maintaining the existing code base and contributing DevOps pipeline automation components. The end of one Sprint leads to the beginning of another in a seemingly never-ending loop of stress that eventually leads to burnout.
Teams are not afforded time to think through and experiment with alternative approaches, leading them to commit to suboptimal approaches that will likely take longer to implement and maintain than more elegant approaches they could have conceived given sufficient time.
“Feature factory” teams are denied time “on the clock” to learn and experiment with new technologies, design patterns, and architectural approaches. As a result, implementations based on those approaches are often half-baked, filled with technical debt, under-performant, and difficult to maintain. The benefits expected from adopting new technological approaches are not realized. Meanwhile system complexity explodes exponentially.
Management consistently chooses delivery of new functionality over addressing technical debt. This short-sided preference leads to increasingly brittle systems that force developers to expend increasing amounts of energy and focus on fixing bugs and babysitting systems.
Overtime, developers start to feel like robotic coding units who get paid to crank out code. A mentality of “getting it done on time no matter what” becomes culture. Any vestige of work/life balance is thrown out the window as the backlog of features grows and the system or product becomes ever more unmanageable.
The Agile Manifesto for Software Development and the 12 Principles of Agile describe the characteristics of Agile software development. Software development approaches that adhere to the values of the Manifesto and the 12 principles are considered Agile. Approaches that deviate from any of the values or principles are not. One Agile principle “feature factory” teams violate is principle number #8:
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
http://agilemanifesto.org/principles.html
Why do so many software development organizations and their sponsors ignore this principle? Many blame the increasing pace of business competition or mission threats for this single-minded focus on production and speed. While pressure from increased competition is a contributing factor, I posit that the root cause is cultural. Many (if not most) of these organizations take a project management approach to software development. Agile, in general, and Scrum in particular, have a product management orientation.
Traditional project management, rooted partly in Taylorism, focuses on managing resources, schedule, and budget. People are a category of resources managed by managers. Traditional project management places a very high value on maximizing resource utilization to achieve greater efficiencies. People are the most important (and expensive) resources to maximize. In the drive to do more with less and meet tight (often arbitrary) deadlines, management often loses sight of the link between team welfare and morale and superior performance.
Product management focuses on managing the product and how the product is made and delivered. Through Agile, management shifts its focus from micromanaging developers to collaborating with them in continually improving product offerings as well as the development and delivery of those offerings. Ensuring development teams are not overwhelmed becomes a major concern since it has such a large bearing on product quality, timeliness, cost, etc.
Another cultural factor that leads to “feature factory” cultures is confusion over the role of planning in Agile. A popular misconception is that there is no planning in Agile software development. This misunderstanding leads to underemphasizing requirements definition, analysis, and design which, in turn, leads to badly conceived and executed software implementations.
Agile teams plan constantly. The difference between Agile planning and planning in Waterfall projects is that Agile planning is adaptive while traditional project planning is predictive. In other words, rather than attempting to plan the entire project upfront, Agile plans change based on current circumstances and new information and insights.
Just like development teams in Waterfall projects, Agile teams perform all the activities encompassed by each of the phases of the Software Development Lifecycle (SDLC). The difference is that the “Agile SDLC” (DevOps) does not segregate those activities into discrete scheduled phases arranged one after the other for the entire system or solution. Instead, every feature/deliverable an Agile development team delivers is the product of its own “mini” SDLC. The resulting system or solution is the sum of all integrated features.
So, what are some concrete steps software development organizations and their sponsors can take to avoid overwhelming and burning out technical contributors?
The first step is understanding and internalizing that Agile development teams are self-organized. They collectively manage their own work, determine technical approaches, develop their own estimates, report their progress, and help define the functionality they build. Team self-organization is a core tenant of Agile supported by three of the 12 Principles of Agile:
Principle #11: The best architectures, requirements, and designs emerge from self-organizing teams.
Principle #5: Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.
Principle #12: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
http://agilemanifesto.org/principles.html
This contrasts with traditional project management practice, in which management is responsible for defining what to build, breaking down and assigning work, developing plans and schedules, and reporting progress.
Agile teams break down their work with an eye towards delivering valuable functionality within short timeframes. They can only achieve this if they align the amount of work they commit to with the team’s ability to get it done. In other words, break down solution scope into slices of capability (features) that can be accomplished within agreed timeframes. By ordering/prioritizing features along the lines of business or mission value, Agile releases deliver the most valuable features/capabilities first.
Agile teams commit to additional work as they complete existing work. Rather than having work pushed on Agile teams, Agile teams pull work when they are ready to do it. Software development teams that do not have control over the amount of work they commit to are not Agile teams.
One way to ensure that Agile teams address technical debt, and other necessary tasks outside the scope of developing new features, is to allocate some percentage of their time to those activities. For example, during Sprint planning, a Scrum Team may allocate some percentage of the next Sprint to addressing technical debt. They could do the same for backlog refinement and DevOps automation work.
The Scaled Agile Framework (SAFe) includes Innovation and Planning (I&P) iterations (Sprints). I&P iterations occur at the end of each Program Increment (PI), which lasts eight to twelve-weeks. Each PI includes two to six iterations depending on the length of each iteration (two to four weeks). Agile teams do not have to adopt SAFe to experiment with I&P iterations or Sprints.
I&P iterations provide teams time to work important tasks, such as the ones below, without significantly disrupting the flow/cadence of development and release of new functionality:
It is common knowledge that productivity, quality, and innovation suffer when workers are stressed and burned out. However, too often responsibility for coping with stressful work environments is placed solely on the shoulders of individual employees/contributors while management sticks to outdated ideas about maximizing efficiency.
Agile, implemented in the spirit of Agile, opens the door to sustainable progress and achievement. Again, development teams that do not control the amount of work they commit to delivering are not Agile teams. Such teams risk being pressured into over-promising while under-delivering.
In the next article of this series, I’ll discuss how the greater visibility into team activities and progress afforded by Agile is often used to against the development teams themselves. Just like this article I will also discuss root causes for the dysfunction and suggest solutions.
Achieve effective Product Backlog ordering by optimally balancing five competing factors
Note: This article includes a video detailing how to calculate Weighted Shortest Job First (WSJF) values.
Table of Contents
Product Backlog “prioritization” is a popular topic in Agile and Scrum forums. It seems like a simple concept. Teams prioritize the most valuable or important product backlog items or features and begin work on the highest priority items. However, teams quickly discover that there is much more to prioritizing backlog items than that.
The difficulty boils down to determining what is more or less valuable or important when every item addresses valid needs. How do we deal with the ambiguity of determining what is “most valuable” or “more important”? How do we account for dependencies and other factors that influence what we build and when? How do we do so in a consistent, non-biased way that ensures delivery of valuable solutions, cost effectively, and in time to be most valuable?
For the uninitiated, before reading more about Product Backlog prioritization, I recommend reading our article on Product Backlog refinement. At a minimum, understanding what a Product Backlog is and what Product Backlog refinement entails is crucial to understanding backlog prioritization.
Product Backlog refinement includes three main activities:
In this article, I focus on Product Backlog ordering. This article does not apply to Sprint or iteration backlogs. Sprint backlogs are solely composed of user stories that take two to four weeks to complete. These items are likely too small in scope to warrant multi-factor prioritization/ordering schemes.
As of 2011, the official Scrum Guide began defining the Product Backlog as “an ordered list” instead of a “prioritized” list. Why was this changed and what is the difference between prioritizing a list and ordering it?
As Barry Overeem states in his Scrum.org article:
The focus on ordering (over ‘prioritization’) underscores the active role that the Product Owner continuously has to play in the ordering and reordering, of the work in a manner that maximizes value.
https://www.scrum.org/resources/blog/myth-5-scrum-product-backlog-prioritized
Simply stated, many factors influence Product Backlog Ordering, not just what is considered most important from a business or mission perspective.
Non-trivial software solutions service many sponsors, stakeholders, customers, and end users. Their interests and priorities differ from, and sometimes contradict, each other. It is the job of the Product Owner (PO) to fuse the various needs, wants, and perspectives of that solution community into one complete, coherent, and cohesive product vision.
There are also project and technical realities that must be taken into account when deciding what portions of a solution to build and deliver and when. Examples:
Business or mission circumstances, solution community needs, solution requirements, and project priorities change as the solution takes shape through iterative and incremental delivery of system capabilities. A PO must lead continuous refinement and ordering of Product Backlog items to align with such changes.
The most optimal ordering of Product Backlog items is a balance between these competing factors:
How Agile defines value is different from how traditional project management defines it. A simple definition of value is whatever solution sponsors, stakeholders, customers, and end users care about or makes them happy. We discuss Agile’s concept of value further here.
A feature’s business or mission value is the value created by implementing the feature (e.g., revenue, cost savings, customer retention, prospective customers, future opportunities, mission accomplishment). Features that map closest to the product vision likely rank highest for this factor.[1]
It is common Agile practice to use relative sizing to estimate how much of a development team’s capacity will be consumed by a particular Product Backlog item. One way Agile teams perform relative sizing is by assigning story points to Product Backlog items. Story points are heuristically-determined, numerical proxies representing the combination of perceived or estimated:
This combination is often referred to as the “bigness” or “size” of a user story. The development team estimates user story points by “sizing” stories relative to each other. More complex stories typically require more time and effort to complete, thus earning more story points than less complex stories.
Just as there is a correlation between story complexity and size, there is also one between story size and the cost of implementing the story. While not a perfect correlation, it is good enough to support efficient ordering of Product Backlog items.
How soon solution capabilities or system dependencies are needed also influences Product Backlog ordering. Again, the goal of Agile is to deliver valuable capabilities, cost effectively, and in time to be valuable. Delivering a capability before it is needed risks spending resources on something that may never be needed (YAGNY: You Aren’t Gonna Need It!). Delivering after it is needed risks delivering it too late to be valuable.
Agile achieves a “just-in-time” balance by constraining planning, development, and release scope to fit within relatively short time frames (iterations/Sprints/releases) and delivering the highest ordered items first. By waiting until “the last responsible moment” to define, develop, and release solution functionality, we avoid over-engineering solutions or losing opportunities to provide value.
Software solutions are inherently complex. As such, their technical components often rely on other components to function. For example, multiple features may require a database backend to store data and state. Product Backlog ordering must take into account these dependencies as solutions grow and evolve.
Software system complexity introduces technical and project risks and opportunities. Diligent backlog refinement spreads risk management actions across the entire software development effort. For example, a team may choose to tackle riskier Product Backlog items earlier. Another may identify an opportunity to shorten the schedule by accelerating delivery of a dependency common across multiple parts of the solution. This differs from traditional project management efforts that, even when risks and opportunities are identified early, tend to accumulate risks, address them after they become issues, and miss opportunities.
Backlog ordering is the art of balancing these five factors to best facilitate Agile development and delivery of software solutions. The three prioritization techniques briefly discussed in this section are not specific to Agile or Scrum. They are well-known product management techniques.
The first two, the Kano Model and the MoSCoW Method, have narrow perspectives and reduce business or mission value considerations to simplistic categories. The third technique, Weighted Shortest Job First (WSJF), takes into account all five factors discussed in the previous section.
Professor Noriaki Kano developed the Kano Model in the 1980s. The model categorizes product features according to customer needs and expectations. In other words, what will satisfy or please customers? Many versions of the model exist. The original classifies items using five categories: Must-be, Attractive, One-Dimensional, Indifferent, and Reverse.
The Kano Model is most useful in organizations that create mostly “Must-Be” features: Features expected or taken for granted by customers. However, products should include “Attractive” and “One-Dimensional” features to succeed in the marketplace.
Using this technique, the Product Backlog would include items from every category except “Reverse”. Releases and Minimum Viable Products (MVPs) would include some set of items from those categories. As MVPs grow and evolve, more items from across those categories would be added.
The problem with this approach is that it narrowly focuses on customer needs and expectations, which is one of many factors in gauging business or mission value. However, Kano Model category definitions are useful in helping to shape thinking about the business or mission value of solution features.
The MoSCoW Method, developed by Dai Clegg, has roots in Agile software development. It is an acronym loosely based on the first letter of each of its categories: Must-Have, Should-Have, Could-Have and Won’t-Have. The categories are defined from the point of view of what a product includes or offers vis-a-vis what customers or users want. It answers the question, “What features will be included in the release or MVP?”
Similar to the Kano Model, using this technique produces a Product Backlog that includes items from every category. “Won’t-Have” items may either be postponed to future iterations or releases or dropped from the backlog altogether.
However, like the Kano Model, the MoSCoW Method focuses on one dimension of value. In the case of the MoSCoW Method, it is determining which set of features best aligns with customer and end user expectations about the product/solution.
In his book, Principles of Product Development Flow: Second Generation Lean Product Development, Don Reinertsen explained the Weighted Shortest Job First (WSJF) prioritization model. The model is used to sequence work in the most economically efficient way.
The folks at Scaled Agile Inc. explain a simplified version of Reinersten’s model here. The simplified approach consists of two formulas:
This simplified model is used in the Scaled Agile Framework (SAFe) to sequence (order) individual pieces of work or “jobs” (e.g., SAFe Features, Capabilities, and Epics) within their respective backlogs.
Unlike Reinertsen’s model, which uses monetary amounts to calculate Cost of Delay (CoD) and job duration, SAFe’s simplified model uses a relative sizing construct. The construct uses a modified Fibonacci sequence as the basis for parameter values. In calculating CoD, values for User-Business Value, Time Criticality, and Risk Reduction/Opportunity Enablement are assigned greater or smaller values relative to other jobs being ordered. For job duration, “size” values are assigned to individual jobs. Every job is sized relative to the other jobs being ordered.
We can create a simple table to track the relative values and calculate CoD and WSJF values:
The following video provides more details on calculating WSJF values:
The initial formulaic answer is a start to the ordering, not a final answer. The computed sequence of Product Backlog items is adjusted to account for technical dependencies between them.
Teams may find it helpful to keep in mind the Kano Model and MoSCoW Method categories as they assign relative business or mission values to Product Backlog items/features and determine which items are ranked too low to include in the next Sprint or release. Just remember that those categories do not represent the entirety of business or mission value.
We can use WSJF sequencing whenever Product Backlog we perform refinement. Since Product Backlog refinement is a continual process, there will be future opportunities to apply WSJF. Aim for “good enough” and trust that whatever issues surface later will be taken into account during the next round of Product Backlog ordering.
It is good practice to not let Product Backlogs grow too large. Clearing items that may never be implemented is a good idea. They can always be reintroduced later for ordering. In the same vein, avoid ordering the entire Product Backlog. Instead, focus on the next three Sprints or so. Ordering items that may never rise high enough in the backlog to be implemented is wasteful.
Remember that Product Backlog ordering is an art, not a science. As leads of this process, POs must leverage input from their Development Teams and solution community stakeholders and their own intuition and experience.
If the product backlog is small or mainly populated with user story-sized items, computation of WSJF values may be overkill. However, it is still advisable to take the five ordering factors discussed here into account when ordering product backlogs.
Scrum Teams can leverage user story maps to organize an ordered Product Backlog across solution workflow steps, Sprints, and releases. Scrum Teams may use them in tandem with Product Backlogs to help identify dependencies as part of backlog refinement. We explain user story mapping and provide an example here.
The Product Backlog contains all of the solution requirements (e.g., epics and user stories) for which the Development Team may implement features. If it is not in the Product Backlog, the Development Team does not implement it. However, just because something is in the Product Backlog does not mean it will be done.
In light of this, it is often difficult for teams to balance delivery of business or mission value with work necessary to:
One way Scrum Teams achieve the right balance between different types of work is through capacity allocation. Teams allocate their capacity to do work within an iteration/Sprint. They allocate percentages of that timebox to specific types of work. For example, a team may agree to devote most of the next Sprint to developing new business or mission capabilities, but allocate a percentage of that time to address technical debt and maintenance. The team agrees to the types of work they will perform, and the percentages of the Sprint those work types will consume, based on current needs and constraints.
The key to effective Product Backlog prioritization is achieving an optimal balance between competing factors. “Optimal” does not mean perfect.
The WSJF technique provides a simple way of accounting for those factors. WSJF is only a tool to help make decisions. It is not a deterministic process that renders an exact solution.
Arriving at a “good enough” Product Backlog ordering depends much more on effective collaboration between POs, development teams, and solution community stakeholders than on applying the techniques discussed in this article. Ultimately, the quality of Product Backlog ordering depends on Agile’s empirical inspect and adapt approach to defining and executing work.
[1] Don McGreal and Ralph Jocham (2018). The professional Product Owner: Leveraging Scrum as a Competitive Advantage 1st Edition. Scrum.org
