Category: Agile Concepts

Why So Many Developers are Fed Up with Agile (Pt. 2)

Greater Visibility & Oversight with Less Control

Recently, I visited an Agile-related Reddit forum/subreddit.  On one thread, developers were chiming in their complaints about Agile software development.  One comment caught my attention.  The commenter stated that, “Agile treats developers like children.”

That comment reminded me of a previous employer who had just purchased JIRA licenses and was excited at the prospect of using the tool to keep an eye on what his company’s developers were working on.  I remember thinking how ironic it was that the greatest benefit the business owner saw in a tool ostensibly designed to support the work of Agile teams was the ability to surveil his employees.  If that was not a clear subversion of Agile principles, I don’t know what is.

That anecdote crystalizes the point of this article:  Agile’s emphasis on greater visibility into progress achieved by Agile teams can easily be turned against them.  In other words, increased visibility over the work performed by Agile teams often leads to more intrusive management oversight that diminishes the ability of teams to manage their own work (self-organize).  

This article is the second of a four-part series about the reasons so many developers are fed up with Agile and so many organizations fail to achieve the benefits touted by Agilists, consultants, and tool vendors.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.
  • Changing Expectations for Developers – Developers are moving away from functioning as “order takers” focused almost exclusively on coding to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

Every article in this series describes each dysfunction, root causes, and suggests solutions that align with Agile values and principles.

Greater Visibility & Oversight with Less Control

One of the features of Agile is its “radical transparency.”  Radical transparency denotes how Agile teams communicate openly with other teams, project sponsors and stakeholders, and customers and end users as they collaborate to define, build, verify, and validate solutions.  Part of this communication involves keeping others apprised of team progress and the impediments they encounter.

The point of this openness is to foster an environment where everyone involved in sponsoring, managing, developing, and using solutions has the information they need to collaborate effectively without impinging on the flow work established by developers. 

Thus, Agile teams prioritize open communication.  For example, a common Agile practice is the use of “information radiators.”  An information radiator is any type of visual display, placed in a plainly visible physical location or accessible virtually, that allows product or project stakeholders to review team and project information at a glance.

Unfortunately, such transparency is often abused by management overly focused on maximizing efficiency and resource utilization.  When the pendulum swings too far in that direction, process execution and status tracking supplant delivery of valuable capabilities as measures of progress.  This outcome reflects a misplaced faith in achieving successful outcomes merely by following processes.

Rather than supporting Agile team self-organization, management steadily increases control over how development teams work.  Eventually, developers are diminished to the status of order takers with little actual control over what they work on and how.

Agile’s Take on Processes and Tools

One of the values of the Agile Manifesto for Software Development is that Agilists value, “Individuals and interactions over processes and tools.”  While Agile teams leverage processes and tools to improve how they work, they do not use them in ways that attempt to replace face-to-face or real-time interactions or communication.

Agile is not “anti-process.”  Agile is against confusing process execution with actual progress and instituting wasteful or counterproductive processes.  Agile neither prohibits nor prescribes processes.  Instead, it advocates for environments where self-motivated individuals self-organize and develop “just enough” process to support their work.  Agile’s culture of inspection and adaptation also applies to processes, as teams improve how they work.

Productivity tools can significantly increase Agile team productivity and effectiveness.  For example, not every interaction in and among Agile teams can be face-to-face.  While Agile best practice is to collocate teams, doing so is not always possible.  Agile development teams with members who work remotely, and enterprises staffed entirely with remote teams are common.  A project team may be too large to sustain ongoing direct communication between members.  Tools bridge distances and time zones between collaborators and help manage indirect communication.

Also, requirements management tools help document and track requirements, manage work, and report progress.  Agile requirements artifacts are purposely lightweight. They work as reminders for future conversations that will flesh out details.  However, even lightweight requirements become difficult to manage as they grow in number.

For far too many teams, however, this nuanced approach to tools and processes is crushed under the weight of traditional project management thinking. 

Surveillance through Productivity Tools

Traditional management shifts the focus from measuring business or mission value delivered to customers and end-users to measuring how faithfully and efficiently development teams follow processes.  Developers become process execution resources to manage.  Under this shift, productivity tools cease to serve Agile team self-organization and become surveillance tools for management.  Some of the negative consequences of this shift include:

  • The amount of time developers spend using tools to document and report on their work begins competing with the amount of time they spend developing features.
  • Management reconfigures “Agile” collaboration tools to track traditional project management productivity metrics that do not align with Agile.
  • Management rejects Agile-based reporting metrics and formats (e.g., burn-up and burn-down charts) because such reporting does not support the traditional project management approach they are used to.
  • Face-to-face or real-time collaboration are replaced by handoffs brokered by tools.
  • Since management’s appetite for status is insatiable, productivity tools become information funnels for management and cease providing meaningful value to developers.

Ultimately, the entire Agile approach is derailed as management places traditional project management expectations on Agile development teams.

Forced Commitments

When developers lose control over how they work, they also lose control over how much work they take on.  As I stated in part one of this series, Agile teams align the amount of work they commit to with their collective capacity to finish the work within a period of time.  Software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Agile teams estimate their own work.  They negotiate with product/project management regarding what features they will deliver and when.  They also determine release schedules collaboratively.  For these estimates to be accurate, teams need to trust that management will respect their assessments and refrain from treating estimates as deadlines.

Agile manages projects through scope.  So, the goal is to work on the highest ordered/prioritized features and periodically assess progress.  Estimates improve as teams learn from engaging in the work and stakeholders get a better sense of what they truly need.

Commitments are not do or die promises.  They are honest professional assessments of what teams can get done during a period of time.  The team endeavors to achieve everything it committed to while maintaining a sustainable pace.  While some amount of overtime is necessary in most projects, it is not a way of life in Agile projects.

Holding commitments over the head of developers ignores the fact that team commitments are based on estimates.  It also pressures developers into cutting corners or inflating estimates and committing to less work.  Agile requires a management approach based on collaboration and mutual respect.  Micromanaging the work of developers through tools and pressuring them to overcommit does not engender trust or respect.  Taylorist notions of maximizing resource utilization have no place in Agile.

Finally, pressuring teams to overcommit sends the message that Agile practices do not matter.  Why should a Scrum Team bother with Product Backlog refinement, estimating user stories, and committing to Sprint goals if the resulting plans are unrealistic and ignored by management?

Proposed Solutions

The solution to this problem boils down to educating management about the fundamental paradigm shift from traditional project management to Agile.  Too many managers are under the impression that Agile is something developers do while everyone else in the organization continues to work as they always have.  The decision to adopt Agile comes with a commitment to continually shift the organization towards a customer-centric, product management mindset.  This involves establishing the following practices:

  • Adopt a product management approach to solutions/systems development
  • Focus on establishing and nurturing high-performance teams
  • Stress the need for planning and managing customer and end-user expectations through commitments
  • Foster collective team ownership of the work
  • Encourage managers and leaders to stand up for teams that feel pressured to overcommit
  • Enforce development team ownership of team/Sprint backlogs

Conclusion

People develop software.  Software development is creative work.  The best software solutions come from skilled, talented, and intellectually curious people who are passionate about their work and the software they create.  They are professionals who take pride in delivering quality software.  They understand the inefficiencies and costs associated with low quality designs and implementations.  Thus, they strive for technical excellence.  Agile advocates for organizational environments that nurture this sense of professionalism and ownership by empowering teams to self-organize.

Agile software development is not for the timid.  There is nowhere to hide on an Agile team.  Agile team members are accountable to each other and collectively commit to delivering value in time to be valuable.  Agile teams are highly transparent and accountable environments.  As such, members need some degree of “psychological safety”, exemplified by trust, respect, and the belief that they will not be punished for making a mistake.  Imposing a traditional project management mindset on Agile teams puts members in the undesirable position of being held accountable for results without having the power to affect change.

In the next article of this series, I’ll discuss how top-down-driven Agile adoption initiatives fail to include the perspectives and expertise of the people most affected by the change, the developers, and how doing so sets up Agile adoptions for failure.



Why So Many Developers are Fed Up with Agile (Pt. 1)

The Never-Ending Agile Grind

Agile has become the way we build software.  Across industries, government institutions, and non-profits, it is the default software development approach.  Debates over whether Agile should supplant traditional project management approaches to software development (i.e., Waterfall) have long been passé.  The focus now is on accelerating the product development lifecycle through DevOps.

However, all is not well in “Agileland”.  Many organizations are not enjoying the benefits promised by the Agile gurus.  Many software developers are fed up with Agile.  If Agile is so great, why are so many people, especially developers, souring on it?  The bottom line upfront (BLUF) is that what passes for Agile in many places is anything but Agile.

This article is the first of a four-part series that describes the most common pain points experienced by software developers and other technical contributors engaged in Agile software development, the root causes, and potential solutions.  The pain points covered in the series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
  • Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.
  • Changing Expectations for Developers – Developers are moving away from functioning as “order takers” focused almost exclusively on coding to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

The Never-Ending Agile Grind

Software development is a demanding intellectual activity.  It requires study, perseverance, and focus.  The work requires time to think and experiment.  Some degree of failure is necessary as long as it yields valuable insights and teams use those insights to continually improve the product and how they work.

The experience of too many developers on Agile teams, however, is working in “feature factory” environments and “death march” projects.  They feel pressured and rushed to deliver code to the point of sacrificing quality.

Sprints become Sisyphean ordeals where overburdened teams are under constant pressure to deliver working code as quickly as possible while also maintaining the existing code base and contributing DevOps pipeline automation components.  The end of one Sprint leads to the beginning of another in a seemingly never-ending loop of stress that eventually leads to burnout. 

Teams are not afforded time to think through and experiment with alternative approaches, leading them to commit to suboptimal approaches that will likely take longer to implement and maintain than more elegant approaches they could have conceived given sufficient time.

“Feature factory” teams are denied time “on the clock” to learn and experiment with new technologies, design patterns, and architectural approaches.  As a result, implementations based on those approaches are often half-baked, filled with technical debt, under-performant, and difficult to maintain.  The benefits expected from adopting new technological approaches are not realized.  Meanwhile system complexity explodes exponentially.

Management consistently chooses delivery of new functionality over addressing technical debt.  This short-sided preference leads to increasingly brittle systems that force developers to expend increasing amounts of energy and focus on fixing bugs and babysitting systems.

Overtime, developers start to feel like robotic coding units who get paid to crank out code.  A mentality of “getting it done on time no matter what” becomes culture.  Any vestige of work/life balance is thrown out the window as the backlog of features grows and the system or product becomes ever more unmanageable.

The Causes

The Agile Manifesto for Software Development and the 12 Principles of Agile describe the characteristics of Agile software development.  Software development approaches that adhere to the values of the Manifesto and the 12 principles are considered Agile.  Approaches that deviate from any of the values or principles are not.  One Agile principle “feature factory” teams violate is principle number #8:

Agile processes promote sustainable development.  The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

http://agilemanifesto.org/principles.html

Why do so many software development organizations and their sponsors ignore this principle?  Many blame the increasing pace of business competition or mission threats for this single-minded focus on production and speed.  While pressure from increased competition is a contributing factor, I posit that the root cause is cultural.  Many (if not most) of these organizations take a project management approach to software development.  Agile, in general, and Scrum in particular, have a product management orientation.

Traditional project management, rooted partly in Taylorism, focuses on managing resources, schedule, and budget.  People are a category of resources managed by managers.  Traditional project management places a very high value on maximizing resource utilization to achieve greater efficiencies.  People are the most important (and expensive) resources to maximize.  In the drive to do more with less and meet tight (often arbitrary) deadlines, management often loses sight of the link between team welfare and morale and superior performance.

Product management focuses on managing the product and how the product is made and delivered.  Through Agile, management shifts its focus from micromanaging developers to collaborating with them in continually improving product offerings as well as the development and delivery of those offerings.  Ensuring development teams are not overwhelmed becomes a major concern since it has such a large bearing on product quality, timeliness, cost, etc.

Another cultural factor that leads to “feature factory” cultures is confusion over the role of planning in Agile.  A popular misconception is that there is no planning in Agile software development.  This misunderstanding leads to underemphasizing requirements definition, analysis, and design which, in turn, leads to badly conceived and executed software implementations.

Agile teams plan constantly.  The difference between Agile planning and planning in Waterfall projects is that Agile planning is adaptive while traditional project planning is predictive.  In other words, rather than attempting to plan the entire project upfront, Agile plans change based on current circumstances and new information and insights.

The Software Development Lifecycle (SDLC)

Just like development teams in Waterfall projects, Agile teams perform all the activities encompassed by each of the phases of the Software Development Lifecycle (SDLC).  The difference is that the “Agile SDLC” (DevOps) does not segregate those activities into discrete scheduled phases arranged one after the other for the entire system or solution.  Instead, every feature/deliverable an Agile development team delivers is the product of its own “mini” SDLC.  The resulting system or solution is the sum of all integrated features.

Potential Solutions

So, what are some concrete steps software development organizations and their sponsors can take to avoid overwhelming and burning out technical contributors?

The first step is understanding and internalizing that Agile development teams are self-organized.  They collectively manage their own work, determine technical approaches, develop their own estimates, report their progress, and help define the functionality they build.  Team self-organization is a core tenant of Agile supported by three of the 12 Principles of Agile:

Principle #11:  The best architectures, requirements, and designs emerge from self-organizing teams.

Principle #5:  Build projects around motivated individuals.  Give them the environment and support they need and trust them to get the job done.

Principle #12:  At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

http://agilemanifesto.org/principles.html

This contrasts with traditional project management practice, in which management is responsible for defining what to build, breaking down and assigning work, developing plans and schedules, and reporting progress.

Align the Work to Team Capacity

Agile teams break down their work with an eye towards delivering valuable functionality within short timeframes.  They can only achieve this if they align the amount of work they commit to with the team’s ability to get it done.  In other words, break down solution scope into slices of capability (features) that can be accomplished within agreed timeframes.  By ordering/prioritizing features along the lines of business or mission value, Agile releases deliver the most valuable features/capabilities first.

Agile teams commit to additional work as they complete existing work.  Rather than having work pushed on Agile teams, Agile teams pull work when they are ready to do it.  Software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Allocate Time for Specific Tasks during Sprints

One way to ensure that Agile teams address technical debt, and other necessary tasks outside the scope of developing new features, is to allocate some percentage of their time to those activities.  For example, during Sprint planning, a Scrum Team may allocate some percentage of the next Sprint to addressing technical debt.  They could do the same for backlog refinement and DevOps automation work.

Establish Innovation & Planning Sprints

The Scaled Agile Framework (SAFe) includes Innovation and Planning (I&P) iterations (Sprints).  I&P iterations occur at the end of each Program Increment (PI), which lasts eight to twelve-weeks.  Each PI includes two to six iterations depending on the length of each iteration (two to four weeks).  Agile teams do not have to adopt SAFe to experiment with I&P iterations or Sprints.

I&P iterations provide teams time to work important tasks, such as the ones below, without significantly disrupting the flow/cadence of development and release of new functionality:

  • Explore new and innovative approaches (e.g., experiment with new technologies; evaluate products, tools, and services)
  • Work on technical infrastructure, tooling, and other impediments to delivery
  • Engage in continuous learning and improvement
  • Cross train in new skills
  • Engage in longer-term planning (e.g., product roadmap discussions, architecture design)

Conclusion

It is common knowledge that productivity, quality, and innovation suffer when workers are stressed and burned out.  However, too often responsibility for coping with stressful work environments is placed solely on the shoulders of individual employees/contributors while management sticks to outdated ideas about maximizing efficiency.

Agile, implemented in the spirit of Agile, opens the door to sustainable progress and achievement.  Again, development teams that do not control the amount of work they commit to delivering are not Agile teams.  Such teams risk being pressured into over-promising while under-delivering.

In the next article of this series, I’ll discuss how the greater visibility into team activities and progress afforded by Agile is often used to against the development teams themselves.  Just like this article I will also discuss root causes for the dysfunction and suggest solutions.



The Five Most Important Factors in Prioritizing Product Backlogs

Achieve effective Product Backlog ordering by optimally balancing five competing factors

Note: This article includes a video detailing how to calculate Weighted Shortest Job First (WSJF) values.

Product Backlog “prioritization” is a popular topic in Agile and Scrum forums.  It seems like a simple concept.  Teams prioritize the most valuable or important product backlog items or features and begin work on the highest priority items.  However, teams quickly discover that there is much more to prioritizing backlog items than that.

The difficulty boils down to determining what is more or less valuable or important when every item addresses valid needs.  How do we deal with the ambiguity of determining what is “most valuable” or “more important”?  How do we account for dependencies and other factors that influence what we build and when? How do we do so in a consistent, non-biased way that ensures delivery of valuable solutions, cost effectively, and in time to be most valuable?

Understanding Product Backlogs and their Refinement

For the uninitiated, before reading more about Product Backlog prioritization, I recommend reading our article on Product Backlog refinement.  At a minimum, understanding what a Product Backlog is and what Product Backlog refinement entails is crucial to understanding backlog prioritization.

Product Backlog refinement includes three main activities:

  1. Management of Product Backlog Items:  The creation, retrieval/tracking, update, and deletion of items

  2. Sizing of Product Backlog Items:  Relative size estimation of items

  3. Ordering:  Deciding in what order to address each item

In this article, I focus on Product Backlog ordering. This article does not apply to Sprint or iteration backlogs. Sprint backlogs are solely composed of user stories that take two to four weeks to complete. These items are likely too small in scope to warrant multi-factor prioritization/ordering schemes.

Prioritization vs. Ordering

As of 2011, the official Scrum Guide began defining the Product Backlog as “an ordered list” instead of a “prioritized” list.  Why was this changed and what is the difference between prioritizing a list and ordering it?

As Barry Overeem states in his Scrum.org article:

The focus on ordering (over ‘prioritization’) underscores the active role that the Product Owner continuously has to play in the ordering and reordering, of the work in a manner that maximizes value.

https://www.scrum.org/resources/blog/myth-5-scrum-product-backlog-prioritized

Simply stated, many factors influence Product Backlog Ordering, not just what is considered most important from a business or mission perspective.

Non-trivial software solutions service many sponsors, stakeholders, customers, and end users.  Their interests and priorities differ from, and sometimes contradict, each other.  It is the job of the Product Owner (PO) to fuse the various needs, wants, and perspectives of that solution community into one complete, coherent, and cohesive product vision.

There are also project and technical realities that must be taken into account when deciding what portions of a solution to build and deliver and when.  Examples:

  • Environments (e.g., development, integration, test, and production environments) may not be ready for use at a particular point in the project schedule

  • Key system dependencies (e.g., features, components, libraries, frameworks) may be missing or incomplete, thus impeding progress

  • The customer may add or take away project scope during the project

Business or mission circumstances, solution community needs, solution requirements, and project priorities change as the solution takes shape through iterative and incremental delivery of system capabilities.  A PO must lead continuous refinement and ordering of Product Backlog items to align with such changes.

Five Factors that Influence Product Backlog Ordering

The most optimal ordering of Product Backlog items is a balance between these competing factors:

  1. Business/Mission Value

  2. Cost/Size

  3. Time Criticality

  4. Technical Dependencies

  5. Risk Reduction/Opportunity Enablement

Figure 1: Product Backlog Ordering Factors

Factor 1: Business/Mission Value

How Agile defines value is different from how traditional project management defines it.  A simple definition of value is whatever solution sponsors, stakeholders, customers, and end users care about or makes them happy.  We discuss Agile’s concept of value further here.

A feature’s business or mission value is the value created by implementing the feature (e.g., revenue, cost savings, customer retention, prospective customers, future opportunities, mission accomplishment).  Features that map closest to the product vision likely rank highest for this factor.[1]

Factor 2: Cost/Size

It is common Agile practice to use relative sizing to estimate how much of a development team’s capacity will be consumed by a particular Product Backlog item.  One way Agile teams perform relative sizing is by assigning story points to Product Backlog items. Story points are heuristically-determined, numerical proxies representing the combination of perceived or estimated:

  • User story complexity, risk, and uncertainty

  • Level of effort required to complete the user story

This combination is often referred to as the “bigness” or “size” of a user story. The development team estimates user story points by “sizing” stories relative to each other. More complex stories typically require more time and effort to complete, thus earning more story points than less complex stories.

Just as there is a correlation between story complexity and size, there is also one between story size and the cost of implementing the story.  While not a perfect correlation, it is good enough to support efficient ordering of Product Backlog items.

Factor 3: Time Criticality

How soon solution capabilities or system dependencies are needed also influences Product Backlog ordering.  Again, the goal of Agile is to deliver valuable capabilities, cost effectively, and in time to be valuable.  Delivering a capability before it is needed risks spending resources on something that may never be needed (YAGNY:  You Aren’t Gonna Need It!).  Delivering after it is needed risks delivering it too late to be valuable.

Agile achieves a “just-in-time” balance by constraining planning, development, and release scope to fit within relatively short time frames (iterations/Sprints/releases) and delivering the highest ordered items first.  By waiting until “the last responsible moment” to define, develop, and release solution functionality, we avoid over-engineering solutions or losing opportunities to provide value.

Factor 4: Technical Dependencies

Software solutions are inherently complex.  As such, their technical components often rely on other components to function.  For example, multiple features may require a database backend to store data and state.  Product Backlog ordering must take into account these dependencies as solutions grow and evolve.

Factor 5: Risk Reduction or Opportunity Enablement

Software system complexity introduces technical and project risks and opportunities.  Diligent backlog refinement spreads risk management actions across the entire software development effort.  For example, a team may choose to tackle riskier Product Backlog items earlier.  Another may identify an opportunity to shorten the schedule by accelerating delivery of a dependency common across multiple parts of the solution.  This differs from traditional project management efforts that, even when risks and opportunities are identified early, tend to accumulate risks, address them after they become issues, and miss opportunities.

Prioritization Techniques

Backlog ordering is the art of balancing these five factors to best facilitate Agile development and delivery of software solutions.  The three prioritization techniques briefly discussed in this section are not specific to Agile or Scrum.  They are well-known product management techniques. 

The first two, the Kano Model and the MoSCoW Method, have narrow perspectives and reduce business or mission value considerations to simplistic categories.  The third technique, Weighted Shortest Job First (WSJF), takes into account all five factors discussed in the previous section.

Kano Model

Professor Noriaki Kano developed the Kano Model in the 1980s.  The model categorizes product features according to customer needs and expectations.  In other words, what will satisfy or please customers?  Many versions of the model exist.  The original classifies items using five categories: Must-be, Attractive, One-Dimensional, Indifferent, and Reverse.

The Kano Model is most useful in organizations that create mostly “Must-Be” features: Features expected or taken for granted by customers.  However, products should include “Attractive” and “One-Dimensional” features to succeed in the marketplace.

Table 1: Kano Model Categories

Using this technique, the Product Backlog would include items from every category except “Reverse”.  Releases and Minimum Viable Products (MVPs) would include some set of items from those categories.  As MVPs grow and evolve, more items from across those categories would be added.

The problem with this approach is that it narrowly focuses on customer needs and expectations, which is one of many factors in gauging business or mission value.  However, Kano Model category definitions are useful in helping to shape thinking about the business or mission value of solution features.

MoSCoW Method

The MoSCoW Method, developed by Dai Clegg, has roots in Agile software development.  It is an acronym loosely based on the first letter of each of its categories: Must-Have, Should-Have, Could-Have and Won’t-Have.  The categories are defined from the point of view of what a product includes or offers vis-a-vis what customers or users want.  It answers the question, “What features will be included in the release or MVP?”

Table 2: MoSCoW Method Categories

Similar to the Kano Model, using this technique produces a Product Backlog that includes items from every category.  “Won’t-Have” items may either be postponed to future iterations or releases or dropped from the backlog altogether.

However, like the Kano Model, the MoSCoW Method focuses on one dimension of value.  In the case of the MoSCoW Method, it is determining which set of features best aligns with customer and end user expectations about the product/solution.

Weighted Shortest Job First (WSJF)

In his book, Principles of Product Development Flow: Second Generation Lean Product Development, Don Reinertsen explained the Weighted Shortest Job First (WSJF) prioritization model.  The model is used to sequence work in the most economically efficient way.

The folks at Scaled Agile Inc. explain a simplified version of Reinersten’s model here.  The simplified approach consists of two formulas:

Figure 2: A Formula for WSJF


Figure 3: Calculating Relative Cost of Delay

This simplified model is used in the Scaled Agile Framework (SAFe) to sequence (order) individual pieces of work or “jobs” (e.g., SAFe Features, Capabilities, and Epics) within their respective backlogs.

Unlike Reinertsen’s model, which uses monetary amounts to calculate Cost of Delay (CoD) and job duration, SAFe’s simplified model uses a relative sizing construct.  The construct uses a modified Fibonacci sequence as the basis for parameter values.  In calculating CoD, values for User-Business Value, Time Criticality, and Risk Reduction/Opportunity Enablement are assigned greater or smaller values relative to other jobs being ordered.  For job duration, “size” values are assigned to individual jobs.  Every job is sized relative to the other jobs being ordered.

We can create a simple table to track the relative values and calculate CoD and WSJF values:

Figure 4: WSJF Table

The following video provides more details on calculating WSJF values:

Calculating WSJF Values

The initial formulaic answer is a start to the ordering, not a final answer.  The computed sequence of Product Backlog items is adjusted to account for technical dependencies between them.

Teams may find it helpful to keep in mind the Kano Model and MoSCoW Method categories as they assign relative business or mission values to Product Backlog items/features and determine which items are ranked too low to include in the next Sprint or release.  Just remember that those categories do not represent the entirety of business or mission value.

We can use WSJF sequencing whenever Product Backlog we perform refinement.  Since Product Backlog refinement is a continual process, there will be future opportunities to apply WSJF.  Aim for “good enough” and trust that whatever issues surface later will be taken into account during the next round of Product Backlog ordering.

It is good practice to not let Product Backlogs grow too large.  Clearing items that may never be implemented is a good idea.  They can always be reintroduced later for ordering.  In the same vein, avoid ordering the entire Product Backlog.  Instead, focus on the next three Sprints or so.  Ordering items that may never rise high enough in the backlog to be implemented is wasteful.

Remember that Product Backlog ordering is an art, not a science.  As leads of this process, POs must leverage input from their Development Teams and solution community stakeholders and their own intuition and experience.

When Not to Employ WSJF

If the product backlog is small or mainly populated with user story-sized items, computation of WSJF values may be overkill. However, it is still advisable to take the five ordering factors discussed here into account when ordering product backlogs.

Leveraging User Story Maps

Scrum Teams can leverage user story maps to organize an ordered Product Backlog across solution workflow steps, Sprints, and releases.  Scrum Teams may use them in tandem with Product Backlogs to help identify dependencies as part of backlog refinement.  We explain user story mapping and provide an example here.

Capacity Allocation

The Product Backlog contains all of the solution requirements (e.g., epics and user stories) for which the Development Team may implement features.  If it is not in the Product Backlog, the Development Team does not implement it.  However, just because something is in the Product Backlog does not mean it will be done.

In light of this, it is often difficult for teams to balance delivery of business or mission value with work necessary to:

  • Implement back-end functionality or infrastructure that supports business/mission-facing functionality

  • Perform system maintenance to ensure overall system performance and quality

  • Address technical debt

  • Explore new requirements and technical approaches to address them

One way Scrum Teams achieve the right balance between different types of work is through capacity allocation.  Teams allocate their capacity to do work within an iteration/Sprint.  They allocate percentages of that timebox to specific types of work.  For example, a team may agree to devote most of the next Sprint to developing new business or mission capabilities, but allocate a percentage of that time to address technical debt and maintenance.  The team agrees to the types of work they will perform, and the percentages of the Sprint those work types will consume, based on current needs and constraints.

Conclusion

The key to effective Product Backlog prioritization is achieving an optimal balance between competing factors.  “Optimal” does not mean perfect. 

The WSJF technique provides a simple way of accounting for those factors.  WSJF is only a tool to help make decisions.  It is not a deterministic process that renders an exact solution. 

Arriving at a “good enough” Product Backlog ordering depends much more on effective collaboration between POs, development teams, and solution community stakeholders than on applying the techniques discussed in this article.  Ultimately, the quality of Product Backlog ordering depends on Agile’s empirical inspect and adapt approach to defining and executing work.

[1] Don McGreal and Ralph Jocham (2018).  The professional Product Owner: Leveraging Scrum as a Competitive Advantage 1st Edition. Scrum.org



Business Agility Depends on Culture

While most organizations would benefit greatly from attaining business agility, many do not possess cultures amenable to Agile transformation.

Organizations, of all types, are jumping on the “business agility” or “Agile transformation” bandwagon. Business agility is the ability to compete and thrive in volatile business or mission environments by responding quickly to changing market or mission circumstances.  Agile transformations involve the application of Agile and Lean principles to business or mission processes and decision making.  Organizations seek to emulate the efficiency gains and flexibility achieved through Agile software development, but in a business or mission context. Business (or organizational) agility is the goal of Agile transformations.

The value proposition for Agile transformation is compelling.  According to proponents of Agile transformation initiatives, organizations stand to become more competitive and thrive in increasingly volatile markets and mission domains by achieving:

  • Cost savings through greater efficiency
  • Faster responses to changing market or mission conditions
  • Better business performance or mission execution

Also driving growth in the number of Agile transformation initiatives is a growing appreciation for the need to align enterprise information technology (IT) investment, governance, and operations with Agile’s iterative and incremental delivery of software system capabilities.

Figure 1: Agile Transformation Taxonomy

These incentives have fueled a multi-billion-dollar Agile transformation services industry.  According to a report published by Allied Market Research in August 2019, “… the U.S. enterprise Agile transformation services market was pegged at $4.91 billion in 2018 and is expected to hit $18.19 billion in 2026 …”.

Despite all of the money spent (or, perhaps, partly because of it), many Agile transformation initiatives fail.  We can only estimate how many, but according to Jeff Sutherland, co-creator of the Scrum framework, 47% of Agile transformations do.

The Organizational Culture Challenge

Most Agile transformation initiatives failures are tied to organizational culture.  The 14th Annual State of Agile Report, published in 2020, supports this assertion.  Four out of the top five reasons, cited by survey participants, for failed Agile scaling initiatives are directly tied to organizational culture:

Figure 2: Challenges to Scaling Agile

For organizations that develop software systems, or sponsor their development, scaling of Agile software development and system operations is a crucial part of Agile transformation.  Cultural issues that frustrate their success also negatively affect business or organizational agility.  The first and third challenges, marked in red in Figure 2 above, explicitly call out organizational culture.  The second and fourth marked challenges point to lack of commitment and participation from organizational leadership and management.  Organizational leadership is responsible for setting the tone for the rest of the organization, so a lack of support from them translates into a lack of support for Agile transformation across the organization.

Addressing the Organizational Culture Challenge

As common as the “culture” challenge is, it is hard to tackle because it is so vague.  What are the reasons for “resistance to change” and how do they manifest themselves?  What is “enough” leadership and management participation and support and what do they entail?  Beyond performing the mechanics of a chosen Agile approach (e.g., Scrum, Kanban, SAFe), when do we know the organization has achieved an “Agile culture”?

Organizational culture influences behavior.  The inverse is also true: changing behavior changes culture.  Merely exhorting people to “think Agile” and “adopt Lean thinking” is a waste of time because they likely do not know how to do so within the context of the organization and their jobs.  On the other hand, engaging in a mechanical implementation of an Agile approach, without understanding its values and principles, guarantees that existing organizational cultural biases will corrupt the approach.

Organizational culture change starts with laying out a clear and compelling vision for the organization.  The organization’s chosen Agile approach becomes the means by which the vision is implemented.  An Agile transformation effort entails some degree of uncertainty and risk.  It will likely be the first time the organization embarks on such an effort.  Mistakes will be made and lessons will be learned.  This is why closely aligning to the chosen Agile approach and understanding its values and principles is so important.

The Agile approach serves as the foundation upon which organization-specific practices eventually emerge.  While the Agile approach likely requires some degree of tailoring to ensure it aligns to the constraints and needs of the organization, such tailoring must be informed by a deep understanding of the implications of each change.  It is best to explore ways of changing existing organizational processes to align to the Agile approach before tailoring the approach to fit the organization’s processes.

As people learn how to do their jobs, in alignment with the Agile approach, through training, coaching, and actual experience, their understanding of the approach’s values and principles take root and increasingly shift organizational culture towards a shared, agile-enabled vision.

Four Organizational Characteristics Most Likely to Derail Agile Transformations

Before embarking on an Agile transformation initiative, organizational leadership would be wise take time to reflect on how well positioned they and their organizations are to achieve a successful Agile transformation.  While it is true that the only way to know for sure is to try, an honest self-assessment enables leaders to approach Agile transformation with a clear-eyed understanding of what cultural obstacles they are likely to encounter. Leaders may come to the conclusion that their organization is not ready for the transformation.  In our experience, the organizations least likely to achieve business agility through Agile transformation share the following four cultural characteristics:

  1. They emphasize process execution over actual progress
  2. They emphasize control to the detriment of speed and flexibility
  3. They discourage experimentation and punish failure
  4. They possess a consistent record of failure

1. Emphasizing Process Execution over Actual Progress

Many organizations operate under the assumption that following plans and procedures ensures good outcomes.  While no organization can operate without them, many organizations lose sight of the purpose of plans and procedures: to accomplish business or mission goals and objectives.  When this happens, the following cultural dysfunctions arise:

  • People are rewarded for following plans and procedures, rather than for achieving desired outcomes
  • Failure is accepted so long as procedures were followed
  • Project status reporting tracks process activity completion rather than progress towards desired outcomes

One of the 12 Principles of Agile Software Development is, “Working software is the primary measure of progress.”  The only way to validate the value of any software capability is by getting it into the hands of customers and end users.  Only they can determine whether expected business or mission outcomes were achieved through use of the software. 

Similarly, the effectiveness of business plans and processes should be validated against business or mission outcomes.  Doing so involves quantifying expected outcomes and continually measuring their attainment against relevant metrics.  Failure to achieve desired outcomes should prompt process changes or the formulation of new processes.  This empiricism is a critical characteristic of Agile and Lean.

2. Emphasizing Control to the Detriment of Speed and Flexibility

The point of carrying out an Agile transformation is to achieve business or organizational agility.  Agile Cheetah defines agility as the combination of speed, control, and flexibility:

Agility = Speed + Control + Flexibility

The elements of this “Agility Triad” are mutually dependent constraints.  In other words, more or less of any one constraint influences the other two.  For example:

  • Accelerating the pace of work requires some degree of control and flexibility
  • Overemphasizing control slows progress and decreases flexibility
  • Underemphasizing control can turn flexibility into chaos, robbing teams of both planning and process stability, thereby slowing progress

Many organizations are guilty of the second bullet.  Overemphasizing control to the detriment of speed and flexibility is emblematic of fear-based organizational cultures.  In such cultures, Taylorism prevails.  Organizational structure tends to be hierarchical with a strong command-and-control management approach.  There is often an unwillingness to delegate decision making down to the most responsible levels.  In the most extreme cases, position, status, and tenure trump expertise, knowledge, and skills.

Source: geek & poke under CC-BY-3.0 license. No changes made.

Rather than accepting that some amount of risk is an inevitable part of solving tough problems, fearful organizations avoid taking risks.  Fear of failure produces cultures where failure to act is deemed preferable to acting and failing.  When failure happens, finding blame takes precedence over finding solutions.  Additional control mechanisms are put in place to ensure it does not happen again. 

Unfortunately, such mechanisms often feed a vicious cycle where people lose some autonomy and latitude to act, resulting in a loss of effectiveness, which breeds further failure, which then prompts further controls.  Left unchecked, this cycle can whittle down organizational effectiveness to the point where the organization loses the ability to reach previous levels of performance.

Figure 3: Vicious Cycle of Degraded Organizational Performance

Organizations with strong hierarchical and command-and-control cultures struggle through Agile transformation more than those with more egalitarian and permissive cultures.  Leaders should ask themselves; how much authority are they willing to delegate to achieve greater flexibility and speed?  What changes need to happen to enable their people to assume greater ownership and responsibility for their work and resulting outcomes?

3. Discouraging Experimentation and Punishing Failure

Organizations that overemphasize control also tend to discourage experimentation and punish failure.  Rather than exploring different solution options, they rush to settle on one approach.  Time that could have been spent investigating options is spent improving a solution that was likely suboptimal from the start.  Fear of ambiguity and risk leads to the formulation of unrealistic plans (In search for certainty, they conjure fantasies).  Premature commitment to plans takes away the flexibility needed to adjust to changing circumstances and new information.  Closed off to alternatives and fearing the consequences of admitting failure, the organization continues to sink resources into failing approaches.

As mentioned earlier, Agile and Lean are based on empiricism.  In an empirical process, information gained through experimentation and observation informs and validates plans, decisions, and actions.  Planning is adaptive as opposed to predictive.  In other words, Agile plans change based on current circumstances and new information and insights. 

Agile plans skew towards shorter planning horizons measured in weeks to months rather than months to years.  When problems or mistakes happen, Agile teams re-plan and correct course.  Short planning horizons afford the flexibility to make new plans and pivot to new solution approaches.

Like experiments, plans and decisions are treated as a hypothesis.  When a hypothesis proves to be wrong, another one is formulated and tried.  The faster teams cycle through these learning cycles, the faster they arrive at solutions that provide real business or mission value, cost effectively, and in time to be valuable.

4. Possessing a Consistent Record of Failure

Agile is not a silver bullet.  Agile, on its own, does not solve problems, it makes them plain for everyone to see so they may take action.  Transforming organizations from traditional project management structures to Agile requires courage, vision, and competence.  Organizations with long and consistent track records of underperformance and failure are most likely to fail at implementing Agile transformations.

Underperforming organizations may actually be missing core competencies that must be developed or acquired prior to embarking on any type of organizational transformation.  Such organizations are more likely to corrupt their chosen Agile approach to accommodate their deficiencies.

Conclusion

While most organizations would benefit greatly from attaining business agility, many do not possess cultures amenable to Agile transformation.  However, cultural change results from changing behavior.  Organizational leaders maximize the odds of successful Agile transformations by acknowledging organizational culture problems and addressing them before and during Agile transformation initiatives.