Agile has become the way we build software.  Across industries, government institutions, and non-profits, it is the default software development approach.  Debates over whether Agile should supplant traditional project management approaches to software development (i.e., Waterfall) have long been passé.  The focus now is on accelerating the product development lifecycle through DevOps.

However, all is not well in “Agileland”.  Many organizations are not enjoying the benefits promised by the Agile gurus.  Many software developers are fed up with Agile.  If Agile is so great, why are so many people, especially developers, souring on it?  The bottom line upfront (BLUF) is that what passes for Agile in many places is anything but Agile.

This article is the first of a four-part series that describes the most common pain points experienced by software developers and other technical contributors engaged in Agile software development, the root causes, and potential solutions.  The pain points covered in the series are:

• The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
• Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
• Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.
• Changing Expectations for Developers – Developers are moving away from functioning as “order takers” focused almost exclusively on coding to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

The Never-Ending Agile Grind

Software development is a demanding intellectual activity.  It requires study, perseverance, and focus.  The work requires time to think and experiment.  Some degree of failure is necessary as long as it yields valuable insights and teams use those insights to continually improve the product and how they work.

The experience of too many developers on Agile teams, however, is working in “feature factory” environments and “death march” projects.  They feel pressured and rushed to deliver code to the point of sacrificing quality.

Sprints become Sisyphean ordeals where overburdened teams are under constant pressure to deliver working code as quickly as possible while also maintaining the existing code base and contributing DevOps pipeline automation components.  The end of one Sprint leads to the beginning of another in a seemingly never-ending loop of stress that eventually leads to burnout. 

Teams are not afforded time to think through and experiment with alternative approaches, leading them to commit to suboptimal approaches that will likely take longer to implement and maintain than more elegant approaches they could have conceived given sufficient time.

“Feature factory” teams are denied time “on the clock” to learn and experiment with new technologies, design patterns, and architectural approaches.  As a result, implementations based on those approaches are often half-baked, filled with technical debt, under-performant, and difficult to maintain.  The benefits expected from adopting new technological approaches are not realized.  Meanwhile system complexity explodes exponentially.

Management consistently chooses delivery of new functionality over addressing technical debt.  This short-sided preference leads to increasingly brittle systems that force developers to expend increasing amounts of energy and focus on fixing bugs and babysitting systems.

Overtime, developers start to feel like robotic coding units who get paid to crank out code.  A mentality of “getting it done on time no matter what” becomes culture.  Any vestige of work/life balance is thrown out the window as the backlog of features grows and the system or product becomes ever more unmanageable.

The Causes

The Agile Manifesto for Software Development and the 12 Principles of Agile describe the characteristics of Agile software development.  Software development approaches that adhere to the values of the Manifesto and the 12 principles are considered Agile.  Approaches that deviate from any of the values or principles are not.  One Agile principle “feature factory” teams violate is principle number #8:

Agile processes promote sustainable development.  The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

http://agilemanifesto.org/principles.html

Why do so many software development organizations and their sponsors ignore this principle?  Many blame the increasing pace of business competition or mission threats for this single-minded focus on production and speed.  While pressure from increased competition is a contributing factor, I posit that the root cause is cultural.  Many (if not most) of these organizations take a project management approach to software development.  Agile, in general, and Scrum in particular, have a product management orientation.

Traditional project management, rooted partly in Taylorism, focuses on managing resources, schedule, and budget.  People are a category of resources managed by managers.  Traditional project management places a very high value on maximizing resource utilization to achieve greater efficiencies.  People are the most important (and expensive) resources to maximize.  In the drive to do more with less and meet tight (often arbitrary) deadlines, management often loses sight of the link between team welfare and morale and superior performance.

Product management focuses on managing the product and how the product is made and delivered.  Through Agile, management shifts its focus from micromanaging developers to collaborating with them in continually improving product offerings as well as the development and delivery of those offerings.  Ensuring development teams are not overwhelmed becomes a major concern since it has such a large bearing on product quality, timeliness, cost, etc.

Another cultural factor that leads to “feature factory” cultures is confusion over the role of planning in Agile.  A popular misconception is that there is no planning in Agile software development.  This misunderstanding leads to underemphasizing requirements definition, analysis, and design which, in turn, leads to badly conceived and executed software implementations.

Agile teams plan constantly.  The difference between Agile planning and planning in Waterfall projects is that Agile planning is adaptive while traditional project planning is predictive.  In other words, rather than attempting to plan the entire project upfront, Agile plans change based on current circumstances and new information and insights.

Just like development teams in Waterfall projects, Agile teams perform all the activities encompassed by each of the phases of the Software Development Lifecycle (SDLC).  The difference is that the “Agile SDLC” (DevOps) does not segregate those activities into discrete scheduled phases arranged one after the other for the entire system or solution.  Instead, every feature/deliverable an Agile development team delivers is the product of its own “mini” SDLC.  The resulting system or solution is the sum of all integrated features.

Potential Solutions

So, what are some concrete steps software development organizations and their sponsors can take to avoid overwhelming and burning out technical contributors?

The first step is understanding and internalizing that Agile development teams are self-organized.  They collectively manage their own work, determine technical approaches, develop their own estimates, report their progress, and help define the functionality they build.  Team self-organization is a core tenant of Agile supported by three of the 12 Principles of Agile:

Principle #11:  The best architectures, requirements, and designs emerge from self-organizing teams.

Principle #5:  Build projects around motivated individuals.  Give them the environment and support they need and trust them to get the job done.

Principle #12:  At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

http://agilemanifesto.org/principles.html

This contrasts with traditional project management practice, in which management is responsible for defining what to build, breaking down and assigning work, developing plans and schedules, and reporting progress.

Align the Work to Team Capacity

Agile teams break down their work with an eye towards delivering valuable functionality within short timeframes.  They can only achieve this if they align the amount of work they commit to with the team’s ability to get it done.  In other words, break down solution scope into slices of capability (features) that can be accomplished within agreed timeframes.  By ordering/prioritizing features along the lines of business or mission value, Agile releases deliver the most valuable features/capabilities first.

Agile teams commit to additional work as they complete existing work.  Rather than having work pushed on Agile teams, Agile teams pull work when they are ready to do it.  Software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Allocate Time for Specific Tasks during Sprints

One way to ensure that Agile teams address technical debt, and other necessary tasks outside the scope of developing new features, is to allocate some percentage of their time to those activities.  For example, during Sprint planning, a Scrum Team may allocate some percentage of the next Sprint to addressing technical debt.  They could do the same for backlog refinement and DevOps automation work.

Establish Innovation & Planning Sprints

The Scaled Agile Framework (SAFe) includes Innovation and Planning (I&P) iterations (Sprints).  I&P iterations occur at the end of each Program Increment (PI), which lasts eight to twelve-weeks.  Each PI includes two to six iterations depending on the length of each iteration (two to four weeks).  Agile teams do not have to adopt SAFe to experiment with I&P iterations or Sprints.

I&P iterations provide teams time to work important tasks, such as the ones below, without significantly disrupting the flow/cadence of development and release of new functionality:

• Explore new and innovative approaches (e.g., experiment with new technologies; evaluate products, tools, and services)
• Work on technical infrastructure, tooling, and other impediments to delivery
• Engage in continuous learning and improvement
• Cross train in new skills
• Engage in longer-term planning (e.g., product roadmap discussions, architecture design)

Conclusion

It is common knowledge that productivity, quality, and innovation suffer when workers are stressed and burned out.  However, too often responsibility for coping with stressful work environments is placed solely on the shoulders of individual employees/contributors while management sticks to outdated ideas about maximizing efficiency.

Agile, implemented in the spirit of Agile, opens the door to sustainable progress and achievement.  Again, development teams that do not control the amount of work they commit to delivering are not Agile teams.  Such teams risk being pressured into over-promising while under-delivering.

In the next article of this series, I’ll discuss how the greater visibility into team activities and progress afforded by Agile is often used to against the development teams themselves.  Just like this article I will also discuss root causes for the dysfunction and suggest solutions.

Last Updated on February 2, 2022