Author: Efrain (Frank) Velazquez

Handling Non-Functional Requirements in Agile Projects

Yes. You have to address non-functional requirements in Agile software projects. Here’s how.

In any software project important enough for sponsors to spend significant amounts of money on, the debate over how to handle non-functional requirements (NFRs) sooner or later comes to a head.  Many people who sponsor, manage, and execute development of software systems simply do not understand NFRs.  This results in increased risks to software projects, Agile or not.  This article explains NFRs in the context of requirements management and how to address them in Agile projects.

Requirements Management

Requirements management is the process of documenting, analyzing, tracking, prioritizing, and controlling changes to system requirements.  Development teams accomplish this work in collaboration with project sponsors, stakeholders, customers and end users (i.e., the Stakeholder Community).  Since software systems change and evolve throughout their useful life, requirements management is a continuous activity, not just something that occurs during development projects.

What is a Requirement?

A requirement is a singular, documented, physical or functional need that a solution (functional or procedural) aims to satisfy.  Requirements specify the things a solution must do and how well it must do them.  The former are functional requirements.  The latter are non-functional requirements.

NFRs Explained

Functional requirements are intuitive for most people.  People feel comfortable voicing what they need a solution to do for them.  The tricky part is characterizing how well a solution must do those things.  This points to how the solution performs or behaves while performing useful work for customers and end users.

For example, a bottle capping machine at a soda plant designed to cap ten bottles per minute would be woefully inadequate if the plant produces 50,000 bottles of soda per day.  Assuming the plant operates 24 hours per day, the machine would need to cap bottles at a rate of at least 35 bottles per minute just to keep up.  That rate may still be too slow in relation to other plant operations and distribution schedules.  No matter how well that capping machine caps bottles (functional requirement) it must do so at an adequate throughput rate (number of bottles per minute). 

A software solution example might entail the number of concurrent users an application must support.  Perhaps the application experiences large swings in the number of users depending on circumstances.  For example, a retail sales website will likely see surges in users around Black Friday.  No matter how well that application handles normal traffic, it must be designed to handle such surges seamlessly.

This is when we get into discussions about how a solution behaves.  In software engineering, we often characterize solution behavior in terms of “-ilities”.  I list examples in the table below:

Table 1: Non-Functional Requirement “-ilities”

These “-ilities” apply to the entire system or solution, not just to individual system features or components.  However, each and every part of the system contributes to, or takes away from, its overall behavior.  One underperforming component could degrade the behavior of an entire system.  Thus, it is crucial to take NFRs into consideration when designing and implementing system features and components.  The more frequently development teams integrate them into the larger system and put them through integration and regression testing, the better.

NFRs constrain functional requirements.  In other words, the capabilities designed and developed to address functional requirements must also support or contribute to meeting NFRs at the system level.

Common sources of NFRs include official standards; laws, regulations, and policies (LRPs); and technical specifications (e.g., operating systems, required middleware, hardware specifications).

Handling NFRs Agilely

There is no fundamental difference between how Agile and traditional software development projects handle NFRs.  In either context, we handle NFRs by:

  • Writing well-written NFRs
  • Ensuring functional requirements address relevant performance/behavioral attributes. 

Well-Written NFRs

A well-written NFR describes a system-level behavior or performance expectation both qualitatively and quantitively.  There is no universally accepted standard way to write NFRs.  However, the following guidelines should help:

  • Start with the outcome in mind by providing context about the requirement. 

For example, why is scalability a desirable goal for our system?  The answer is tied to the business or mission processes the system supports.  What does scalability look like from a business or mission perspective?

  • Quantitatively, we must define what the right degree of scalability is.  For example:
    • How many concurrent users per day, hours, minute, second?
    • What data volumes can we expect?
    • How much fluctuation in users and data volumes can we expect?  When?  How often?

The more specific we are about our needs, the better we’ll define functional requirements that realize NFRs.  Focus on identifying needs rather than on defining solution approaches or designs.

Addressing NFR Performance & Behavioral Attributes

Functional requirements realize NFRs.  Agile functional requirements adhere to the INVEST criteria:

Table 2: Agile Requirements INVEST Criteria

Agile functional requirements are customer and end-user centric.  They are written from the perspective of what customers and end users need rather than from the point of view of what the system will provide.

To make this discussion more concrete, I will use the User Story requirements format.  A User Story includes:

  • A description of what the user needs to accomplish an outcome
  • Acceptance criteria that explain the specific behavior and performance characteristics the resulting functionality must exhibit to comply with requirements

User Story Description

The de facto standard format for User Stories is the following:

As <a user, role, or automated client> I want <some capability> so that <desired outcome>

This simple and concise statement tells us:

  • Which specific user, role, or automated client needs the capability or feature:  User Stories serve as reminders for future conversations.  They are not meant to document requirements in detail.  User Stories tell us the types of users we need to talk to or learn more about.
  • The business or mission capability the user needs:  User Stories are written from the perspective of the business or mission capability a user needs to accomplish a desired outcome.  The resulting software functionality will enable the accomplishment of that business or mission outcome.
  •  The desired business or mission outcome the capability or feature will enable:  User Stories answer what tangible business or mission outcome will be achieved as a result of using the implemented feature.  The outcome is often the execution of a business or mission process or task.

User Story Acceptance Criteria

The second part of a well-written User Story are acceptance criteria.  This is where the NFR attributes defined earlier constrain the design and implementation of the resulting functionality.  Acceptance criteria are the results or outcomes the functionality must achieve when executed.  We demonstrate adherence to acceptance criteria through testing.  So, in essence, acceptance criteria are test criteria for the User Story.

Well-written acceptance criteria are testable.  That means that the result is not left to interpretation.  The feature either delivered expected results or did not.  The behavior and performance the feature demonstrates should contribute to (or not degrade) the overall system-level behavior and performance documented by NFRs.

Here is an example:

NFR: The system shall implement authentication and authorization functionality as per corporate security policy XYZ.

Functional Requirement:

As a system administrator (SA), I need to restrict user access to files so that authenticated users only have access to files for which they have access permissions.

Acceptance Criteria:

  1. The SA can grant specific authorized users access to specific libraries or documents.
  2. The SA can revoke access to specific libraries or documents from specific authorized users.
  3. All file access permissions for specific users are automatically revoked if the SA revokes their authorization to access the system.

Conclusion

It is quite common for people engaged in sponsoring, managing, and developing software systems to misunderstand NFRs and struggle in applying them to Agile requirements management.  User Stories make mapping of system-level NFRs to functional requirements easier.  Acceptance criteria link functional requirements to tests that demonstrate satisfaction of NFRs.  Coupled with frequent integrations and regression testing, this approach to NFRs ensures the delivery of expected system behavior to customers and end users.



Why So Many Developers are Fed Up with Agile (Pt. 4)

Changing Expectations for Developers

Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with solution sponsors, stakeholders, customers, and end users (i.e., Stakeholder Community) to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

This article is the last of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in this series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
  • Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.

Difficulties Collaborating with Stakeholders

One of the defining characteristics of Agile is the fostering of close and ongoing collaboration between software development teams and their Stakeholder Communities.  As the fourth Principle of Agile Software Development states:

Business people and developers must work together daily throughout the project

http://agilemanifesto.org/principles.html

Needs, expectations, and feedback from the Stakeholder Community guide what Agile teams build.  Delivering features and capabilities iteratively and incrementally affords more opportunities to inspect and adapt deliverables and team processes accordingly.

The ultimate validation of a solution’s value is use by real customers and end users.  Prior to release, Agile teams test and validate software increments in collaboration with their respective Stakeholder Communities.  After release, development and/or operations teams monitor and evaluate how evolving solutions are used and perform.  Lessons learned inform decisions about upcoming development.

Many Agile projects fail because this vital feedback loop breaks down or is never established.  Achieving effective collaboration between groups of people who hold different ideas about what is needed or important is difficult.  The trust and respect upon which collaboration depends take time and effort to build and are easily undermined.  Avoiding or mitigating misunderstandings, disagreements, and conflicts starts with people understanding each other’s perspectives.

The Developers’ Perspective

Below are some of the most common issues developers encounter when dealing with Stakeholder Community members.

Table 1: Developer Complaints about Stakeholders

There is also the issue of whether development teams have access to the right stakeholders (or any stakeholders at all).  Too often, those who provide requirements do not have intimate understanding of relevant business or mission processes or of customer and end-user needs.  This is especially true in organizations with traditional project management cultures where project or program managers, architects, and senior engineers are expected to “know” customer needs yet rarely interact with them.

The Stakeholders’ Perspective

Collaboration is a two-way street.  Agilists approach requirements from a customer-centric perspective.  Unfortunately, too many developers think about solutions from the perspective of what they want to build.  They define requirements as development tasks to accomplish or as system functionality to develop.  In the eyes of the Stakeholder Community, these attitudes come across in the following ways:

Table 2: Stakeholder Complaints about Developers

Expanding Technical Responsibilities

Along with increased expectations for Agile development teams to interact with stakeholders, the scope of work done by software developers continues to grow.

Back in the 80s and 90s, a software developer was considered proficient if he or she specialized in one operating system and one programming language.  In the 90s and the aughts, many, if not most, developers specialized as front-end or back-end developers. There were much fewer tools to master and most development was custom rather than based on third-party components and frameworks.  Also, the work associated with the full lifecycle management of applications was highly specialized and assigned to different people.

The advent of the Cloud and DevOps changed all of that.  Now developers are expected to be “full-stack developers” capable of developing both the front and backend components of complete releasable features.  Virtual computing and the Cloud broke down operating system-based development siloes. Open-source development wildly increased the number of tools and frameworks available.  Great efficiencies brought about by DevOps came at the price of developers learning DevOps tools, developing automation scripts, and managing DevOps pipelines.

A proficient developer today is a master of multiple programing languages, highly conversant in more than one operating system, comfortable using a plethora of tools and frameworks, and regularly contributes to testing and DevOps automation.  Software developers now perform a lot of work that used to be set aside for specialists such as system administrators, configuration managers, build engineers, testers, etc. While a lot of that work is now automated or abstracted away from developers, the scope of what is considered development work is much larger than before.

Many software developers feel overwhelmed by expectations to keep up with the ever-increasing pace of technological change, consistently deliver value, and collaborate with stakeholders.  Most developers entered the field for the opportunity to develop software for a living and concentrate on becoming better technologists.  They become highly proficient at building software but lack the business or mission insight necessary to better align solutions with business or mission needs.

Proposed Solutions

Development teams do not need to choose among better collaboration with stakeholders, maintaining technical excellence, and delivering value cost effectively and in time to be valuable.  To achieve that balance, we recommend the following actions.

Promote Strong Product Ownership

While not all Agile teams are Scrum teams, Scrum’s concept of Product Ownership is useful for managing Stakeholder Communities and can be tailored to work with non-Scrum Agile teams.  In Scrum, the “product” is synonymous with whatever technical capability a Scrum Team develops, be it a system, solution, application, etc.

A Scrum Product Owner (PO) is responsible for fusing the various needs, wants, and perspectives of product and project stakeholders into one complete, coherent, and cohesive vision. The Scrum Development Team looks to the PO as the primary (but not the only) source of business or mission knowledge and understanding.  The PO enlists the help of the development team and the Stakeholder Community in defining the product/solution but is primarily responsible for outcomes.

An empowered PO is able to do most of the heavy lifting associated with managing a solution’s Stakeholder Community.  While software developers are free to work individually with any stakeholder, the PO ultimately decides what features make it into the product and the nature of those features.  This arrangement provides developers a dedicated Scrum Team liaison to the Stakeholder Community who is well-informed and respected by the community.

Mentor Developers in Customer Engagement Skills

Customer engagement and requirements elicitation are learned skills.  While some people have more of a knack for them, everyone can benefit from training and mentoring in those skills.  Such training would include topics such as:

  • Organizing and Leading Meetings
  • Presentation Skills
  • Active Listening
  • Holding User Experience (UX) Workshops

Investment in these “soft skills” pays off in better communications between development teams and stakeholders, better requirements definition, less rework, and greater customer and user acceptance of delivered solutions.

Align Work to Team Capacity

Development teams have more time to interact with stakeholders if the amount of work they commit to is in line with their capacity to complete it.  As explained in the first article of this series, software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Invest in Technical Training

Another way to open up time for developers to collaborate with stakeholders is by providing access to technical training and setting aside time to take advantage of it “on the clock”.  Done well, this investment can bring about better prepared and more knowledgeable technical contributors, better solution implementations, cost savings due to decreased rework, and a boost in employee morale and goodwill towards employers.

Pass Along Knowledge through Technical Consultants

Often, development teams do not possess specific hard-to-find skillsets or may not have the bandwidth to learn them.  One way to meet short-term development needs while passing along knowledge is by partnering developers with technical consultants.  This learning by doing model accelerates learning, thus providing development teams more time to interact with stakeholders.

Conclusion

Developers and stakeholders benefit greatly from closer collaboration in defining solutions.  The keys to improving that collaboration are:

  • Developing well-managed Stakeholder Communities through a Product Ownership or Management model
  • Aligning the amount of work done by development teams to their capacity
  • Leveraging training and mentorship to open up more time for collaboration with stakeholders

This series covered a lot of ground concerning the frustrations with Agile software development shared by many developers.  I too have participated in “Agile” implementations that failed to be Agile.  What those failed implementations had in common was an inability or unwillingness to let go of traditional project management biases rooted in Taylorism.  The challenge is not in training people on the mechanics of individual Agile approaches.  It is in chipping away the Industrial Age organizational mentality that stifles progress in today’s cyber-driven world. 

The Agile Manifesto is a clarion call to improve how we work by acknowledging the skills, knowledge, talents, and professionalism of today’s knowledge workers.  Agile is an ever-evolving domain filled with practitioners dedicated to improving it.  It is that ethos that keeps me optimistic about leveraging Agile to improve not just software development, but world of work in general.



Why So Many Developers are Fed Up with Agile (Pt. 3)

Top-Down Driven Agile

No one likes being told how to do their job, especially by people less knowledgeable about the work than themselves.  One of the defining characteristics of knowledge workers is that they know more about the work they do than their supervisors.

Knowledge workers amass a wealth of education, training, and experience in their specialties.  Their success depends on how well they apply those intellectual resources to their work.  Knowledge workers do not merely produce tangible or quantifiable products and services. They help employers or clients achieve business or mission outcomes.  Thus, they usually enjoy much greater latitude in determining how they work than workers in traditional service or manufacturing sectors.  Software development teams are made up of knowledge workers.

Unfortunately, traditional project management culture is still highly influenced by Taylorist notions regarding work organization and worker management.  Too many organizations reflexively assume that management should determine how employees work.  This predisposition runs directly counter to the Agile principle of:

Build[ing] projects around motivated individuals, [giving] them the environment and support they need, and trust[ing] them to get the job done.

http://agilemanifesto.org/principles.html

Organizations possessing traditional project management cultures often adopt Agile in ways that largely ignore the perspectives and input of those most affected by the change: software development teams.

This article is the third of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
  • Changing Expectations for Developers – Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

A Reaction Against Traditional Project Management

Agile software development started as a reaction against the decades-old, heavyweight, project management approach to software development commonly referred to as Waterfall.  It began as a collection of “lightweight” software development approaches implemented by a few like-minded, software industry thought leaders.

Realizing their approaches shared a common philosophy, they gathered at a ski resort in Utah in February 2001.  There they drafted “The Agile Manifesto for Software Development”.  Adoption of the Manifesto and its supporting 12 Principles spread quickly across the software industry.  That formulation of what constitutes software development agility helped popularize Agile and shape the ongoing evolution and refinement of Agile approaches. 

Agile was a bottom-up movement started by software developers.  As stated in the first sentence of the Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

http://agilemanifesto.org/

Viewed through the lens of management practice at the time, what these early Agilists advocated was disruptive; almost subversive.  They carved out space within which they could work without the distractions or frustrations of managerial micromanagement and organizational politics. They broke down organizational silos and included customers and end users in defining, verifying, and validating software solutions.  While Agile helped shift software and IT management culture towards becoming more cooperative and participatory, progress has been spotty.

Few organizations fundamentally changed their traditional management cultures to align with Agile.  While many digital native companies established during the first two decades of the 21st century internalized Agile and Lean principles from the start, many (if not most) organizations did not.  Traditional project management biases still color how most organizations view Agile.  They perceive Agile as a new way of developing software rather than a new way of delivering value through software.  For those organizations, Agile is something developers do while the rest of the organization functions as before.

This fundamental mismatch between Agile’s bottom-up culture and top-down traditional project management culture is the primary reason so many Agile implementations go wrong.  It is also why those with the most at stake in adopting Agile often have the least say in its implementation.

Aligning Organizations to Agile Delivery

Many organizational leaders are unaware of the need to align their organizations with Agile’s iterative and incremental mode of value delivery.  In Agile, value is whatever solution sponsors, stakeholders, customers, and end users (i.e., the Stakeholder Community) care about or makes them happy.  This definition equates value with real business or mission value rather than the cost incurred in developing functionality.  Agile teams help organizations achieve predefined and measurable outcomes by delivering valuable software capabilities.

Agile teams determine what to build through negotiation and collaboration.  The Stakeholder Community is actively engaged with development teams in defining requirements and verifying and validating solution increments throughout development.  This way of working affords plenty of opportunities to inspect and adapt emerging solutions as they grow and evolve.  Frequent releases provide the most important validation of all: customer and end user acceptance of solutions and achievement of desired outcomes.  What teams learn along the way informs future development.

Organizations with strong traditional project management cultures are not organized to participate in this form of value delivery and empiricism.  They are organized to outsource the definition and development of software solutions to development teams after a period of requirements definition.  They then wait until the solution is finished to validate its value. 

Outsourcing Agile Adoption

When organizational leadership sees Agile as a process for developers to follow, instead of a culture change for the entire organization, they are more likely to “outsource” Agile adoption to Agile trainers, coaches, and consultants.

Many organizations spend large sums training developers on the mechanics of an Agile approach but fail to internalize the values and principles behind the approach.  The result are development teams that perform the practices of the chosen Agile approach without understanding how those practices foster agility.  Meanwhile, management continues to operate under Waterfall assumptions and expectations.  Organizations end up with crops of newly Agile-certified employees but no plans to apply what they learned towards achieving greater agility.

Many developers feel they do not get enough value from Agile trainers and coaches.  Some trainers and coaches lack enough understanding of the day-to-day work of software developers or the organizational environment to be effective.  Others are Ill-equipped to help organizations move beyond executing team-level Agile practices and establish Agile-based organizational cultures.

Another manifestation of organizational leadership outsourcing Agile adoption is hiring consultants to develop “Agile strategies.”  Rather than leveraging consultants to help them develop Agile strategy, organizational leadership pays consultants to develop their Agile strategy with little organizational input or understanding of the organizational culture.  When this happens, the resulting strategies tend to be long on theory and short on actionable objectives for the organization to pursue. 

“Hybrid Agile” (Scrumfall, Agilefall)

Many organizations struggling to transform from traditional project management to Agile decide to go for a “Hybrid Agile” approach.  “Hybrid Agile” is not a recognized Agile software development approach. It is an attempt to gain the benefits of Agile while avoiding internalizing and applying Agile values and principles. Rather than reevaluating existing processes to achieve greater agility, these organizations contort Agile approaches to fit within existing organizational culture. This leads to a tailoring of Agile misinformed by traditional project management biases.

Development teams working in “Hybrid Agile” projects experience the worst of both Waterfall and Agile.  Teams are expected to:

  • “Plan” their work while being denied control over the amount of work they commit to
  • “Self-organize” while having no say over team structure, tools, and processes
  • Provide “estimates” that are treated as deadlines
  • Document what they work in “Agile” productivity tools to keep management informed
  • Develop what they are told to according to an “approved” design while expected to accommodate customer and end-user change requests

Proposed Solution

Developing valuable software solutions and products takes more than writing code.  Regardless of the software development approach followed, all projects are accountable to their target Stakeholder Communities and must provide business or mission value, cost effectively, and in time to be valuable.  Management and development teams have to support each other to be successful.

Bottom-up Agile adoption typically fails to foster organizational agility beyond development teams.  Top-down Agile adoption tends to lack in-depth understanding of the realities of software development and buy-in from development teams.  A “meet-in-the-middle” approach accounts for the strengths and weaknesses of the other two approaches.  The meet-in-the-middle approach entails:

Prepare Everyone to Work Agilely

While Agile is inherently bottom-up, organizational transformations required to achieve agility are not.  Organizational leadership and management must play an active role in leading the transformation.  However, to do so in ways that do not crush agility, they must adopt a “Servant Leadership” model.  This starts with a recognition that the primary role of management is to empower those who deliver value to the customer and to create work environments and customer relationships in which value providers can succeed and flourish.

A big piece of leading the change involves training, coaching, and mentoring employees, at all levels and roles, to understand Agile values and principles and how to apply them to their work.  This may require different types of training/mentoring or individual study for different roles throughout the organization.  For example, coaching and mentoring for organizational leadership and management could be geared towards:

  • Understanding the differences between traditional project management and Agile
  • Learning and internalizing the tenets of Agile Servant Leadership
  • Aligning organizations to Agile value delivery in ways that align with Agile values and principles

I stress the need for coaching and mentoring.  Setting people loose after a two-day Agile certification class and telling them to “be Agile” guarantees failure.  Most people outside of software development have not worked in Agile environments. Thus, they do not understand how an organization rooted in Agile values and principles would impact how they work.  Organizational agility is not about everyone organizing themselves as a Scrum team. Like any new skill, Agile thinking and practice must be honed to gain the most benefit.

Hire Knowledgeable and Experienced Trainers & Coaches

This point is obvious but warrants explanation: hire knowledgeable and experienced trainers and coaches.  They need experience in more than just teaching classes and holding Agile certifications.  The more experience trainers and coaches possess in developing software solutions or leading their development, the better. 

I temper this with the caveat that the best trainers and coaches possess subject matter expertise and instruction, facilitation, and communication skills specific to their role.  The most technical people are often not the best fit for teaching and coaching Agile.  What are needed are generalists who possess a mix of theoretical knowledge, relevant industry or mission experience, exposure to the work of software development, and strong interpersonal and communication skills.

Hire, Educate, Train, & Mentor Developers for Leadership Roles

Employers in technical fields tend to select out “soft skills” while overemphasizing technical skills.  They do this at their peril.  Agile depends on the development of self-organizing “high-performance” teams.

Agile teams develop leaders.  Hiring well-rounded people, who work well in teams, and developing their technical and leadership skills ensures development teams have the skills they need to self-organize and to build productive relationships with their Stakeholder Communities.

Conclusion

Reconciling Agile’s inherent bottom-up perspective to organizational change with the need for transformative leadership from above is difficult.  There are no one-size-fits-all approaches for doing so.  Successful implementation of any Agile approach requires deep understanding of Agile, the chosen Agile approach, the organizational environment, the Stakeholder Community, and the day-to-day realities of development teams.  It requires discernment and a willingness to make tough decisions.

Software developers are on the front lines of Agile transformations because they create solutions that enable business or mission success.  They have a huge stake in the direction Agile adoptions take because they are directly affected by the outcome.  Thus, they need and deserve a significant voice in their implementation.

In the next and final article of this series, I’ll discuss difficulties software developers face managing Agile and DevOps-driven changes to their role.



Why So Many Developers are Fed Up with Agile (Pt. 2)

Greater Visibility & Oversight with Less Control

Recently, I visited an Agile-related Reddit forum/subreddit.  On one thread, developers were chiming in their complaints about Agile software development.  One comment caught my attention.  The commenter stated that, “Agile treats developers like children.”

That comment reminded me of a previous employer who had just purchased JIRA licenses and was excited at the prospect of using the tool to keep an eye on what his company’s developers were working on.  I remember thinking how ironic it was that the greatest benefit the business owner saw in a tool ostensibly designed to support the work of Agile teams was the ability to surveil his employees.  If that was not a clear subversion of Agile principles, I don’t know what is.

That anecdote crystalizes the point of this article:  Agile’s emphasis on greater visibility into progress achieved by Agile teams can easily be turned against them.  In other words, increased visibility over the work performed by Agile teams often leads to more intrusive management oversight that diminishes the ability of teams to manage their own work (self-organize).  

This article is the second of a four-part series about the reasons so many developers are fed up with Agile and so many organizations fail to achieve the benefits touted by Agilists, consultants, and tool vendors.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.
  • Changing Expectations for Developers – Developers are moving away from functioning as “order takers” focused almost exclusively on coding to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

Every article in this series describes each dysfunction, root causes, and suggests solutions that align with Agile values and principles.

Greater Visibility & Oversight with Less Control

One of the features of Agile is its “radical transparency.”  Radical transparency denotes how Agile teams communicate openly with other teams, project sponsors and stakeholders, and customers and end users as they collaborate to define, build, verify, and validate solutions.  Part of this communication involves keeping others apprised of team progress and the impediments they encounter.

The point of this openness is to foster an environment where everyone involved in sponsoring, managing, developing, and using solutions has the information they need to collaborate effectively without impinging on the flow work established by developers. 

Thus, Agile teams prioritize open communication.  For example, a common Agile practice is the use of “information radiators.”  An information radiator is any type of visual display, placed in a plainly visible physical location or accessible virtually, that allows product or project stakeholders to review team and project information at a glance.

Unfortunately, such transparency is often abused by management overly focused on maximizing efficiency and resource utilization.  When the pendulum swings too far in that direction, process execution and status tracking supplant delivery of valuable capabilities as measures of progress.  This outcome reflects a misplaced faith in achieving successful outcomes merely by following processes.

Rather than supporting Agile team self-organization, management steadily increases control over how development teams work.  Eventually, developers are diminished to the status of order takers with little actual control over what they work on and how.

Agile’s Take on Processes and Tools

One of the values of the Agile Manifesto for Software Development is that Agilists value, “Individuals and interactions over processes and tools.”  While Agile teams leverage processes and tools to improve how they work, they do not use them in ways that attempt to replace face-to-face or real-time interactions or communication.

Agile is not “anti-process.”  Agile is against confusing process execution with actual progress and instituting wasteful or counterproductive processes.  Agile neither prohibits nor prescribes processes.  Instead, it advocates for environments where self-motivated individuals self-organize and develop “just enough” process to support their work.  Agile’s culture of inspection and adaptation also applies to processes, as teams improve how they work.

Productivity tools can significantly increase Agile team productivity and effectiveness.  For example, not every interaction in and among Agile teams can be face-to-face.  While Agile best practice is to collocate teams, doing so is not always possible.  Agile development teams with members who work remotely, and enterprises staffed entirely with remote teams are common.  A project team may be too large to sustain ongoing direct communication between members.  Tools bridge distances and time zones between collaborators and help manage indirect communication.

Also, requirements management tools help document and track requirements, manage work, and report progress.  Agile requirements artifacts are purposely lightweight. They work as reminders for future conversations that will flesh out details.  However, even lightweight requirements become difficult to manage as they grow in number.

For far too many teams, however, this nuanced approach to tools and processes is crushed under the weight of traditional project management thinking. 

Surveillance through Productivity Tools

Traditional management shifts the focus from measuring business or mission value delivered to customers and end-users to measuring how faithfully and efficiently development teams follow processes.  Developers become process execution resources to manage.  Under this shift, productivity tools cease to serve Agile team self-organization and become surveillance tools for management.  Some of the negative consequences of this shift include:

  • The amount of time developers spend using tools to document and report on their work begins competing with the amount of time they spend developing features.
  • Management reconfigures “Agile” collaboration tools to track traditional project management productivity metrics that do not align with Agile.
  • Management rejects Agile-based reporting metrics and formats (e.g., burn-up and burn-down charts) because such reporting does not support the traditional project management approach they are used to.
  • Face-to-face or real-time collaboration are replaced by handoffs brokered by tools.
  • Since management’s appetite for status is insatiable, productivity tools become information funnels for management and cease providing meaningful value to developers.

Ultimately, the entire Agile approach is derailed as management places traditional project management expectations on Agile development teams.

Forced Commitments

When developers lose control over how they work, they also lose control over how much work they take on.  As I stated in part one of this series, Agile teams align the amount of work they commit to with their collective capacity to finish the work within a period of time.  Software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Agile teams estimate their own work.  They negotiate with product/project management regarding what features they will deliver and when.  They also determine release schedules collaboratively.  For these estimates to be accurate, teams need to trust that management will respect their assessments and refrain from treating estimates as deadlines.

Agile manages projects through scope.  So, the goal is to work on the highest ordered/prioritized features and periodically assess progress.  Estimates improve as teams learn from engaging in the work and stakeholders get a better sense of what they truly need.

Commitments are not do or die promises.  They are honest professional assessments of what teams can get done during a period of time.  The team endeavors to achieve everything it committed to while maintaining a sustainable pace.  While some amount of overtime is necessary in most projects, it is not a way of life in Agile projects.

Holding commitments over the head of developers ignores the fact that team commitments are based on estimates.  It also pressures developers into cutting corners or inflating estimates and committing to less work.  Agile requires a management approach based on collaboration and mutual respect.  Micromanaging the work of developers through tools and pressuring them to overcommit does not engender trust or respect.  Taylorist notions of maximizing resource utilization have no place in Agile.

Finally, pressuring teams to overcommit sends the message that Agile practices do not matter.  Why should a Scrum Team bother with Product Backlog refinement, estimating user stories, and committing to Sprint goals if the resulting plans are unrealistic and ignored by management?

Proposed Solutions

The solution to this problem boils down to educating management about the fundamental paradigm shift from traditional project management to Agile.  Too many managers are under the impression that Agile is something developers do while everyone else in the organization continues to work as they always have.  The decision to adopt Agile comes with a commitment to continually shift the organization towards a customer-centric, product management mindset.  This involves establishing the following practices:

  • Adopt a product management approach to solutions/systems development
  • Focus on establishing and nurturing high-performance teams
  • Stress the need for planning and managing customer and end-user expectations through commitments
  • Foster collective team ownership of the work
  • Encourage managers and leaders to stand up for teams that feel pressured to overcommit
  • Enforce development team ownership of team/Sprint backlogs

Conclusion

People develop software.  Software development is creative work.  The best software solutions come from skilled, talented, and intellectually curious people who are passionate about their work and the software they create.  They are professionals who take pride in delivering quality software.  They understand the inefficiencies and costs associated with low quality designs and implementations.  Thus, they strive for technical excellence.  Agile advocates for organizational environments that nurture this sense of professionalism and ownership by empowering teams to self-organize.

Agile software development is not for the timid.  There is nowhere to hide on an Agile team.  Agile team members are accountable to each other and collectively commit to delivering value in time to be valuable.  Agile teams are highly transparent and accountable environments.  As such, members need some degree of “psychological safety”, exemplified by trust, respect, and the belief that they will not be punished for making a mistake.  Imposing a traditional project management mindset on Agile teams puts members in the undesirable position of being held accountable for results without having the power to affect change.

In the next article of this series, I’ll discuss how top-down-driven Agile adoption initiatives fail to include the perspectives and expertise of the people most affected by the change, the developers, and how doing so sets up Agile adoptions for failure.