Software developers are “creatives.”  Most don’t fit the stereotypes of artists, designers, authors, etc.  However, even those who typify the hyper-analytical “software engineer” persona crave opportunities to work on projects where they can express their creativity by building great solutions. 

Many imagine (and often start) “green field” projects that allow them to do just that.  They want to avoid integrating their work with previous (legacy) implementations and accommodating technical and project management constraints.  They long for autonomy in deciding what to develop, how, and with what.

Most often, however, this drive for autonomy clashes with the realities of creating successful products and services.  Success entails more than writing code.  Also, developers, like practically everyone else, expect compensation for their work.  Unless the effort is altruistic or some sort of hobby, the needs of sponsors, stakeholders, customers, and users influence what the solution becomes, its development, and the mix of technologies and tools employed.

Every software development project, regardless of the chosen development or project management approach, must address the phases of the software development lifecycle (SDLC) and the constraints of the “Iron Triangle”:  Cost, Schedule, and Scope.  There is no free lunch.

Balancing Team Autonomy and Project Control

Striking the right balance between development team autonomy and project management control involves aligning software development and project management approaches to deliver the greatest value possible.  That balance is different for every organization and project.

Before Agile, the need to subordinate software development practice to the needs of project management was an unquestioned assumption.  Documentation and processes that supported project planning masqueraded as engineering or technical artifacts.  It did not matter that those detailed requirements, design documents, and development plans became obsolete doorstops soon after development started.  What mattered was that they provided information with which to create management plans, schedules, and budgets.

Two major factors opened the door to experimentation with iterative and incremental software development and delivery.  The first was the decades-long track record of failure amassed by traditional software projects in delivering complete solutions on time, within budget, and at expected levels of quality.  The second was the impact of technological advances that flipped the cost differential between computer memory and employees (i.e., employees became more expensive than computing resources).

Differences between Software Development Approaches

We now have multiple software development approaches to choose from.  Each balances team autonomy vs. management control differently based primarily on how they address the following priorities:

• Adaptive vs. Predictive Planning
• Tactical vs. Strategic Focus
• Innovation, Flexibility, and Speed vs. Control and Risk Avoidance
• Iterative and Incremental Delivery vs. “Big Bang” Delivery

Adaptive vs. Predictive Planning

Figure 2 above depicts some of the most common software development approaches on a spectrum, from most adaptive on the left side to most predictive on the right side. On the adaptive extreme (cowboy coding):

• Little to no planning occurs prior to or during the project
• Developers execute project activities in an ad hoc fashion, based on their own judgement and project needs, rather than following a deliberate plan
• They have a free hand in choosing what they develop, design approaches, technologies, tools, etc.

As we move from left to right on Figure 2, software development and program management process overhead increase.  So does the amount of planning done prior to software development.

As Figure 2 depicts, planners have latitude in tailoring and implementing software development approaches.  Different choices yield different amounts of process burden and upfront planning. Ultimately, project and technical constraints and circumstances shape the implementation of all software development approaches.

On the predictive extreme, project activities are tightly coordinated and sequenced based on plans that encompass the entire development project, from requirements definition to release.

Rather than performing most project planning during the Planning and Analysis phases of the SDLC, Agile spreads planning across its entirety. Agile software development executes SDLC activities as one integrated flow of work that grows and evolves solutions iteratively and incrementally.

Tactical vs. Strategic Focus

As we move across the software development approaches, from left to right, solution alignment with organizational strategic aims increases.  This is because predictive software development approaches are typically top-down in nature. Adaptive development projects are more closely aligned with short-term tactical goals and deliver capabilities in smaller increments than predictive development projects.

Alignment between strategic and tactical imperatives is possible and necessary for both predictive and adaptive approaches. However, the inherent top-down vs. bottom-up nature of predictive vs. adaptive planning results in very different ways of achieving strategic and tactical alignment.

Learn how Scrum aligns organizational strategy and IT via a product management mindset.

Innovation, Flexibility, and Speed vs. Control and Risk Avoidance

Choosing the right software development approach also depends on project sponsor and stakeholder priorities.  Projects best suited for an adaptive development approach prize innovation, flexibility, and speed.  Predictive software development approaches are most appropriate in organizations seeking to enforce top-down control and avoid risk.

The nature of a project’s technology and business/mission domains also influences choices between software development approaches. Projects best suited for a predictive approaches:

• Have stable and well-understood technology and business domains
• Standard and routine solution approaches
• Project staff experienced in delivering systems of similar size and scope

An adaptive software development approach is best for innovative (non-routine) technical projects and for projects that require ongoing refinement and experimentation to arrive at the most optimal solution.

Some software projects in regulated industries or government may require predictive software development approaches.  Regulatory oversight increases the need for program management control and risk avoidance.  Most software system development projects do not require that level of oversight and control, thus making them likely candidates for adaptive software development approaches.

Iterative and Incremental Delivery vs. “Big Bang” Delivery

Another defining feature of software development approaches is whether they deliver capabilities iteratively and incrementally or as monolithic “Big Bang” releases.  While most organizations deliver software solutions incrementally across multiple releases, many develop large increments of capability over months or years. Such release increments are essentially Spiral or serial Waterfall projects.

Iterative and incremental SDLC approaches, like Agile, increase the frequency of releases and shorten the time between them considerably.  For example, software teams that employ Agile approaches such as Extreme Programming (XP) and Scrum develop release candidates every two to four weeks. Smaller, more frequent releases lower risk and provide more opportunities to align solutions to current business/mission, organizational, and technical circumstances.

Conclusion

Control is a vital element of agility.  Accelerating the pace of software delivery requires some degree of control and flexibility.  Overemphasizing control slows software development and decreases flexibility.  Underemphasizing control can turn flexibility into chaos, robbing software development teams of both planning and process stability, thereby slowing progress.

The term “Cowboy Coder” is a pejorative for a reason.  Developing functionality without disciplined technical and project management processes leads to low-quality implementations and project failure.  It also makes it harder to align solutions to market, business, mission, and oversight requirements.

The opposite extreme is ill suited for most software development efforts.  Most software development efforts do not need the amount of process and documentation overhead Waterfall software development brings. In fact, the predictive planning necessary to carry out Waterfall process and documentation runs counter to the way modern software systems are developed.

Software development teams must not ignore project and technical constraints.  Nor should they try to address every constraint and risk prior to development.  Instead, they should leverage them to create better solutions through inspection and adaptation, iterative and incremental development and release, and adaptive planning.

Recently, I visited an Agile-related Reddit forum/subreddit.  On one thread, developers were chiming in their complaints about Agile software development.  One comment caught my attention.  The commenter stated that, “Agile treats developers like children.”

That comment reminded me of a previous employer who had just purchased JIRA licenses and was excited at the prospect of using the tool to keep an eye on what his company’s developers were working on.  I remember thinking how ironic it was that the greatest benefit the business owner saw in a tool ostensibly designed to support the work of Agile teams was the ability to surveil his employees.  If that was not a clear subversion of Agile principles, I don’t know what is.

That anecdote crystalizes the point of this article:  Agile’s emphasis on greater visibility into progress achieved by Agile teams can easily be turned against them.  In other words, increased visibility over the work performed by Agile teams often leads to more intrusive management oversight that diminishes the ability of teams to manage their own work (self-organize).  

This article is the second of a four-part series about the reasons so many developers are fed up with Agile and so many organizations fail to achieve the benefits touted by Agilists, consultants, and tool vendors.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

• The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
• Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.
• Changing Expectations for Developers – Developers are moving away from functioning as “order takers” focused almost exclusively on coding to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

Every article in this series describes each dysfunction, root causes, and suggests solutions that align with Agile values and principles.

Greater Visibility & Oversight with Less Control

One of the features of Agile is its “radical transparency.”  Radical transparency denotes how Agile teams communicate openly with other teams, project sponsors and stakeholders, and customers and end users as they collaborate to define, build, verify, and validate solutions.  Part of this communication involves keeping others apprised of team progress and the impediments they encounter.

The point of this openness is to foster an environment where everyone involved in sponsoring, managing, developing, and using solutions has the information they need to collaborate effectively without impinging on the flow work established by developers. 

Thus, Agile teams prioritize open communication.  For example, a common Agile practice is the use of “information radiators.”  An information radiator is any type of visual display, placed in a plainly visible physical location or accessible virtually, that allows product or project stakeholders to review team and project information at a glance.

Unfortunately, such transparency is often abused by management overly focused on maximizing efficiency and resource utilization.  When the pendulum swings too far in that direction, process execution and status tracking supplant delivery of valuable capabilities as measures of progress.  This outcome reflects a misplaced faith in achieving successful outcomes merely by following processes.

Rather than supporting Agile team self-organization, management steadily increases control over how development teams work.  Eventually, developers are diminished to the status of order takers with little actual control over what they work on and how.

Agile’s Take on Processes and Tools

One of the values of the Agile Manifesto for Software Development is that Agilists value, “Individuals and interactions over processes and tools.”  While Agile teams leverage processes and tools to improve how they work, they do not use them in ways that attempt to replace face-to-face or real-time interactions or communication.

Agile is not “anti-process.”  Agile is against confusing process execution with actual progress and instituting wasteful or counterproductive processes.  Agile neither prohibits nor prescribes processes.  Instead, it advocates for environments where self-motivated individuals self-organize and develop “just enough” process to support their work.  Agile’s culture of inspection and adaptation also applies to processes, as teams improve how they work.

Productivity tools can significantly increase Agile team productivity and effectiveness.  For example, not every interaction in and among Agile teams can be face-to-face.  While Agile best practice is to collocate teams, doing so is not always possible.  Agile development teams with members who work remotely, and enterprises staffed entirely with remote teams are common.  A project team may be too large to sustain ongoing direct communication between members.  Tools bridge distances and time zones between collaborators and help manage indirect communication.

Also, requirements management tools help document and track requirements, manage work, and report progress.  Agile requirements artifacts are purposely lightweight. They work as reminders for future conversations that will flesh out details.  However, even lightweight requirements become difficult to manage as they grow in number.

For far too many teams, however, this nuanced approach to tools and processes is crushed under the weight of traditional project management thinking. 

Surveillance through Productivity Tools

Traditional management shifts the focus from measuring business or mission value delivered to customers and end-users to measuring how faithfully and efficiently development teams follow processes.  Developers become process execution resources to manage.  Under this shift, productivity tools cease to serve Agile team self-organization and become surveillance tools for management.  Some of the negative consequences of this shift include:

• The amount of time developers spend using tools to document and report on their work begins competing with the amount of time they spend developing features.
• Management reconfigures “Agile” collaboration tools to track traditional project management productivity metrics that do not align with Agile.
• Management rejects Agile-based reporting metrics and formats (e.g., burn-up and burn-down charts) because such reporting does not support the traditional project management approach they are used to.
• Face-to-face or real-time collaboration are replaced by handoffs brokered by tools.
• Since management’s appetite for status is insatiable, productivity tools become information funnels for management and cease providing meaningful value to developers.

Ultimately, the entire Agile approach is derailed as management places traditional project management expectations on Agile development teams.

Forced Commitments

When developers lose control over how they work, they also lose control over how much work they take on.  As I stated in part one of this series, Agile teams align the amount of work they commit to with their collective capacity to finish the work within a period of time.  Software development teams that do not have control over the amount of work they commit to are not Agile teams. 

Agile teams estimate their own work.  They negotiate with product/project management regarding what features they will deliver and when.  They also determine release schedules collaboratively.  For these estimates to be accurate, teams need to trust that management will respect their assessments and refrain from treating estimates as deadlines.

Agile manages projects through scope.  So, the goal is to work on the highest ordered/prioritized features and periodically assess progress.  Estimates improve as teams learn from engaging in the work and stakeholders get a better sense of what they truly need.

Commitments are not do or die promises.  They are honest professional assessments of what teams can get done during a period of time.  The team endeavors to achieve everything it committed to while maintaining a sustainable pace.  While some amount of overtime is necessary in most projects, it is not a way of life in Agile projects.

Holding commitments over the head of developers ignores the fact that team commitments are based on estimates.  It also pressures developers into cutting corners or inflating estimates and committing to less work.  Agile requires a management approach based on collaboration and mutual respect.  Micromanaging the work of developers through tools and pressuring them to overcommit does not engender trust or respect.  Taylorist notions of maximizing resource utilization have no place in Agile.

Finally, pressuring teams to overcommit sends the message that Agile practices do not matter.  Why should a Scrum Team bother with Product Backlog refinement, estimating user stories, and committing to Sprint goals if the resulting plans are unrealistic and ignored by management?

Proposed Solutions

The solution to this problem boils down to educating management about the fundamental paradigm shift from traditional project management to Agile.  Too many managers are under the impression that Agile is something developers do while everyone else in the organization continues to work as they always have.  The decision to adopt Agile comes with a commitment to continually shift the organization towards a customer-centric, product management mindset.  This involves establishing the following practices:

• Adopt a product management approach to solutions/systems development
• Focus on establishing and nurturing high-performance teams
• Stress the need for planning and managing customer and end-user expectations through commitments
• Foster collective team ownership of the work
• Encourage managers and leaders to stand up for teams that feel pressured to overcommit
• Enforce development team ownership of team/Sprint backlogs

Conclusion

People develop software.  Software development is creative work.  The best software solutions come from skilled, talented, and intellectually curious people who are passionate about their work and the software they create.  They are professionals who take pride in delivering quality software.  They understand the inefficiencies and costs associated with low quality designs and implementations.  Thus, they strive for technical excellence.  Agile advocates for organizational environments that nurture this sense of professionalism and ownership by empowering teams to self-organize.

Agile software development is not for the timid.  There is nowhere to hide on an Agile team.  Agile team members are accountable to each other and collectively commit to delivering value in time to be valuable.  Agile teams are highly transparent and accountable environments.  As such, members need some degree of “psychological safety”, exemplified by trust, respect, and the belief that they will not be punished for making a mistake.  Imposing a traditional project management mindset on Agile teams puts members in the undesirable position of being held accountable for results without having the power to affect change.

In the next article of this series, I’ll discuss how top-down-driven Agile adoption initiatives fail to include the perspectives and expertise of the people most affected by the change, the developers, and how doing so sets up Agile adoptions for failure.