There is a misconception that agile is just about processes, and primarily for software development. This is a narrow view and as long as we see it that way, we deprive ourselves of the real benefits of Agile.
Agile is much more than that. It is an approach and a way of working (Agile Method). It is a way of thinking and a state of mind (Agile Mind). In short,
Agile = Agile Method + Agile Mind
Even in software development, it is not limited to a few processes but can be applied to all processes. In fact once the Agile is used fully for software development, it will start being used in all our walks of life. After all, it is one life and can’t be compartmentalized.
In next few weeks, let us explore this line of thought in detail.
Recently I across a job description for the Scrum master role. It started with the mandatory sentence “Scrum master serves and protects the team”. But beyond that followed a long list of activities, mostly of administrative nature, which had virtually no relation to the serve and protect part. This made me start thinking. Here is how I would like rephrase the job description.
A Scrum master must be acutely aware:
• Aware of Agile, not just Scrum
• Aware of the power of agile tool that the team uses, both in terms of how it can help or harm; and
• Aware of the potential of his team members and inhibitions harbored by them
Why does he need to be holistically aware of agile and not just limited to Scrum?
Scrum is a good framework but it is just one way of doing agile. There are others, and those can further enrich the team’s capabilities and make their life easier.
There are agile principles which can guide him when in doubt. For example the one called “Simplicity” is so beautifully defined as “The art of maximizing the amount of work not done”, which can serve him in a vast range of situations.
Why is the agile tool so important?
Initially agile was tentatively tried out for simple cases, hence the manual information sharing and face to face interactions were good enough. But as it has matured over the years, now it is being confidently practiced for much more complex situation, for example bigger projects in offshore development mode.
In such situations using a good agile tool is a must. Otherwise there is an excessive overhead to keep everybody in sync.
A good tool also can capture lot of useful data through the transactions it handles, and provide rich reports based on the data provided it is reliable. This means all the relevant data is in the tool and not outside it.
Secondly, if the tool has to automatically capture the time related information, such information must enter the tool as soon as it happens. It takes discipline to do so, and the Scrum master must take care of it.
The tool provides benefit because it automates lot of manual work. But such automation has its cost and introduce certain degree of rigidity. The Scrum master must understand these aspects and guide all concerned, including the team, in a right direction.
Why do people matter?
People bring unique potential and valuable diversity. But this also introduces its own challenges.
One is the assumptions and beliefs held by the people, which sometimes can severely restrict or distort use of agile.
Secondly neither processes nor tools have any emotions, but people do. This requires very careful handling. The Scrum master should appreciate and accordingly serve and protect the team.
[Note: The term “serves and protects” can sometimes be confusing and open to multiple interpretations. If in doubt, look at a mother who doesn’t feel it below her dignity to do whatever it takes to look after her child’s interests. She doesn’t think twice when the child is threatened by an external or internal challenge and rushes to its rescue. And she leads by example. She truly personifies the role of a Scrum master.]
In short to be effective a Scrum master must grasp the essence of his role and love being one. Otherwise any amount of knowledge or training won’t help much. He should really understand agile in its totality, know the details of the tools used, and empathize with people he deals with both within and outside the team.
The three principle entities in any software development structure are, releases products and teams. I have seen a wide range of permutations and combinations, from very straightforward to quite complex. A simple and flexible process architecture is outlined below, to deal with range of such situations.
Product is the central entity. Any software system worth its salt is not a single homogeneous product, but actually a product suite which contains multiple product lines. Each of these product lines have their individual backlogs to work from, and hence each can be considered as a separate project for the purpose of this discussion.
On one side, each project needs one or more teams to do the actual work and take the project to its next stable state. Currently I am dealing with an interesting situation, not uncommon these days, where most of the teams are specialized in their own area, while few others are generic cross-functional teams.
On the other side, stories from multiple projects need to be complete before a given release. So this is a many-to-many situation on both sides of a project. If we are able to come with a simple process architecture for this complex situation, hopefully it can be equally applicable to situations that involve one-to-many or even one-to-one relationships. Key to this solution lies in making the main three entities loosely coupled.
We are using agile approach for software development. Hence it needs a shift in the way we look at Scrum Kanban and Lean. Rather than treating them as independent methodologies, we are looking at them as just tools which we can use in combination as needed. Something like when we have a hammer spanner and Screw driver, we use a hammer to hit something in, and then tighten with a screw driver while holding it with a spanner. Once we are clear about the specific benefits & limitations of each tool, we use them just in the right way.
Each project has its own backlog, and the epics and stories are defined at the project level. Agile provides us a way to achieve incremental stability through sprints. The teams work on these user stories to complete the sprints. Releases depend on timely completion of these stories, which is managed through sprints. Scrum works for the projects. Team doesn’t need its own sprints. It can easily pull only the relevant stories from multiple projects on a single Kanban board. In the same way, each release manager can design his own Kanban board which at a single place shows him all the relevant information.
Stories have their own lifecycle, which may extend on each side of a sprint. We need a way for the teams to visualize their current work status as well as progress, for the entire span of their involvement. Each team may have different span depending on the functional compositional of their team. For example, architects may be involved primarily in the grooming activity, product owner & business analysts in grooming certification and acceptance, while the developers (including testers) primarily in the development and certification stage. Being able to visualize their work would help not only in having everybody interested on the same page, but would also provide early warning signals on the risks developing.
The holistic view, focus on customer value, and constant attention to avoiding waste provided by Lean fits well into this picture.
Since we are not using Scrum kanban or Lean alone by itself, there will be interesting challenges that will surface and so will have to be handled. More to follow; stay tuned.
Processes have been there, and will continue to be there. They were required when we did waterfall; they were there when we did incremental development; they are here when we do agile development, and they will be there when some new way of building software will emerge in future. They are a given.
But what agile brings to the table is something unique, and to really benefit from it we need to grasp this uniqueness. But before we do that, we need to understand the processes better.
Processes love to be left alone. In a way it is good, because it releases our attention to focus on things that matter most. But from time to time, our needs change and our context also changes. If the processes can’t keep up with these changes, they start becoming stale and gradually become outdated. They become more of a pain than pleasure.
There is another challenge to keeping the processes in tune with the changing needs. There are vested interests which make processes into a religion. And like all religions, the middlemen make things more complex and rigid than they need to be.
One of the possible solutions is for the agile teams to take charge and decide what works for them. The role of an agile coach would then be to empower the teams improve their processes using agile principles, rather than defining and enforcing the processes for them.
Agile manifesto, which was primarily aimed at software development, can be suitably modified as below to guide the teams for taking responsibility for improving their own processes. The possible agile manifesto for agile teams could be,
“We as an agile team are uncovering better ways of executing our work, by doing it and helping others do it. We value,
• Human collaboration over impersonal processes
• Usable practices over borrowed processes
• Empirical approach over defined processes
• Responding to reality over sticking to a plan”
To summarize, processes need help from us humans to stay alive to the reality. And we need help from Agile to do so.
Definition of done and acceptance criteria are two important components of a requirement captured in the form of a user story. However in my interactions with agile teams, I have often noticed lack of clarity about how they differ from each other and why the distinction is important. Here is how I see it. Responses from the readers are welcome.
Lean has a concept of %C&A (Complete and Accurate) used widely in value stream mapping. While the end user interacts with the system, at different stages of the interaction he has a sense of how far his needs are being fulfilled, both in scope and correctness. Scope is about what is included, and what is not, in the requirement. Correctness is about how accurately the aspects in scope are accurately implemented. The user experience at each point of interaction with the system is the net result of both; and leads to either satisfaction / delight or dissatisfaction / irritation.
The crisp three part user story format, I as (a role) want (the expectation) so that (value I would get), is very useful but not sufficient. It must be supplemented by the details of C&A. Confusion between the two probably arises because of use of the term “Done” in definition of done. Another way of expressing these two terms in a simpler format could be; “This includes” for scope, and “Ways to verify” for accuracy.
Who would benefit by this clear separation of the intent? Almost all the roles involved.
When a business analyst captures the requirement in discussion with the end users or their proxy, adding “This includes” to the three-part format will clarify the scope. As the story is groomed with involvement from the development team, further aspects of “This includes” will be uncovered and the “Ways to verify” will be captured. UX specialists can complement this with their inputs to ensure that the implicit user expectations are not forgotten.
When the story reaches the first point of stability “Groomed”, the user feedback can be obtained and alignment reached. This will also provide an opportunity to agree on the user acceptance tests and avoid defect leakages from the requirements.
The distinction will help the developers, including the testers, to define the tasks correctly to ensure the scope is fully covered. QA can start working on the certification criteria keeping in mind the UAT (User acceptance tests) and share them with the Dev-testers to make sure those are checked before the story reaches the next point of stability “Developed”. This will avoid defect leakages from the “Development” stage.
This stability will help the QA + UAT to focus more on making sure that the story is not only complete in itself but fits well with the rest of the system before taking it to stable “Certified” stage. Chance of defect leakages to production are thus minimized.
At this point the product owner would be more confident that the scope is completely covered, and it has been accurately implemented. He can therefore focus on making sure of the overall integrity and the expected user experience. The UX specialists can provide him valuable help. This will enable him to take the story to the next stable state of “Accepted”.
So in short, the difference between definition of done and acceptance criteria can be clearly separated without confusion if we use simple and clear terms to depict them. This clarity will help those in a variety of functional roles and help prevent defect leakages at each point in the system as story completes its life cycle from “Open” to “Closed” through different stages of incremental stability.
CPIE captures what a customer values most. It stands for Certainty / Productivity / Innovation / Elegance. The concept can also be extended to internal customers.
When agile manifesto was announced in Feb 2001, twelve supporting agile principles were also identified. Even after 13 years very few people know about these principles; even those who are using agile practices in their work.
These 12 principles can be broadly divided in 3 categories.
Customer: Satisfy Customer | Working Software | Deliver Frequently | Sustainable Development
People: Motivated Individuals | Work together | Self-organization | Face to face
Process: Welcome Change / Technical excellence / Inspect & Adapt / Simplicity
It would be interesting to see how these agile principles can help and guide us to achieve CPIE.
Certainty is a perception that customers form based on their interactions with us over a period of time. It is partly subjective and partly objective. To provide certainty, we must have the desire and will to keep our commitments. If we don’t have that, the customers will see it through our actions. But however well we plan, everything is not under our control. Things change, new situations emerge. “Welcome change” mindset helps us to see the new reality because we are not stuck to our existing assumptions and beliefs, and through “Inspect & Adapt” can quickly adjust to the new situation. “Deliver frequently” provides instant feedback about the changes in customer expectations. All this helps to make mid-course corrections to meet the current situations and customer expectations by the promised date.
Productivity is all about output upon input. Output is the value delivered to the customer. Input is the efforts we put in. “Simplicity” – the art of maximizing the amount of work not done – helps us to get more done with less. “Satisfy customer” helps us to be always focused on the customer value. Similarly focus on “Working software” helps to avoid unnecessary activities that are not adding any value.
Innovation is required in two areas; technical and practices. Continuous attention to “Technical excellence” and good design fosters the culture of Innovation. “Motivated individuals” can quickly identify opportunities for practice innovation and when they “Work together” it leads to useable results.
Elegance is an elusive concept which can only be experienced. It has been variously described as neatness, precision, simplicity, tasteful design, dignified gracefulness, and restrained beauty. It requires continuous attention to quality which is possible only with “Sustained development”. To experience the elegance requires full use of multiple senses, which is enhanced with “Face to face” interactions. It is difficult to achieve elegance with command & control culture. It requires “Self-organization” to bring out the best from each team member.
Though we have looked at each element of CPIE separately for better understanding, they need to be approached holistically. Together they produce the kind of synergy that is impossible if attention is divided.
To summarize, understanding the agile principles and related practices is important to achieve CPIE. But even more important is an agile mind to understand and use the right principles / practices as may be required for a given situation.
Have you ever been on a group tour? Everything is planned to the last detail, and executed meticulously. Totally expected, no surprises.
I hesitate to go on such tours. I prefer family tours, where there is a lot more flexibility. You can choose when to get up, where to eat. May be cancel some planned activity and just relax walking around the neighbourhood.
All of us have an internal compass. It takes us in the right direction, at least metaphorically, if not literally. More we trust our compass, more will be the surprises.
As a child we all have gone on a treasure hunt, some of us still do. There is no map to take us to the treasure. Just a few pointers. Even those we don’t have all when we start. When we reach one milestone, we get the next clue. It has lot more adventure and fun.
Isn’t agile software development similar? We go to the end of one sprint, have the satisfaction of achieving “Done”, show to others what we have done and get feedback; our clues to the next milestone. We have our product vision and a broad roadmap. That is our compass. It will keep us in the right direction.
When we get comfortable with our internal compass and trust forces beyond us, an external compass called destiny comes into play. Instead of remaining passive puppets in the hands of destiny, we actively embrace it.
Columbus trusted his internal compass and took to uncharted seas to reach India. The destiny had other plans. Fortunately, Columbus did not have Google maps and the GPS. Otherwise US would have remained undiscovered to the rest of the world.
Indore has a tradition of annual festival of Indian classical music, called “Sanghi Sangeet Sammelan”, which features high quality performances. Last night, we had a good fortune to experience violin concert by Dr. N. Rajam. Her daughter Sangeeta Shankar, and grand-daughter Ragini Shankar accompanied her. The three generations of musicians were performing together.
The performance started on dot. They started with raaga “Yaman”. Since the audience was quite familiar with Dr. Rajam having heard her before. But most of us were watching Sangeeta and Ragini for the first time, so Dr. Rajam introduced them and in the initial parts the three took turns so that the audience could have a good feel of each of them individually. Once we were familiar with playing style of all three, they kept reducing the duration between two change-overs from one to the other. Soon before we realized, all three were playing in unison. Sometimes, even one line of the lyric would be divided between the three and we would not notice it. The handover was so smooth and efficient that the the individuals were transcended and it gave a feeling as if a single person is playing. It was a great feeling and the audience just got focused on the music. It was also a pleasure to watch the players’ expression of joy as they were enjoying a perfect team work. It looked a completely self-organizing team; there was no indication of Dr. Rajam giving even silent instructions to the other two.
They were improvising as they went along and the audience loved it. The collective movement through the raaga was so smooth like a bird flying effortlessly, that there was no indication what-so-ever of a great musical discipline in place. In a classical music, even a small variation from the essential structure would be noticed immediately and frowned upon. It was a great combination of perfect self-discipline, engineering excellence and total freedom to try out different variations spontaneously. Can there be a better example of a perfectly agile team at work?
But the agility didn’t stop there. We had till then not noticed Himanshu Mahant who was accompanying them on Tabla so effectively. To bring our attention to him, Dr. Rajam played a short duet demonstrating how a great support person anticipates the needs of the team and responds correctly and appropriately. Both were enjoying the perfect co-ordination, as was quite visible on their faces.
Dr. Rajam was in complete control of the situation, but there was no attempt to command & control. There was no indication of any tension. Each one anticipated and respected others’ needs and responded accordingly; while enjoying being part of a great team. Dr. Rajam was not just a player but also the team leader, a coach and a mentor all rolled-in-one. There were no separate roles and job descriptions. Yet there was no confusion.
After a little over an hour, before we noticed the first raaga was over and there was a deafening applause, which just wouldn’t stop. The team had got their immediate feedback.
After a short Bhajan, it was 9 pm and interval was announced. But the agility didn’t stop even there. In spite of sitting through an intense performance for an hour and half, hardly anybody stirred from their seats. The organizers sensed the expectations of the audience and did a quick check. As the performers were collecting their instruments preparing to leave and the curtain was slowly closing, it stopped mid-way and the well-known well-respected announcer Sanjay Patel came to the stage requesting Dr. Rajam for continuation of another 30 minutes. After a quick exchange with other team members, she agreed. Here was a clear evidence all-around of value placed on “Respond to change over following a plan” of the agile manifesto.
Finally, after another 30 minutes of riveting performance, it was time for the interval. Here was a clear example of not just an agile team, but agile support, and rather of an agile eco-system. It was a great experience rarely seen even, in our domain of software development as it keeps struggling to be agile.
When we were leaving after the program, I remembered what Dr. Rajam had said while introducing her grand-daughter. She said that in their family, it is a common custom to hand-over a violin to a child when he / she reaches the age of three, and before it realizes how difficult it is to play violin, it has started playing with it. I wondered why we also couldn’t do something similar. How nice it would be if the software industry, rather than waiting for them to complete their education, involved students early enough in their school career and helped them create some simple but interesting software.
Introduction: It is understandable but unfortunate that the very mention of the term software engineering evokes such a strong reaction in the agile community. It is understandable because of the burden of the past when good intentions turned into bad practices and unfortunate because in the process we tend to throw the baby out with the bath water.
Agility: The essence of agility is transparency inspection and adaptation. If we look at the Agile Manifesto, we see that the earlier focus on terms like “processes and tools, comprehensive documentation, contract negotiation and following a plan” had frozen the continuity. This I like to call as Agile -1.
Change over continuity: The terms on the left of the value statements in the Manifesto like “Individuals and interactions, working software, customer collaboration and responding to change” emphasized the importance of a human observer who would inspect the content with the context and adapt the approach to align them. The intention was very good but the potential energy of the earlier imbalance was released and swung the pendulum on the other side. This was a phase of Agile 1.
Agile is now mature: By the end of the first decade, agile has become a mainstream methodology and matured enough to be able to give equal importance to both the sides of the equation and provide a way for the practitioners to adapt their practices to the changing context. It has thawed the frozen continuity and can help us to consider logically the benefits of managing continuity well while also responding to change. So we are entering a phase of Agile 0. At first sight, 0 appears empty and hence meaningless. But being empty also frees it from the burden of the past, and opens up limitless possibilities.
Continuity with change: Let us look more closely at continuity and change. When we start developing a new software system, it is all about change. As it grows, elements of software maintenance start entering the picture. Over a period of time, the proportion changes and ultimately we reach a stage when the bulk of the work is to maintain the existing system with little new functionality added from time to time. Managing continuity becomes more important than managing the change. So at different stages, the mix of continuity and change differs and our practices also need to change accordingly.
Systems perspective: It would help to look at this from the systems perspective. The internal state of the system is represented by the state of its content. Similarly, the external state of the system is represented by the current state of its context. Whenever there is a misalignment between the internal state and the external state of the system, it results in creative tension. This tension can be reduced by suitably changing the content or the context or both. But before this happens we need to inspect the current states of the system, which in turn needs access to the right information in time; in other words transparency.
Real life example: Let us understand this with an example. A person brought up in a traditional joint family marries and starts a nuclear family in a big city. He still carries with him the values of a joint family but has to deal with the expectations of living in a big city. He needs to objectively see this and change either his values or manage expectations of others. How he decides to adapt is up to him but unless he is aware of the misalignment, he may end up bring an emotional wreck.
System states: The internal state of a system is the assumptions beliefs and the values it holds along with the current set of practices being used repeatedly. The external state is the interface projected to others and expectations of others from the system, along with the repetitive interactions. In the above example, the tension is generated by the misalignment between what he is and how he wants to appear to outsiders.
System stability: Apart from the system states, it is important to understand system stability. A stable system has minimum hindrance to the free flow of system energy, or synergy for short. Internal stability comes from well-coordinated actions, while external stability is about well-coordinated interactions with other systems. An internally stable system may or may not be externally stable, and vice versa.
From the software system perspective when we start making any changes to it, the internal system becomes unstable till it goes to the next stable state. The iterative development brought in small jumps from one stable state to another as compared to waterfall. This is also expressed as “potentially shippable product” in Scrum. The external stability is governed by the interface exposed and the changing expectations of others from the system. So during iteration as long as the earlier stable version exposed outside remains in sync with the expectations, there is external stability.
Adaptation options: The concept of internal & external stability is important for deciding between continuity and change while adapting to the new reality, as it becomes visible after inspection. During inspection we observe the internal and external states of the system. If they are misaligned, it may call for change. However, the change may be required to the internal state or external state or both.
If the system is internally unstable, we need changes to make it stable. But we have a choice to either take it back to the earlier stable state or forward to a new stable state. This in turn depends on how it would impact external stability.
If the system is internally stable but externally unstable, we have to decide whether to make external changes or internal ones and accordingly plan our actions. It is not always prudent to start changing the software as soon as new requirements come, without considering the resultant states. Sometimes it may be necessary to delay or deny the requested changes in the overall interest.
So far we have seen responses which were reactive. What if the system is currently stable both internally and externally? Should we let it be or proactively anticipate changes in future? We may decide to either continue the status-quo or to go for change, but we will reach the next level of stability much faster.
Inspection: With this background, let us examine what is involved in inspection. Traditionally, inspection means an action planned to know the current state and more often than not involves measurement, which requires collecting data and analyzing it. There is another part of inspection which is normally neglected. It is about observing the instances of exceptional behavior, capturing them and understanding the hidden cause or causes. Such exceptions have a tremendous potential to give us a glimpse into the emerging reality before it starts showing up through measurement.
Transparency: Reliable inspection depends on transparency. Transparency is the ability to see the reality without any obstruction or distortion. Normally, the persons who see it may not be persons who need to act on it. In case of inspection using measurement, it means that the persons who have access to the raw data do not hide it and those who analyze do not distort it. In case of observation, since the exceptions are not controlled by us, they need to be shared by those observing it to those who can analyze and ultimately to those who can act on it. Since this can’t be defined as a process, it needs to become part of the culture as an informal communication, almost bordering on gossip.
What’s in it for me? How does all this help us? Once we become aware of our attachment to one way of thinking and the knee jerk reaction against the other, we can start exploring the fundamentals at the human systems level and apply them to software development to see them in this new light. It will provide us pointers to examine our current practices and suitably modify them as we see them working. We can also benefit in other areas of our life beyond software.
Explore and share with others: Paraphrasing from the Agile Manifesto Let us together “Uncover better ways of managing our practices, by doing it and helping others do it”. When we are able to do it across the software community and get it into the collective consciousness, we would reach the stage of a perfect balance or in other words Agile 0.0
As per the news item posted by Semat http://www.semat.org on 28th Feb 2012, OMG had put out an RFP for a proposed standard “Foundation for the Agile Creation and Enactment of Software Engineering Methods (FACESEM) RFP (OMG Document ad/2011-06-26)”. Semat working group has submitted a proposal entitled “Essence – Kernel and Language for Software Engineering” on 20th Feb 2012. This is an interesting development and I would like to share my thoughts on those aspects which I believe would be of interest to the software practitioners, including those who are familiar with agile ways of doing things.
It is interesting to note that the RFP asks for an “AGILE” foundation for creation and enactment of software engineering methods. This is in sharp contrast to the current practice of methods being created once by process experts to be used repeatedly thereafter by the practitioners. The term “process”, which is so widely used currently, is also conspicuous by its absence from the defined terms in the proposal. Instead it works with practice which is defined as “repeatable approach to doing something with a specific purpose in mind” and method which is “composition of practices forming a description of how an endeavor is performed”.
The direction software engineering is likely to take in future is indicated by the key differentiators mentioned in the proposal,
- Finding the essence of software engineering so that it can be applied to and reused across different application domains and software systems of differing complexity
- Working with methods in an agile way that are as close to practitioners’ practice as possible, so that they can evolve the methods and adapt them to their particular context
- Focusing on what helps the least experienced developers over what helps the more experienced developers
- Supporting practitioners over process engineers
- Emphasizing intuitive and concrete graphical syntax over formal semantics
- Focusing on method use over method definition
In short, the proposals by Semat and others leading to a standard can be expected to benefit the practitioners, giving them a greater say in deciding how they go about developing software, or to use the term from the proposal, their “Way of working”. The universal kernel and a common language would also support the concept “think globally act locally” and provide a common foundation to tackle variety of contexts and situations in a consistent way.
When I was discussing this topic with my colleagues, I was surprised to see how quickly they concluded that it is only related to agile software development whereas it is applicable to all kinds of software development, agile or otherwise. It is actually more about how we decide to develop software rather than how we develop software.
This reminded me of Mike Cohn’s Blog “Succeeding with Agile” http://blog.mountaingoatsoftware.com/reflections-on-the-10-years-since-the-agile-manifesto which I had read some time back. As he says, we stopped talking about objects a while ago and similarly we will stop talking about agile. Rather than “agile software development” it is just “software development.” Rather than “agile project management” it is just “project management”— Of course it’s agile.
But for this to happen, I feel we will need a greater understanding of the opposing forces at play before we can optimally balance them to suit specific situations. Few examples of such universal opposing forces applicable to all human systems are,
- Defined versus empirical
- Transparency verses privacy
- Inspection verses automation
- Planning versus preparing
- Compulsion verses discretion
- Early verses late binding (of decisions to actions)
- (Internal) cohesion versus (external) coupling
Does the above thought process make sense? What are your views on this?