Most people say they are doing Agile when they are actually doing Scrum. This is because Scrum is the most widely used agile method; with a good reason. Scrum provides a complete framework that includes roles, ceremonies and artifacts. Over last 20 odd years of its existence, it has matured and scaled to large projects.
However Scrum does not provide sufficient guidance to the teams about how to manage their work within a sprint. There is no guidance on when to start work on which story / task. It does not provide guidance on how to keep the work moving smoothly without interruptions.
It mainly focuses on capacity utilization and relies heavily on burn-down chart to anticipate the risk of spill-overs. But there are cases where in spite of available capacity there is just not enough time for quality work before the due date. Hence there is a need to look beyond Scrum for help on these challenges.
Scrum is not the only agile method. There are others like Kanban and Lean. Kanban came from manufacturing, especially Japanese auto industry, and was pioneered in software development 10 years back.
Kanban is a light-weight method and is commonly used where the 2-3 week batch mode of Scrum doesn’t fit very well. Such situations include support activities, projects in maintenance mode, or anywhere the requirements come in at random and with a short completion time.
So a common approach is to choose between either Scrum or Kanban, but not both together. However there is an area of Scrum, single iteration or sprint, within which the Kanban principles can be considered for improving the execution effectiveness.
Expected benefits of introducing Kanban within a sprint are,
• Improved Agility –Measured as turn-around time (Elapsed time from the start to finish)
• Higher Productivity – Average velocity per person per week
• Better Quality – Reduced defects and rework
Kanban relies on pull mechanism to decide when to start an activity, and has just three simple principles.
• Limit the work in progress – Complete the work-in-hand before taking up the next task.
• Observe and manage the queues – Long queues introduce an inherent delay to every item. Lengthening queues indicate a capacity imbalance.
• Identify and remove the bottlenecks – Worst bottleneck decides the system throughput. Removing it reveals the next worst.
How Kanban principles help each improvement area:
Limit WIP: Pull system allows JIT re-prioritization (Agility). Uninterrupted attention reduces setup time (Productivity). Focused attention helps do a thorough job reducing defects and rework (Quality).
Manage Queues: Short queue reduces waiting time for all items in the queue (Agility). Reduced waiting time minimizes spill-overs (Productivity). Reduced waiting time provides more testing time before the due date (Quality).
Resolve bottlenecks: Removing bottlenecks Improves flow throughout the system (Agility). Improved flow results in higher capacity utilization (Productivity). Absence of a bottleneck means no need to compete for getting beyond it in hurry (Quality).
For best results it helps to follow a step-by-step implementation. The suggested steps are,
Set up the board to clearly visualize the work:
By default a board normally contains just three states – To Do / In Progress / Done. But the total span of a team’s work typically contains multiple stages. Visualizing them on a board as different columns helps to clearly see the flow. Further breaking the stages in pairs of doing & waiting clearly separates WIP and Queues.
Put in place data discipline:
The team will take some time to ensure that the data in the tool is in sync the with ground reality. But it is important to achieve this discipline to get reliable data for measurement and analysis.
Modify board design to facilitate measurements:
Initial measurements on live data will reveal areas of high variability. The board design would be modified through setting up swim lanes and filters to reduce this variability.
Baseline the current agility / productivity / quality:
Before we can measure the improvements, the current level needs to be captured.
Agility would be measured as the elapsed time between given points in the flow. Agile tools like JIRA provide powerful reports for cycle times and cumulative flow.
When doing Scrum, the productivity and quality measures would be in place to baseline the current levels.
Keep measuring and improving:
After the successful completion of first 4 phases, the teams can start reaping the benefits. Areas of improvement and the extent of benefits achieved will vary from project to project. There is no out of the box solution; but there are available pointers to indicate possible areas of improvement. Continuous improvement needs to become an iterative activity, with a short feedback loop.
Scrum is the most widely used and powerful framework. Its power can be further enhanced by understanding and applying a few Kanban principles. The best place to start is within each sprint.
There is a misconception that agile is just about processes, and primarily for software development. This is a narrow view and as long as we see it that way, we deprive ourselves of the real benefits of Agile.
Agile is much more than that. It is an approach and a way of working (Agile Method). It is a way of thinking and a state of mind (Agile Mind). In short,
Agile = Agile Method + Agile Mind
Even in software development, it is not limited to a few processes but can be applied to all processes. In fact once the Agile is used fully for software development, it will start being used in all our walks of life. After all, it is one life and can’t be compartmentalized.
In next few weeks, let us explore this line of thought in detail.
I saw a news item yesterday about freight train movement between Mumbai and Delhi. It has only one stop at Vadodara in the total distance of around 1400 Km. The track ahead is kept free to ensure “Sustained pace”. This reminded me of my childhood days spent in a railway town. The primary mode of travel for us was the trains, and I still remember the poor goods train waiting patiently in a loop line while a mail train thundered by.
Over all these years, both the throughput and velocity of freight trains have substantially increased. The freight train mentioned in the news is two KM long and carries substantially greater weight from origin to destination. It also moves in an “Expedite” mode in preference to other trains. The terms sustained pace and expedite that came to my mind shifted my attention to agile software development, and I started thinking whether the way we use “Velocity” is correct. Isn’t size delivered per sprint a measure of throughput rather than velocity? I looked up the definition of velocity and found that it is “The rate and direction of the change in the position of an object”. It is closer to speed than throughput.
That shifted my attention back to trains. A passenger train which stops at every station and an express train stopping at only the major stations may carry more or less same number of passengers (throughput), but they cover the same distance in substantially different times (velocity). A passenger train might attain, for a brief period, speed similar to an express train. But its average speed is much lower because of, not only the waiting time at the stations, but more so because of the frequent acceleration and deceleration involved.
This explains why there is so much importance given in agile for a team to take up only a few items at a time, and take up additional items only when those in hand are fully “Done”. Unlike the trains which follow physical laws, agile teams made up of humans have an additional challenge. Once our attention is diverted to something else, it takes time and effort to switch it back to what we were doing earlier. In addition the information we used might have also become stale in the meantime. So if the agile teams want to achieve high velocity and maintain it, they should be aware and manage the number of items being worked on at a time.
I also remembered that when I was waiting in a passenger train for an express / mail train to pass, sometimes the wait was quite short while at other times it was considerably longer. This was because of the dependency on the time at which the higher priority train would pass. And we had no information when that would happen. Agile encourages transparency. A perfectly flat transparent glass allows a viewer to see on the other side very clearly. Similarly, all those involved in any work should have the information, without filtering or distortion, when they need it. Agile processes and tools provide high degree of transparency, and it is for us to make sure that it remains so.
Another way to minimize the harmful effects of dependency is with enhanced and effective collaboration. When cross-functional team members work on the same item, speed and quality with which it gets completed is phenomenal. Hand-offs are the worst enemy of velocity.
In earlier days, there were single tracks between two stations. This resulted in only unidirectional movement at a time. The other train had to wait at the station. Presently with parallel tracks in both directions this is not an issue. Hence in agile we should think of velocity (a vector) and not just speed (a scalar). Rework moves in the opposite direction to the normal work and slows it down considerably.
There is a saying “Words make the world”. We should not mix throughput and velocity or use one in place of another. Commonly used words package lot of information and meaning with it. More importantly, the words also package the emotions. A mere mention of such a word starts the train of thoughts and images associated with it, and recreates the emotions involved as if they are being experienced now. Wrongly using the velocity when we actually mean throughput deprives us of the immense benefits mentioned above that can make our work simpler and life easier.
Recently I across a job description for the Scrum master role. It started with the mandatory sentence “Scrum master serves and protects the team”. But beyond that followed a long list of activities, mostly of administrative nature, which had virtually no relation to the serve and protect part. This made me start thinking. Here is how I would like rephrase the job description.
A Scrum master must be acutely aware:
• Aware of Agile, not just Scrum
• Aware of the power of agile tool that the team uses, both in terms of how it can help or harm; and
• Aware of the potential of his team members and inhibitions harbored by them
Why does he need to be holistically aware of agile and not just limited to Scrum?
Scrum is a good framework but it is just one way of doing agile. There are others, and those can further enrich the team’s capabilities and make their life easier.
There are agile principles which can guide him when in doubt. For example the one called “Simplicity” is so beautifully defined as “The art of maximizing the amount of work not done”, which can serve him in a vast range of situations.
Why is the agile tool so important?
Initially agile was tentatively tried out for simple cases, hence the manual information sharing and face to face interactions were good enough. But as it has matured over the years, now it is being confidently practiced for much more complex situation, for example bigger projects in offshore development mode.
In such situations using a good agile tool is a must. Otherwise there is an excessive overhead to keep everybody in sync.
A good tool also can capture lot of useful data through the transactions it handles, and provide rich reports based on the data provided it is reliable. This means all the relevant data is in the tool and not outside it.
Secondly, if the tool has to automatically capture the time related information, such information must enter the tool as soon as it happens. It takes discipline to do so, and the Scrum master must take care of it.
The tool provides benefit because it automates lot of manual work. But such automation has its cost and introduce certain degree of rigidity. The Scrum master must understand these aspects and guide all concerned, including the team, in a right direction.
Why do people matter?
People bring unique potential and valuable diversity. But this also introduces its own challenges.
One is the assumptions and beliefs held by the people, which sometimes can severely restrict or distort use of agile.
Secondly neither processes nor tools have any emotions, but people do. This requires very careful handling. The Scrum master should appreciate and accordingly serve and protect the team.
[Note: The term “serves and protects” can sometimes be confusing and open to multiple interpretations. If in doubt, look at a mother who doesn’t feel it below her dignity to do whatever it takes to look after her child’s interests. She doesn’t think twice when the child is threatened by an external or internal challenge and rushes to its rescue. And she leads by example. She truly personifies the role of a Scrum master.]
In short to be effective a Scrum master must grasp the essence of his role and love being one. Otherwise any amount of knowledge or training won’t help much. He should really understand agile in its totality, know the details of the tools used, and empathize with people he deals with both within and outside the team.
The three principle entities in any software development structure are, releases products and teams. I have seen a wide range of permutations and combinations, from very straightforward to quite complex. A simple and flexible process architecture is outlined below, to deal with range of such situations.
Product is the central entity. Any software system worth its salt is not a single homogeneous product, but actually a product suite which contains multiple product lines. Each of these product lines have their individual backlogs to work from, and hence each can be considered as a separate project for the purpose of this discussion.
On one side, each project needs one or more teams to do the actual work and take the project to its next stable state. Currently I am dealing with an interesting situation, not uncommon these days, where most of the teams are specialized in their own area, while few others are generic cross-functional teams.
On the other side, stories from multiple projects need to be complete before a given release. So this is a many-to-many situation on both sides of a project. If we are able to come with a simple process architecture for this complex situation, hopefully it can be equally applicable to situations that involve one-to-many or even one-to-one relationships. Key to this solution lies in making the main three entities loosely coupled.
We are using agile approach for software development. Hence it needs a shift in the way we look at Scrum Kanban and Lean. Rather than treating them as independent methodologies, we are looking at them as just tools which we can use in combination as needed. Something like when we have a hammer spanner and Screw driver, we use a hammer to hit something in, and then tighten with a screw driver while holding it with a spanner. Once we are clear about the specific benefits & limitations of each tool, we use them just in the right way.
Each project has its own backlog, and the epics and stories are defined at the project level. Agile provides us a way to achieve incremental stability through sprints. The teams work on these user stories to complete the sprints. Releases depend on timely completion of these stories, which is managed through sprints. Scrum works for the projects. Team doesn’t need its own sprints. It can easily pull only the relevant stories from multiple projects on a single Kanban board. In the same way, each release manager can design his own Kanban board which at a single place shows him all the relevant information.
Stories have their own lifecycle, which may extend on each side of a sprint. We need a way for the teams to visualize their current work status as well as progress, for the entire span of their involvement. Each team may have different span depending on the functional compositional of their team. For example, architects may be involved primarily in the grooming activity, product owner & business analysts in grooming certification and acceptance, while the developers (including testers) primarily in the development and certification stage. Being able to visualize their work would help not only in having everybody interested on the same page, but would also provide early warning signals on the risks developing.
The holistic view, focus on customer value, and constant attention to avoiding waste provided by Lean fits well into this picture.
Since we are not using Scrum kanban or Lean alone by itself, there will be interesting challenges that will surface and so will have to be handled. More to follow; stay tuned.
Processes have been there, and will continue to be there. They were required when we did waterfall; they were there when we did incremental development; they are here when we do agile development, and they will be there when some new way of building software will emerge in future. They are a given.
But what agile brings to the table is something unique, and to really benefit from it we need to grasp this uniqueness. But before we do that, we need to understand the processes better.
Processes love to be left alone. In a way it is good, because it releases our attention to focus on things that matter most. But from time to time, our needs change and our context also changes. If the processes can’t keep up with these changes, they start becoming stale and gradually become outdated. They become more of a pain than pleasure.
There is another challenge to keeping the processes in tune with the changing needs. There are vested interests which make processes into a religion. And like all religions, the middlemen make things more complex and rigid than they need to be.
One of the possible solutions is for the agile teams to take charge and decide what works for them. The role of an agile coach would then be to empower the teams improve their processes using agile principles, rather than defining and enforcing the processes for them.
Agile manifesto, which was primarily aimed at software development, can be suitably modified as below to guide the teams for taking responsibility for improving their own processes. The possible agile manifesto for agile teams could be,
“We as an agile team are uncovering better ways of executing our work, by doing it and helping others do it. We value,
• Human collaboration over impersonal processes
• Usable practices over borrowed processes
• Empirical approach over defined processes
• Responding to reality over sticking to a plan”
To summarize, processes need help from us humans to stay alive to the reality. And we need help from Agile to do so.
Definition of done and acceptance criteria are two important components of a requirement captured in the form of a user story. However in my interactions with agile teams, I have often noticed lack of clarity about how they differ from each other and why the distinction is important. Here is how I see it. Responses from the readers are welcome.
Lean has a concept of %C&A (Complete and Accurate) used widely in value stream mapping. While the end user interacts with the system, at different stages of the interaction he has a sense of how far his needs are being fulfilled, both in scope and correctness. Scope is about what is included, and what is not, in the requirement. Correctness is about how accurately the aspects in scope are accurately implemented. The user experience at each point of interaction with the system is the net result of both; and leads to either satisfaction / delight or dissatisfaction / irritation.
The crisp three part user story format, I as (a role) want (the expectation) so that (value I would get), is very useful but not sufficient. It must be supplemented by the details of C&A. Confusion between the two probably arises because of use of the term “Done” in definition of done. Another way of expressing these two terms in a simpler format could be; “This includes” for scope, and “Ways to verify” for accuracy.
Who would benefit by this clear separation of the intent? Almost all the roles involved.
When a business analyst captures the requirement in discussion with the end users or their proxy, adding “This includes” to the three-part format will clarify the scope. As the story is groomed with involvement from the development team, further aspects of “This includes” will be uncovered and the “Ways to verify” will be captured. UX specialists can complement this with their inputs to ensure that the implicit user expectations are not forgotten.
When the story reaches the first point of stability “Groomed”, the user feedback can be obtained and alignment reached. This will also provide an opportunity to agree on the user acceptance tests and avoid defect leakages from the requirements.
The distinction will help the developers, including the testers, to define the tasks correctly to ensure the scope is fully covered. QA can start working on the certification criteria keeping in mind the UAT (User acceptance tests) and share them with the Dev-testers to make sure those are checked before the story reaches the next point of stability “Developed”. This will avoid defect leakages from the “Development” stage.
This stability will help the QA + UAT to focus more on making sure that the story is not only complete in itself but fits well with the rest of the system before taking it to stable “Certified” stage. Chance of defect leakages to production are thus minimized.
At this point the product owner would be more confident that the scope is completely covered, and it has been accurately implemented. He can therefore focus on making sure of the overall integrity and the expected user experience. The UX specialists can provide him valuable help. This will enable him to take the story to the next stable state of “Accepted”.
So in short, the difference between definition of done and acceptance criteria can be clearly separated without confusion if we use simple and clear terms to depict them. This clarity will help those in a variety of functional roles and help prevent defect leakages at each point in the system as story completes its life cycle from “Open” to “Closed” through different stages of incremental stability.
When we can have fusion music and fusion food, why not fusion Agile? Scrum and Kanban originated in two different worlds, but blending them together in a right mix opens up huge possibilities.
Scrum has its origins in software development. It evolved as a natural progression from waterfall to incremental to Scrum, while all along shrinking the time gap to the next stable increment. Given these roots it is not surprising that it continues to be a batch process; though with a small batch size of 2 to 4 weeks of work.
On the other hand, Kanban (and Lean) came from the manufacturing world; rather from the auto industry. So it is to be expected that there is a strong influence of the JIT (just-in-time) movement pioneered by the Japanese auto industry, which focuses on continuous flow.
In software development, Scrum started around 20 years back. Initially it was tried with small projects having collocated teams. With greater confidence from the initial successes, the word spread around and it is now being used in large and far more complex projects. This made it necessary to have a number of small cross-functional teams working on the same backlog. As if this was not enough, the offshore development model has added its own challenges; especially in the area of communication and collaboration.
There is also a strong desire to try Scrum for all kinds of situations, and even for organizational areas beyond software development. This led to renewed interest in other agile methods like Kanban and Lean, which though less than 10 years old in software development have been fast catching up. We are currently at an interesting point. There are some attempts at combining Scrum & Kanban, e.g. Scrumban, but these are few and far between. There is tremendous scope to take a fresh look from a higher agile perspective and dispassionately choose agile practices from these two methods as appropriate, without fear or favour.
Though the Scrum community at large appears to be a little skeptical and reluctant to explore other agile methods like Kanban and Lean, the Scrum guide which is the official Scrum reference has already moved towards a more realistic and flexible approach that makes it possible to continue with Scrum and at the same time borrow useful practices from Kanban and Lean.
Here are a few examples from the latest Scrum guide July 2013. Those who are still caught up in the old rigid ways of doing Scrum may find this uncomfortable. But others with an open mind will find it interesting.
• The Product Owner is responsible for maximizing the value of the product and the work of the Development Team. The Product Owner tracks the total work remaining at least every Sprint Review. The Product Owner compares this amount with work remaining at previous Sprint Reviews to assess progress toward completing projected work by the desired time for the goal. This information is made transparent to all stakeholders.
• Various projective practices upon trending have been used to forecast progress, like burn-downs, burn-ups, or cumulative flows. These have proven useful. However, these do not replace the importance of empiricism. In complex environments, what will happen is unknown. Only what has happened may be used for forward-looking decision-making.
• The Sprint Goal gives the Development Team some flexibility regarding the functionality implemented within the Sprint. The Sprint Goal can be any coherence that causes the Development Team to work together rather than on separate initiatives.
• The Sprint Backlog is the set of Product Backlog items selected for the Sprint, plus a plan for delivering the product Increment and realizing the Sprint Goal. The Development Team modifies the Sprint Backlog throughout the Sprint, and the Sprint Backlog emerges during the Sprint.
• During the sprint planning meeting, work planned for the first days of the Sprint by the Development Team is decomposed by the end of this meeting, often to units of one day or less. As new work is required, the Development Team adds it to the Sprint Backlog. As work is performed or completed, the estimated remaining work is updated. When elements of the plan are deemed unnecessary, they are removed.
• At the end of a Sprint, the new Increment must be “Done”. It must be in useable condition regardless of whether the Product Owner decides to actually release it.
In short, the Scrum guide accepts the dynamism in complex situations, keeps the focus on the long term goal and tracks progress towards it sprint by sprint. While it recognizes usefulness of practices like burn-down burn-up and cumulative flow, it appreciates the importance of watching and adjusting to reality, not only by observations but also from data collected from the past. Emergence of powerful agile tools helps to provide progress visualization along with trends and patterns without any extra overhead once there is discipline of timely data updates in the tool. Even at the sprint level, initial planning is done in enough detail to gain confidence in the team’s ability to deliver on time, but the work details emerge throughout the sprint.
How can Lean and Kanban add value to Scrum? Lean brings the focus on end user value. When all the functions involved in the work focus on this value, their interactions automatically become collaborative rather than competitive. Handovers and delays between functions are avoided and the work flows fast and smooth. Kanban provides features to manage the WIP, Queues and bottlenecks which make the flow even more smooth and fast.
How can Scrum add value to Lean & Kanban? Scrum brings specific goals & milestones for incremental stability, which keeps the work on track in the desired direction.
When we combine these two streams together, we get a simple & flexible framework to manage a broad variety of situations from very simple to very complex. More on this in the upcoming posts.
CPIE captures what a customer values most. It stands for Certainty / Productivity / Innovation / Elegance. The concept can also be extended to internal customers.
When agile manifesto was announced in Feb 2001, twelve supporting agile principles were also identified. Even after 13 years very few people know about these principles; even those who are using agile practices in their work.
These 12 principles can be broadly divided in 3 categories.
Customer: Satisfy Customer | Working Software | Deliver Frequently | Sustainable Development
People: Motivated Individuals | Work together | Self-organization | Face to face
Process: Welcome Change / Technical excellence / Inspect & Adapt / Simplicity
It would be interesting to see how these agile principles can help and guide us to achieve CPIE.
Certainty is a perception that customers form based on their interactions with us over a period of time. It is partly subjective and partly objective. To provide certainty, we must have the desire and will to keep our commitments. If we don’t have that, the customers will see it through our actions. But however well we plan, everything is not under our control. Things change, new situations emerge. “Welcome change” mindset helps us to see the new reality because we are not stuck to our existing assumptions and beliefs, and through “Inspect & Adapt” can quickly adjust to the new situation. “Deliver frequently” provides instant feedback about the changes in customer expectations. All this helps to make mid-course corrections to meet the current situations and customer expectations by the promised date.
Productivity is all about output upon input. Output is the value delivered to the customer. Input is the efforts we put in. “Simplicity” – the art of maximizing the amount of work not done – helps us to get more done with less. “Satisfy customer” helps us to be always focused on the customer value. Similarly focus on “Working software” helps to avoid unnecessary activities that are not adding any value.
Innovation is required in two areas; technical and practices. Continuous attention to “Technical excellence” and good design fosters the culture of Innovation. “Motivated individuals” can quickly identify opportunities for practice innovation and when they “Work together” it leads to useable results.
Elegance is an elusive concept which can only be experienced. It has been variously described as neatness, precision, simplicity, tasteful design, dignified gracefulness, and restrained beauty. It requires continuous attention to quality which is possible only with “Sustained development”. To experience the elegance requires full use of multiple senses, which is enhanced with “Face to face” interactions. It is difficult to achieve elegance with command & control culture. It requires “Self-organization” to bring out the best from each team member.
Though we have looked at each element of CPIE separately for better understanding, they need to be approached holistically. Together they produce the kind of synergy that is impossible if attention is divided.
To summarize, understanding the agile principles and related practices is important to achieve CPIE. But even more important is an agile mind to understand and use the right principles / practices as may be required for a given situation.
Have you ever been on a group tour? Everything is planned to the last detail, and executed meticulously. Totally expected, no surprises.
I hesitate to go on such tours. I prefer family tours, where there is a lot more flexibility. You can choose when to get up, where to eat. May be cancel some planned activity and just relax walking around the neighbourhood.
All of us have an internal compass. It takes us in the right direction, at least metaphorically, if not literally. More we trust our compass, more will be the surprises.
As a child we all have gone on a treasure hunt, some of us still do. There is no map to take us to the treasure. Just a few pointers. Even those we don’t have all when we start. When we reach one milestone, we get the next clue. It has lot more adventure and fun.
Isn’t agile software development similar? We go to the end of one sprint, have the satisfaction of achieving “Done”, show to others what we have done and get feedback; our clues to the next milestone. We have our product vision and a broad roadmap. That is our compass. It will keep us in the right direction.
When we get comfortable with our internal compass and trust forces beyond us, an external compass called destiny comes into play. Instead of remaining passive puppets in the hands of destiny, we actively embrace it.
Columbus trusted his internal compass and took to uncharted seas to reach India. The destiny had other plans. Fortunately, Columbus did not have Google maps and the GPS. Otherwise US would have remained undiscovered to the rest of the world.