Lessons Learned and Not Learned

A start up takes many expediencies. As it starts to grow and especially when it brings in new people in management, product management and operations roles there will be need for change. I have seen many companies review operations and tabulate lessons learned. In theory this is a good exercise but there are several pitfalls.

Some Constructive Ideas for High Level Corporate/Departmental Reviews

Lessons learned are often symptoms. This is one of the biggest pitfalls. Patching symptoms can be very tedious and expensive because they tend to pop up everywhere. Make sure that your lessons learned are the right lessons. Try to understand the root causes of perceived problems. Addressing one root cause will likely address many symptomatic problems. Avoid the pitfall of essential lessons NOT learned.

Be respectful to the people that got you to where you are today. They are the doers. Be careful when introspecting not to demoralize and demotivate the people that actually know how to get things done. Don't talk about the current state and past actions negatively. If you believe that your problem is competency you need to fix that before doing any kind of reviews or post mortems.

Consider that the lessons learned might be wishful thinking. Don't encourage wishful thinking and whining. Don't legitimize it. A good sign that your lessons learned are whining is that they are symptomatic of some underlying issue, like limited resources.

Don't get stuck in this introspective, critical mode. I've seen companies perpetually talking about lessons learned. For years. If you embark on self analysis, give it a specific time frame and make specific conclusions and actions and then stop.

Don't make a laundry list. If you have more than 5 lessons learned there is something wrong. You might notice many issues but filter that down to no more than a handful of lessons.

Don't undermine your strengths. What some people consider a weakness might be the result of a very important strength. Fixing a minor weakness might have serious side effects. Be careful not to lose the spark that got you where you are today.

Be wary of solutions. Every solution makes the environment more complicated. Every complication adds new problems. Especially be wary of adding controls and processes. They are often more difficult to do well than they appear.

Be wary of silver bullets. Real world problems rarely have trivial solutions.

Be balanced. Post mortems are often top down - managers, department heads. Nearly all of these people have the same perspective. Often your engineers know the root causes and best practices to address the real problems that you face. If your lessons learned are all processes, controls, etc. then there is a good chance that your analysis is one sided.

Seek Closure

Closure is surprisingly difficult to achieve. Every software project has many details that need to be implemented, tested and then rolled out. Meanwhile, market pressures are driving the product in new directions with bugs and new features.

It's not just software. When I do a home  project, I tend to procrastinate and waffle on the last 10%. Psychologically, that last 10% is often the hardest part. I have a bathroom that is still missing a piece of trim from a reno I did 3 years ago.

We also often hear about emotional closure. After breakup we need to move on with our lives. There are always loose ends and baggage to deal with. Without closure we aren't moving forward.

A new product or feature has goals which will never be 100% met. There are always loose ends. The reality is that we can't be perfectionists There will be details that can't be cleared up in a timely fashion. It takes a conscious effort to tie them off. Otherwise the project enters a form of limbo - some stuff is not quite finished but at the same time isn't important enough to get finished.

Closure requires a conscious effort, discipline and determination. It doesn't happen organically or accidentally.

Principles

We need to be able to make decisions and stick with them. It isn't easy. It takes a few basic principles, a concerted effort and a pragmatic attitude.

The first principle is to

Bundle activities into milestones and manage the milestone, not the individual issues.

Seek closure for the milestone, rather than continually trying to get a grip on a chaotic and changing set of unqualified individual issues. A concrete milestone gives you a means to focus.

The second principle is to

Once the scope is agreed, don't let anything else into the milestone.

Creep is a basic force of the universe. Individual tasks tend to grow in scope. New features will be wanted. Creep is the enemy.

The third principle is to

Focus people on the milestone. Closure of the milestone depends 100% on closure of the individual issues. Lack of focus and competing priorities are another enemy.

That trim in my bathroom would probably never be finished if my wife didn't nag me. It's basic human nature. This is where a project manager can be very useful.

As always, there are going to be exceptions but you need discipline and to be prepared to take a hard line on these three basic principles at times.

The Basic Process

The process is preciously simple.

Stage 1. Planning

• Consider your goal and determine the contents of the milestone.
• Estimate a reasonable completion date, or requirement for a market opportunity.
• Are you being greedy? Try to trim the milestone.

Stage 2: Execution

• Resolve issues in priority order. The temptation may be to knock down some of the easy issues first. Don't do that! Avoid back end loading your milestone with all of the hardest issues.
• Verify the issues as early as possible. Avoid back end loading your milestone with all of the hardest testing activities.
• Manage creep. Don't just watch it happen.

Stage 3: Closure (the Hard Part - that last 10%)

• Nag. Make sure that people are working on closure.
• Be practical. The easiest way to get closure is to avoid creep and move outstanding issues to the next milestone.

An Illustration

You are in the late stages of development for a software release but there are still some tasks to complete and a few weeks of calendar time left. It's been some time since the last software release and there's pressure from a customer. The temptation is real and you add a few more tasks. Bob isn't on the critical path and he can do the work without impacting the schedule.

During the next two weeks you find some bugs during verification that need to be fixed. Bob is sick for a few days. Some new market information suggests that one of the new features in the release might need to be changed. Now it seems that it will be at least another week.

During that week you get some closure on the original scope but Bob learns that his estimate was incorrect. In addition to the sick time he will probably need another week.

During that week there is an additional push from marketing to change feature X they were so hot about. Bob gets his feature done but no one thought about how to verify it. The verification team does some ad hoc tests but engineering management is concerned that more testing is needed. There is a meeting to discuss dropping the new feature but the feeling is that you are very close and it will make a customer happy. The verification team is asked to do some more comprehensive testing.

After the meeting sales calls down to say that a hot new prospect that is evaluating the product will make a big order if you can do one little thing. Joe is available and can do it in a couple days which won't affect the schedule. The verification team reports that they have found what may be a serious problem in one of the original, key features of the release.

...

You get the idea. Each change at this stage risks the ball game. The more time that goes on, the more opportunity and pressure to make more changes. If this becomes a pattern, people don't take schedules seriously and focus becomes increasingly difficult.

Making the Hard Part Easier

How do you do this? Here are some specific techniques that anyone can use successfully, regardless of the size of your company.

Defend. Someone will always be prepared to argue for some new feature. Creep always has proponents. There needs to be someone that is prepared to argue for the feature to not be included. An effective solution is to require at least three key people to agree to a scope change.

Prioritize. Increase the priority of closure the closer you get to milestone completion.

Be pragmatic as you approach milestone completion. If you have a difficult problem that is holding up a milestone, consider pushing it out or splitting it up and pushing out part of it.  Review all remaining tasks and their importance. Streamline and focus the review. Make P1 or P2 bugs mandatory, P3 defer-able with discussion, P4 or P5 auto defer-able.

Be disciplined. If someone checks in a change that wasn't in scope, back it out, especially near milestone completion. It doesn't matter if it is a nice to have. Put it in the next milestone. Create a culture where people know better.

Branch ad hoc requirements. If you have a customer that needs something fast, do it on a branch for that customer only and merge it in the next milestone.

Be mature. If you find a new bug, resist the temptation to fix it right away. If it is a legacy issue don't even entertain a scope change - if customers already have a release that contains it and they haven't noticed yet then there's a good chance they won't notice this time either.

Use a tool to associate tasks with milestones. Use the 'fix version' or 'milestone' field. The unresolved tasks are the remaining development scope. Close issues after they are verified. The resolved tasks are the remaining verification scope. Status is readily available in real time - just query the open issues in the milestone.

Use a triage process. Only a triage team/individual should be able to assign an issue to a milestone. Make your triage team your first line of defense against creep - by default new bugs should not be triaged into the current release

There are always exceptions to the rule. You want to make creep decisions difficult enough that they are taken seriously. The proof is in the pudding - follow these principles and closure will become a regular event. It is a healthy, liberating pattern.