Use Commit Discipline to Limit Code Entropy

Source control software is a basic tool that we are all familiar with but it is often perceived most importantly as a management tool to associate work with bug/feature tickets and manage deployment to release branches.

It is also a very useful for keeping code clean which can be especially important if you have mission critical code areas.

It is very simple and not particularly time consuming: look at all the differences when committing code and make sure your commit is clean. If your code base is clean you want to keep it that way and to aid troubleshooting of new bugs you want to be able to trace the change history.

One common scenario is that a change has required experimentation and debug logic. Once you finalize an approach you may have remnants of experiments, debugging logic and fragmented logic. Review this code from the perspective of the finalized approach. Ensure it is clean. It is surprising how often I've avoided committing vestiges of the development process.

Keep the granularity of commits as small as possible. It is a great way to focus on the change in context. For example, as part of a new application feature you've added a new method to your Rectangle class. Reviewing and committing the Rectangle change separately creates a focus on the Rectangle class that will help to maintain the intent and overall design of the Rectangle class.

Avoid coupling refactoring and gratuitous changes that obscure differences with functional changes. Implement refactoring and cosmetic changes as a separate commit and note in the commit comment that there are no functional changes. A simple example is moving code around. As a class grows it can help to reorganize the code layout but most source control software sees these moves as differences. If there is an actual difference in the function you have moved it is likely you won't notice it.

It's also a good time to review the area around your change. Are there code comments that are affected by your change? Have you obviated or orphaned existing code or variables?

Keeping mission critical commits clean is a very simple way to keep your mission critical code and its history clean. The second law of thermodynamics applies to all systems, including a software system. All changes will tend to increase entropy (disorder) but we can minimize the disorder by the these simple practices.

You don't have to be as disciplined everywhere but I find that developing this simple discipline where it is most important becomes habit forming and helps improve the quality of all commits.

Seek Closure

Closure is surprisingly difficult to achieve. Every software project has many details that need to be implemented, tested and then rolled out. Meanwhile, market pressures are driving the product in new directions with bugs and new features.

It's not just software. When I do a home  project, I tend to procrastinate and waffle on the last 10%. Psychologically, that last 10% is often the hardest part. I have a bathroom that is still missing a piece of trim from a reno I did 3 years ago.

We also often hear about emotional closure. After breakup we need to move on with our lives. There are always loose ends and baggage to deal with. Without closure we aren't moving forward.

A new product or feature has goals which will never be 100% met. There are always loose ends. The reality is that we can't be perfectionists There will be details that can't be cleared up in a timely fashion. It takes a conscious effort to tie them off. Otherwise the project enters a form of limbo - some stuff is not quite finished but at the same time isn't important enough to get finished.

Closure requires a conscious effort, discipline and determination. It doesn't happen organically or accidentally.

Principles

We need to be able to make decisions and stick with them. It isn't easy. It takes a few basic principles, a concerted effort and a pragmatic attitude.

The first principle is to

Bundle activities into milestones and manage the milestone, not the individual issues.

Seek closure for the milestone, rather than continually trying to get a grip on a chaotic and changing set of unqualified individual issues. A concrete milestone gives you a means to focus.

The second principle is to

Once the scope is agreed, don't let anything else into the milestone.

Creep is a basic force of the universe. Individual tasks tend to grow in scope. New features will be wanted. Creep is the enemy.

The third principle is to

Focus people on the milestone. Closure of the milestone depends 100% on closure of the individual issues. Lack of focus and competing priorities are another enemy.

That trim in my bathroom would probably never be finished if my wife didn't nag me. It's basic human nature. This is where a project manager can be very useful.

As always, there are going to be exceptions but you need discipline and to be prepared to take a hard line on these three basic principles at times.

The Basic Process

The process is preciously simple.

Stage 1. Planning

• Consider your goal and determine the contents of the milestone.
• Estimate a reasonable completion date, or requirement for a market opportunity.
• Are you being greedy? Try to trim the milestone.

Stage 2: Execution

• Resolve issues in priority order. The temptation may be to knock down some of the easy issues first. Don't do that! Avoid back end loading your milestone with all of the hardest issues.
• Verify the issues as early as possible. Avoid back end loading your milestone with all of the hardest testing activities.
• Manage creep. Don't just watch it happen.

Stage 3: Closure (the Hard Part - that last 10%)

• Nag. Make sure that people are working on closure.
• Be practical. The easiest way to get closure is to avoid creep and move outstanding issues to the next milestone.

An Illustration

You are in the late stages of development for a software release but there are still some tasks to complete and a few weeks of calendar time left. It's been some time since the last software release and there's pressure from a customer. The temptation is real and you add a few more tasks. Bob isn't on the critical path and he can do the work without impacting the schedule.

During the next two weeks you find some bugs during verification that need to be fixed. Bob is sick for a few days. Some new market information suggests that one of the new features in the release might need to be changed. Now it seems that it will be at least another week.

During that week you get some closure on the original scope but Bob learns that his estimate was incorrect. In addition to the sick time he will probably need another week.

During that week there is an additional push from marketing to change feature X they were so hot about. Bob gets his feature done but no one thought about how to verify it. The verification team does some ad hoc tests but engineering management is concerned that more testing is needed. There is a meeting to discuss dropping the new feature but the feeling is that you are very close and it will make a customer happy. The verification team is asked to do some more comprehensive testing.

After the meeting sales calls down to say that a hot new prospect that is evaluating the product will make a big order if you can do one little thing. Joe is available and can do it in a couple days which won't affect the schedule. The verification team reports that they have found what may be a serious problem in one of the original, key features of the release.

...

You get the idea. Each change at this stage risks the ball game. The more time that goes on, the more opportunity and pressure to make more changes. If this becomes a pattern, people don't take schedules seriously and focus becomes increasingly difficult.

Making the Hard Part Easier

How do you do this? Here are some specific techniques that anyone can use successfully, regardless of the size of your company.

Defend. Someone will always be prepared to argue for some new feature. Creep always has proponents. There needs to be someone that is prepared to argue for the feature to not be included. An effective solution is to require at least three key people to agree to a scope change.

Prioritize. Increase the priority of closure the closer you get to milestone completion.

Be pragmatic as you approach milestone completion. If you have a difficult problem that is holding up a milestone, consider pushing it out or splitting it up and pushing out part of it.  Review all remaining tasks and their importance. Streamline and focus the review. Make P1 or P2 bugs mandatory, P3 defer-able with discussion, P4 or P5 auto defer-able.

Be disciplined. If someone checks in a change that wasn't in scope, back it out, especially near milestone completion. It doesn't matter if it is a nice to have. Put it in the next milestone. Create a culture where people know better.

Branch ad hoc requirements. If you have a customer that needs something fast, do it on a branch for that customer only and merge it in the next milestone.

Be mature. If you find a new bug, resist the temptation to fix it right away. If it is a legacy issue don't even entertain a scope change - if customers already have a release that contains it and they haven't noticed yet then there's a good chance they won't notice this time either.

Use a tool to associate tasks with milestones. Use the 'fix version' or 'milestone' field. The unresolved tasks are the remaining development scope. Close issues after they are verified. The resolved tasks are the remaining verification scope. Status is readily available in real time - just query the open issues in the milestone.

Use a triage process. Only a triage team/individual should be able to assign an issue to a milestone. Make your triage team your first line of defense against creep - by default new bugs should not be triaged into the current release

There are always exceptions to the rule. You want to make creep decisions difficult enough that they are taken seriously. The proof is in the pudding - follow these principles and closure will become a regular event. It is a healthy, liberating pattern.

Quality Is Bottom Up, Not A Spice You Can Add

All firmware/software product engineering companies struggle with quality.

All companies. 

Quality costs money and takes time. When time to market is important, it is often more important than quality. Successful innovators must be able to go boldly into the market and make tough decisions to go fast and worry about quality, if necessary, later.

Lack of quality also costs money. Some customers will turn away from a purchase or return a purchase if quality is perceived poorly for the price. If a product is under warranty, returns can be very expensive, especially if it is a software/firmware problem that must be addressed.

Startups can be fast to market by being very time focused and accepting quality risk. A successful start up is often forced to mature quickly. Once it has products in the market place, quality problems can threaten their market position and strangle a small engineering department with maintenance.

How does a small company make the transition from time to market to quality over a product's lifetime?

Most don't.

Some keep blasting aggressively forward, damning the torpedoes, risking losing their customer base and being overwhelmed by maintenance. Some look to the industry for best practices, hire consultants, create processes that address symptoms but not the root cause, etc., at the risk of being overwhelmed by constraints and losing their innovative edge.

Where is the middle ground? How does a small company mitigate quality risk while avoiding being handcuffed by introspection, finger pointing and red tape.

The answer is shockingly simple. Quality has to be built in. It's not a feature that can be added in. It's not a spice. There is no magic quality dust that you can sprinkle on your product. Improving quality takes calendar time which in turn takes patience. Management often cannot get its head around this fact. They want a quick fix - "We need better quality now!".

The best advice I can give a growing small company is to be very wary of anyone that promises a silver bullet. There aren't any. Don't trust anyone that tries to sell you anything that sounds either fast or easy.

Quality is everywhere. In a software product, it is quite literally everywhere. Every line, function, library contributes to quality. When there is a quality problem, the root cause is also everywhere, every line, function and library. It is inside out, bottom up. 

The only way to raise quality is to patiently raise the quality of the entire code base. Priority can be focused in areas that are either more essential or problematic but the solution requires a patient, disciplined approach.

The process is also shockingly simple. Testing. While software development best practices can help mitigate quality risks for new or updated software, ultimately, it comes down to testing. All kinds of testing. Unit tests, functional tests, stress tests. Testing.

Small company management hates to hear about testing. "That would take a lot of time and effort. We need something now". Exactly! Denial is very costly. The longer a company hedges on testing the longer it will waste time and money and continue to risk credibility in the market.

Rather than believe that it is a bottom up maturation of the entire product code base and invest in a rational bootstrapping of testing infrastructure, companies often waste precious time with internal reviews, finger pointing and whining about their plight. Then they may hire consultants, install tools and processes, make changes in mid management and waste further time.

Denial is also why there is so much snake oil in the market. Consultants make a lot of money on denial. Tool vendors also share in the bounty. But, there's no money to be made in the hard truth that quality is a long term bottom up journey.

Companies that make a successful transition are the ones that get over the whining quickly and accept what has to be done. Only then can they look objectively at their product and start to prioritize a test driven quality agenda.

The problem is that the damn the torpedoes Type A personalities that were essential to the company's early success can have a very difficult time embracing an agenda requiring patience and ongoing investment.