Emergent Challenges

I recently read "Prey" by Michael Crichton which is an exciting, easy read that explores the surface of nano-tech programming gone wrong.

A key element of the story is that high level goals and behaviors are too sophisticated to program into such simple devices. A programmer devises individual behaviors that interact to result in a desired collective, overall, emergent behavior. This is also how insects work. No individual bee knows how to build a hive. It is the outcome of a collection of primitive individual behaviors.

Where it goes wrong is in how difficult it is to predict and control emergent behaviors.

This idea of emergent behavior exists in more complex systems. In fact, I imagine it exists in every system. This article is about how it exists in technology projects in a real way that can affect quality, time to market and success or failure.

Pointers are a powerful feature in C but so many problems emerge from programming using pointers that modern languages like Java and C# have completely abstracted pointers and object life cycles out of the language and moved them into a virtual machine with automatic garbage collection. A solution was too dependent on garbage collection to be solved with pointers. Solving the emergent pointer problems of C required a paradigm shift.

Sometimes emergent problems could be solved in the original domain but laziness and new features often drive new paradigms, e.g. Microsoft's shift from WinForms to WPF. Although it is a paradigm shift in some ways I don't think the advantages would be compelling to most small companies. What Microsoft doesn't tell you is that, while WPF may be the answer to some of the problems that emerged from WinForms, a whole new set of emerging problems will be caused by WPF. Here is one of the most sensible things I've seen written about WPF - http://stackoverflow.com/a/75387.

Every new technology will have new emergent challenges. Don't forget this simple fact when tempted to switch to a new technology. The real problem is that a new technology is new to everyone and the problems that are going to emerge aren't yet known. Also, the evangelists are biased. Don't expect them to caution you about realities. Think about how aggressively Microsoft evangelized COM. Where is it now?

Even large scale systems have emergent behaviors. For instance, the obsession with copyright in the western world has driven a lot of innovation into the far east. We arm patent owners with trivial or self-evident patents and powerful legal leverage that prevents derivative innovation. The emergent result is to empower countries where those patents are not enforceable to innovate and threaten our dominance in a technology.

Another example is the success of personal injury lawsuits in American society. Bell used to be a prominent motorcycle and bicycle helmet company. Because they were American they were the successful target of anyone in the western world that wanted to sue them after personal injury because they were wearing a Bell helmet. The result was to stifle the helmet industry in North America and create a vacuum which was filled in the Pacific Rim where personal injury lawsuits could not easily get traction. For years the motorcycle industry has been dominated by Arai, Shoie, HJC, etc.

Be wary of new paradigms. Even if the end result is compelling, the transition will almost certainly cost more than anticipated, take longer the estimated and present a new set of problems that will take further time before new practices are established and incorporated in your development environment.

My best advice is to be aware of new techonologies and to experiment with them so that you can judge the cost/benefit issues for your own situation. When you want to transition, do it in a series of phases that mitigates the cost, learning curve and risk to delivery dates for your next few releases.