Programming and Complexity
Today, we release software and hardware and use its customers as testers to uncover its flaws. In essence, we develop successive versions of a system to help us better understand what we should have done in the first place. And the bigger the system, the less we understand what we should do. Imagine if civil engineers built bridges or houses that way. Would you live in such a house? Of course, getting a new computer system to work well isn't like building yet another house, because each new project is just that---new. So the problems are often new.
Programming is about building castles in the air, but those castles have to work---they must have plumbing, lights, and garbage disposal. We can't really compare computer programming to house building because house builders don't have to worry about many, many things. They don't have to fear that gravity will be turned off one day a week, or that air pressure will suddenly halve, or that a tornado might suddenly form in the living room. They don't have to design a single house that works well for Bedouins, Tibetans, Eskimos, astronauts, and Jacques Cousteau. They don't have to ask whether their toilets will work in zero gravity, or whether their faucets will flow in the Antarctic, or whether there are any termites that eat ferroconcrete.
Physical law and terrestrial environments and normal timescales limit the problems house builders have to worry about. No such limits apply to software. We can ask software to do anything we can think of. So it can get really complex really quickly.
Further, when house builders finish a house, that's it. You don't call them up next month and ask them to turn it into an igloo. But that's exactly what happens with software. Giving people a computer system always results in unforeseen complications. And the bigger the system, the more the complications.
Armed with a new computer system, we start doing things we couldn't have done before simply because it was too much trouble. So demand for new uses booms. Which leads to new demands on the system beyond whatever we originally thought the system would be used for. In other words, the new system lets us see what could be done. So we react to what it does well or badly and demand changes. So often, solving a problem with a computer simply changes the problem.
City planners have to face the same problems of complexity. For example, the M25 roadway around London was supposed to reduce traffic in and around the city. Of course, since it did indeed shortcut London's crazed snarl of streets, everyone started using it. Consequently, traffic on the M25 slowed to a crawl. Soon, Londoners started calling it the world's largest carpark.
So the problem isn't really in software or in hardware, but in complexity itself. Our biggest problems have now grown so complex that we simply don't understand them anymore. So when customers first explain what they think they want, computer engineers can't be sure that that's really what they want. And after a complex piece of software exists, everybody has to work hard to adjust it to handle things that nobody understood well at first. Further, once the software exists, demands on it change. Its own existence changes the problems it's asked to solve.
Today, it's normal for big software projects to cost tens of millions of dollars, be months or years behind schedule, and be double or triple their estimated costs---or be cancelled entirely. Even when finished and working successfully, they still have errors. Some computer systems have grown so complex that attempts to patch them merely result in more flaws showing up elsewhere. So it's becoming standard practice simply to leave known flaws in place rather than trying to fix them. Thanks to the complexity barrier, large-scale computer programming has become as hard as politics.
Consequently, it's pointless to try to build one large, complex program to solve a hard problem once and for all. Instead, it's much better to build programs in small, replaceable, extensible pieces. Flexibility is now much more important than completeness.