Sudarshan Acharya

In 1974, within the context of Structured Programming, Donald Knuth said “Premature optimization is the root of all evil”.

All evil huh, Donald?

Eventually, with evolution “Don’t optimize early” or “Don’t optimize unless something is a problem” became one of the mantras of software development. In fact, it is one of the most obediently followed mantras in software industry. Probably the reason they follow it so closely is because the rule says “Don’t do it”. Don’t do anything, and you are following the principle. Don’t do anything (that you already don’t wanna do) and you are getting a pat on your back for doing great. How awesome! How easy! Don’t you wish all other principles in software development (and life in general) were as lenient as this one?

That cute little tip from the Guru decades ago has now been cleverly molded to a free-pass to build things of the barest minimum quality just enough to save your ass at the moment. As long as it seems to work somehow, its fine. In a way, its legal and ethical to do things the wrong (often quicker) way as long as its not a problem today. In the uber-speedy AGILE’s caricatures of today, it means “Just do it and push it into production”. In a slightly different scenario, it could mean “Just do it for now and  realize at the end of the week what you just did was a crap”. Whether you are releasing the insecure and bloated piece of unresponsive crap that nobody is proud of or fixing the crap to make it less crap depends on how shameless and ignorant you are. Unless a customer has found the problem, a problem is not a problem. So release it anyway coz its not a problem yet. Let it be a problem tomorrow.

Businesses, in general, seem to have a phobia for quality. Few companies make a quality product, and worry about scaling it to quantities. The majority get quantities of low-quality products, and then spend their life trying to uplift the curse. Some don’t have any clues, they just exist coz they were established and they have to keep moving.

Early analysis & optimization is a measure of your commitment towards quality. Ironically, when you hear of optimization, you think its is a developer’s problem - you think its that extra for-loop in the developers code. I agree, developers are not perfect. But neither is anyone nor is the whole process.

Of course, in a practically perfect team, the customers would know exactly what they want. And the analysts would know exactly whats feasible and appropriate. The architect would architect the perfect holistic view of the business requirements with the system requirements and the technologies. The designers would design the perfect interfaces and workflows. The developers would write the perfect code. And all this would be accomplished in a perfect environment provided by the manager. Oh well, do we still need testers coz we have already written the perfect implementation for the perfect set of requirements?

The reality is that the customer has no clue about what he wants. The analyst has no idea about whats feasible. The architect is the one who retired from programming and hasn’t learnt any new technologies in the last five years. The developers have no clue about anything other than the few shortcuts in their favorite IDEs. Oh ok. We need testers by now. After all, we all are humans and to err is human. So testers are the people that would come at the end and validate the work.

But it solves only a tiny piece of the puzzle. Quality assurance will verify that the codes and UI behaves as per the requirements, and the verification is based on the few sets of tests it runs. It doesn’t validate the requirements. It doesn’t validate the architecture. It doesn’t validate the code. It doesn’t validate the aesthetics or the usability. It doesn’t validate the infrastructure. If you can have bugs in the developers code, you can have bugs in the requirements too. You can have bugs in the architecture too. You can have bugs in the process and management itself. And the effect of defects in the rest of the process is usually a lot bigger than a that of the code and the coder.

You see the fundamental flaw in the process that I am talking about? Validation is done at the end of the whole cycle, and worse, it validates a small portion of the whole work. The rest goes unvalidated for the most part, and you try to hide the filth under the carpet for ever. But the ghost will appear from under the carpet.

What should happen is, after every phase of the software development life cycle, you should validate the output against your standards and optimize it right then if necessary. Don’t let the flaw bubble up the stack. You should optimize your requirement, you should optimize your design, you should optimize your architecture. Everything! If you are just postponing your problems, you are making them bigger and incurable. What would take 2 days to fix it during design will take a month to fix it after its developed.

Early optimization is a taboo word in software development industry. Its a shameful thing to do coz you are breaching the most popular software principle.

If you are a developer programming for the web, its is even more complicated than it appears. You develop the product on your laptop like a laboratory test. What runs on your box will now be shipped to the Internet to be used by people all over the world with different connections, with different clients and tools with different degrees of understanding. Its such a complicated thing that you do have to set your standards and priorities correct. If you don’t think about performance until its a problem, then you are doomed. If you don’t think about scalability, until its a problem, then you are doomed. Because if you got your basics wrong in design and analysis, the only way to fix and optimize it would be to redo the whole thing, which is usually not an option. You cannot optimize a crappy product and make it great again, all you can do is to make it less crappy.

Optimization isn’t about spending weeks in a bat-cave trying to find a problem in somebody else’s shitty code. Its more about doing things right from the beginning and minimizing the stupid things that could be avoided. It about setting up a realistic standard for yourself, and living upto it everyday. It about having a vision even before you do something, and doing what it takes to make it happen. Optimization is about fixing things that you know will be a problem in relevant future. Its about curing the cancer before you even develop it. Its about being optimistic that you will be in the business one year from now.

How many times did you rush a feature in a week, and then spent the next three weeks fixing the problems in production? How many times you made the wrong design decisions coz you thought Early Optimization is evil? How many times have you worked on the rewrites of a recent project? How many times did you try to achieve things like performance, scalability and security backwards, rather than incorporating it from ground up?

NOT Optimizing early is clearly a much bigger problem in software development than optimizing early. But even then, if you can get away with a bad product, why bother making a good product. Right?