John Cook is an applied mathematician working in Houston, Texas. His career has been a blend of research, software development, consulting, and management. John is a DZone MVB and is not an employee of DZone and has posted 168 posts at DZone. You can read more from them at their website. View Full User Profile

100x better approach to software?

  • submit to reddit

Alan Kay speculates in this talk that 99% or even 99.9% of the effort that goes into creating a large software system is not productive. Even if the ratio of overhead and redundancy to productive code is not as high as 99 to 1, it must be pretty high.

Note that we’re not talking here about individual programmer productivity. Discussions of 10x or 100x programmers usually assume that these are folks who can use basically the same approach and same tools to get much more work done. Here we’re thinking about better approaches more than better workers.

Say you have a typical stem of 10,000,000 lines of code. How many lines of code would a system with the same desired features (not necessarily all the actual features) require?

  1. If the same team of developers had worked from the beginning with a better design?
  2. If the same team of developers could rewrite the system from scratch using what they learned from the original system?
  3. If a thoughtful group of developers could rewrite the system without time pressure?
  4. If a superhuman intelligence could rewrite the system, something approaching Kolmogorov complexity?

Where does the wasted effort in large systems go, and how much could be eliminated? Part of the effort goes into discovering requirements. Large systems never start with a complete and immutable specification. That’s addressed in the first two questions.

I believe Alan Kay is interested in the third question: How much effort is wasted by brute force mediocrity? My impression from watching his talk is that he believes this is the biggest issue, not evolving requirements. He said elsewhere

Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves.

There’s a rule that says bureaucracies follow a 3/2 law: to double productivity, you need three times as many employees. If a single developer could produce 10,000 lines of code in some period of time, you’d need to double that 10 times to get to 10,000,000 lines of code. That would require tripling your workforce 10 times, resulting in over 57,000 programmers. That sounds reasonable, maybe even a little optimistic.

Is that just the way things must be, that geniuses are in short supply and ordinary intelligence doesn’t scale efficiently? How much better could we do with available talent if took a better approach to developing software?

Published at DZone with permission of John Cook, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)



Chuck Dillon replied on Thu, 2012/04/26 - 1:07pm

I'm pretty sure that a structural engineer would say that the ancient pyramids have incredible structural integrity.  I guess we need to know the requirements he had in mind when he made that statement

In terms of approaching what you and Kay are describing from a "productivity" perspective, it seems like you will end up more or less where the agile folks are.   Agile is based on the notion that you have to discover the real requirements as you go and as you customers figure them out.  IOW, they embrace "I'll know it when I see it" as unavoidable and that virtually all software is BETA in search of "seeing it" and so optimize for iteration of functional solutions.

It seems to me that there's far more theoretical efficiency gain if he could find a way to revolutionize requirements analysis so we get to "seeing it" in routinely significantly fewer iterations.

Paul Russel replied on Sun, 2012/06/10 - 9:23am

We still don’t know how to program (paraphrasing Sussman).

Our languages, libraries, and systems fail badly at composition, we always end up writing monolithic programs with huge baggage. If we pick a simple problem, e.g. a contact web crud with acid semantics, most of it is unrelated to the particular problem because we end up using a web server that implements much more than we require (e.g. we don’t need multipart for our crud), ditto for the database system (e.g. no need to support joins for a single table system). The libraries to interface with these systems expose a very large API, much of which we don’t require and often require configuration for these unnecessary parts. The languages don’t offer any way out.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.