Revisiting the Cathedral: Musings on Software Engineering
After hearing all the fuss about the Microsoft MSDN Blog entry about the reason why MS Windows Vista has slipped its delivery date here, I got into some software engineer (SE) thinking. By the way, the article was pulled so here is a full copy in Slashdot. The article is called the “Broken Windows Theory” and appropriately can apply to any large centrally managed and coordinated software development project. It is pretty obvious for anybody that the larger the codebase becomes the potentially more unmanageable the state it ends up in. I have done some preliminary thinking (time between reading this article early in the morning and taking a shower which is about 10 minutes only) and have come up with some thoughts I need to thresh out.
Here are some musings.
- The strict CMM lifecycles make software development more stable because it prioritizes thoroughness at the expense of speed. Therefore, I believe this works for the general software development sweatshop as it potentially stabilizes productivity and make results a bit more predictable. However, for bleeding edge pioneering work a more chaotic development structure is necessary. We also know that innovation comes in two (2) forms: incremental and revolutionary. Incremental innovation depends on structured development and is typically very slow. Revolutionary is chaotic but potentially very fast. The CMM lifecycles work for incremental innovation, but, not revolutionary ones. Creativity is basically aa chaotic process.
- Common SE knowledge points out that really large code is easier to manage if broken down into modules. Modules are easier to manage as they are smaller (and more manageable) pieces of code assigned to different teams. Each of these teams can then focus on one module. This clear division of labor makes more focused use of time and resources. However, as the “mytical man month” concludes “more people do not equate to more productivity”. Beware the asymptote.
- Unix SE knowledge implements this by pointing out that modules have to have well defined public portable interfaces, only one task (and does this one task really well) and expected behavior, and must fail noisly when in trouble. Sometimes, modules are so interdependent on each other than a change in one immediately affects the other. In this case modularization was not sufficient enough to …. modulize … the code? Some times well designed layers of interfaces are necessary. Layers might cause performance problems but it also allows different people to focus. Groups of related modules can then share interfaces to other groups of related modules. In the Linux world, the GUI management components are separate from the Linux kernel. This way we can change the GUI stuff and it won’t break the kernel. Once in a while, it causes something funny to happen. But, on the average case, they mind their own business. However, this causes a layer between the GUI and kernel to be created. This can potentially translate to inefficiencies and delays.
- Modules must be small enough to be completely re-engineered (or totally re-written) at any time. If modules are made this small it is possible to re-engineer the entire thing to make them more efficient, add a new feature or re-design. Code refresh is good for some projects. Also if this is put into mind when developing code, modularization becomes more important. At least, refreshing a small piece of code is not too difficult compared to the whole project. However, we cannot make their modules to small as there would be too many modules too.
- Java SE allows us to put Unix SE into practice by providing an OO system to better module management, an exception system for better problem detection and resolution. If it breaks make a lot of noise. Don’t let it slip. Poka-yoke! The Java folks have basically made it mandatory to modularize and report problems. A lot of people like Java because re-use of Java code is easy. In the open source world, C/C++ libraries come in development packages with include and library files. Java programs come in nice programmable Jars. So, if one is programming with another language these things just have to be done manually.
- Micro-management is not always good. A veteran’s valuable experience always helps. Helpful suggestions are always welcome. Insightful thoughts always help. However, looking down at the nitty gritty details of the developers work might not be very helpful. This is especially true with new things. It might just be best to admit that there are people who know it better than you. Let them do their thing.
- Be open to your development teams and watch out for slippage. It is true that most developers are very conservative when it comes to time tables. Therefore, a software engineer’s estimates at times are not realistic. Managers then find it necessary to push the developer with a tight deadline. Tight deadlines force them to move quickly. But, we must watch for signs of slippage. These signs are sometimes very obvious but we too don’t want to see them and confirm that a slippage might occur. Don’t deny it. When the tall tale signs are clear start making adjustments. Re-evaluate deliverables every so often. Watch the major milestones. If one of these slips the entire project will definitely slip.
These are just some randoms thoughts with no tangible solutions in mind. The science or art of software development is still inexact. A lot of lessons still need to be learned. The most important thing to do is not shrug the problems away or ignore it. The problem won’t disappear by itself. Look at it and face it head on. The best solution is to adapt to the scenario and make the best out of it. Evolution on the Fly.
Let me end with some food for thought from the same article:
Give me a cathedral, give me a bazaar - really, either would be great. Just not this middle world in which some decisions are made freely while others are made by edict, with no apparent logic separating each from the other but the seeming curiosity of someone in charge.
