We will continually refine the design of the system, starting from a very simple beginning. We will remove any flexibility that doesn’t prove useful.
One of the many prophecies of Agile Development Process and in particular XP. I have been trudging the trail of spick-and-span product development for last 1year at least. Although org guidelines fixate a firm waterfall model of development, I have been reading quite a lot on agile processes. Every now and then I tend to cast some aspersions on agile way of development. One thing is it doesn’t scale well on big teams and big organizations. A case in point is development of a module in ERP system. Suppose I am doing Order Management (OM) module; I know upfront that OM depends on Shipping, Pricing, Invoice modules to pick a few. So I need to clean up my interface area with these modules upfront and declare them. My data model for interface with them has to be fixed. I should anticipate the future requirements and design the interface. For a change later in cycle can be really costly and moreover the cost has to be borne by the impacted team and not me.
One of the things which is not always true is that cost of change may not rise dramatically over time as XP suggests. In practical scenarios a cost of change can be significant.
XP makes us believe that cost of change can be flattened if we follow a good test-driven programming. So as per XP daily builds and running automated tests is just enough to absorb this cost of changes. But alas! the reality is much more stark in contrast. After all software development is not just building automated unit test and running it daily to prove code is unbreakable. The underlying design should be flexible enough to accept the changes with minimal impact. Yes that is what XP also says, I know that. We in software world are in constant race against time. So suppose we just design things keeping in mind minimal things which just solves current requirement in hand and a week before deadline of delivery a requirement pokes up which is sort of destabilizing the underlying data model, what can we do? How many managers will have the gumption to take up an overhaul of data model late in the cycle?
I tend to take the argument “We will take away any flexibility that doesn’t prove useful” with a pinch of salt always. If the flexibility is like providing many-to-many relation between two tables compared to one-to-one relation existing today, I accept it. I know the cost of changing from one-to-one to many-to-many is manageable. Suppose I am coding a reporting tool on top of a warehouse, for my graphs I won’t provide all flexibilities like formatting X-Axis, Y-Axis, data points (with different colors, marker shapes etc) if the requirement is not there today. I know the uptake of this late in cycle is not going to destabilize the existing code. If I anticipate a requirement which is going to destabilize my whole code, I believe I should have thought about it at the beginning even though the requirement was not there at that point of time. A late cycle infusion of change can be jittery for the developers, for the management and taking undue pressure on all the parties concerned. As in our case if I am doing Business Intelligence software on top of a warehouse; I guess the design of the warehouse (called dimension modeling) should be done upfront considering all the requirements that are there today or that can be there tomorrow. If we leave holes of flexibility in that which we tend to think can be accommodated tomorrow its sure recipe of disaster. For a late cycle change like adding a new dimension to the warehouse can be impacting the guys who write code to build the warehouse (ETL), the guys who do reporting on top of the warehouse (analytics) and it can be too complicated to incorporate it.
XP can work very well, if I was doing a Library Management System for my College Library. The domain area is quite clear, boundaries are drawn and I don’t expect many dependencies with other systems. XP sulks in a product development environment where you are trying to connect disparate systems and frameworks and are trying to build a solution which you think customer is going to derive value. In a generic product development environment I guess it’s hard to get a customer to sit by you and validate each and every step you take. Well I know this is not a problem of XP, and it also means XP is not the panacea for all software development woes.
Another interesting aspect in XP is Collective Ownership of code. Yes its really worthy if we don’t build silos of specialists and instead carve out jacks of all areas. This is very much practical in the Library Management project in College. In practical cases, each module requires some core strength from the designer or architect involved. I don’t mind if we are just shuffling developers around modules and make them code. But I do believe there should be a owner/architect/designer for a module in a project, who know the historic decisions that have been taken and more importantly why they were taken and shape up the module in elegant way during the due course of time. If we are going to shuffle owners every month, it’s a recipe of acrimony. For in big enterprise systems it’s impossible for a single guy to grasp the complexities and nuances of each and every area make clear decisions. A case in point the warehousing and analytics solution, the ETL itself is so complicated that a guy who has worked in 3 months in analytics module may not find it a cake walk to suddenly switch over one day and chip into ETL and incorporate design changes into it. So the key is having an owner, a pillar for each module who decides how to incorporate changes without setting in entropy. And obviously programmers can be made to code as per specs in different modules. Here again its practically impossible for the person who is coding to take design decisions on the fly when the module he is coding involves 900 classes, each class spanning 100-500 lines. I would expect one guy who has fair deal of idea and handle to these classes and what it does before setting upon a design change.
Having said all these let me not be gushy about the agile way. Automated testing, Test before code, pair programming, adeptness to changes, 40hr work week are all too tempting and sinews of agile way to be hushed away in the background.
So I think it all boils down to the situation in which you are in and obviously it’s a balancing act and judicious mix of agile and waterfall methods which can sweeten your time during the travails of development.