This is Part 3 of the series of articles "Why go Agile?"

In essence, RUP and the waterfall approach, even when done under judicious process discipline, still attempted to be predictive software development methodologies. That is to say, they wanted to predict in advance, before beginning the project, matters such as: what the cost of the software development would be, exactly what functionality the software would have, how "X" or "Y" functionality would be expressed on the screen, etc.

Despite the academic efforts to predict software engineering, in our opinion, the project has not met with success. The Standish Group, a renowned technology consultancy firm based in the US, publishes the Chaos Report, a compilation of interviews of several thousand project leads that compiles their opinion on the success / failure of their software development initiatives. Projects are polled all over the developed world, from the US to Germany to Japan, as well as in technology intensive nations such as India and nearshore / offshore software development Latin American countries such as Colombia.

Recent Chaos Report statistics found that only 31% of software development projects are reasonably on time and on budget, while on average projects cost 189% of what was originally estimated. Despite the academic efforts of methodologies such as Function Point estimations, Cocomo I/II, Feature Points, SLIM, etc. predictions on the cost of a software development project are several orders of magnitude wrong.

A wrong prediction contradicts expectations and causes dissatisfaction. But the real problem is that predictive software development methodologies not only do not work, but add insult to injury by actually making a project worse off and not better off. Why so?:

• Big Requirements Up Front written documentation is expensive, and slows down project progress: The formality involved in documenting, in advance, everything that is going to be developed in a great level of written detail, consumes thousands of hours of very smart and expensive resources, both at the client’s side as well as the developer’s side. Notice written communication is one of the most inefficient ways human beings have of communicating (it is slow, it takes forever to make it clear, and it is difficult to “ping-pong” a conversation in writing). So software development projects tends to slog forward in meetings and debriefings about the status of the documents. Furthermore, documentation doesn’t make software; coding makes software. Often during “traditionally run” developments, the team presents management with tons of documentation, lots of expenses, and no code to make up for the investment. This is highly demoralizing and risky. In the final analysis, at this point in the software development project –and several months could well have elapsed just writing documents—the software may be a complete flop and nobody yet knows it.
   
• Traditional approaches to software development are administratively heavy: In software development projects based on BRUF and highly specific written documentation, processes are rigid and highly controlled. This requires a whole level of PM’s dedicated to managing the paperwork, the formal requests for changes in requirements, the formal estimations, etc. Again, money is being spent in administrative duties rather than in code.
   
• BRUF creates bloated monsters: Predictive software development methodologies require that the user ideally foresee everything that a software will need before the software begins to be developed. Typically in such projects a fixed price budget is approved up-front for the construction of the project (i.e. the budget that corresponds to the effort estimation). The client’s lead users are very conscious of how difficult it is for a budget to be changed once it is approved. Hence, they err on the side of caution and include everything they think they need and they might need in their requirements for the software. Bloating is so prevalent and so severe in BRUF projects that the Standish Group’s statistics point to 45% of a software functionality being seldom or never used once the software goes live. Imagine that, if they process did not create such perverse incentives, the software development would have cost 45% less at no real, functional expense to the business!
  
  
• We want to close by being fair. The negative impact of traditional software development methodologies such as RUP can be mitigated by working iteratively. Also, when deployed correctly, they produce software with very few bugs. But they tend to produce the software the client asked for, but not the software the client needed!

This is Part 2 of the series of articles "Why go Agile?"

Well into the 1980s the largest buyer of software development services in the world, the US Department of Defense, was having trouble getting projects done on time, on budget and with the right specifications. Despite working with some of the best and most renowned software development companies out there, it still had close to a 50 / 50 chance of getting a project right. Worried about its performance, it set up to fund, together with Carnegie Mellon University, the Software Engineering Institute. This institute was tasked with compiling a set of software development best practices that could provide both a benchmark aginst which to measure current vendors, as well as concrete guidelines current vendors should follow to improve their software engineering practices. The initiative ultimately gave birth to the Capability Maturity Model Integration (CMMI) and its levels of 1 to 5.

The Capability Maturity Model ranks companies into levels, going from those that are at Level 1 (operating under processes that are not managed, or ad-hoc) to those that reach Level 5 (optimizing). It is not our purpose here to go into detail about what each level means, except to point out that at Level 5 companies must have complied with following the tenets of more than 16 software and systems development process areas, including the ability to (at level 2) manage their software development process (configuration management, project monitoring and control, project planning, requirements management); define their approach to software development (decision analysis and resolution, organizational training, risk management, validation, verification); quantitatively manage their development process (measure organizational process performance, reliably measure productivity, error injection and other key variables) and optimize their process, which involves constantly reviewing the causes of process problems and taking measures to improve and resolve them.

The nature of the DoD’s needs influenced CMMi at the core. The DoD’s software development projects tend to be large and often interact with hardware. Furthermore, they are government funded, which requires measures of “financial transparency” that limit budget flexibility, often requiring projects to provide a fixed cost. Thus, it is not surprising to see a CMMi model that was slanted towards traditional RUP (the Rational Unified Process) methodologies, especially in what regards to:

1. Big Requirements Up Front (BRUF). BRUF is necessary to adopt so-called formal estimation techniques (such as COCOMO I/II, Function Points, Feature Points, etc.) that allow a software development team to have a shot at estimating, in advance, the effort required –and hence the cost—of a project;
    
2. High formality and preciseness in the way requirements are elicited and documented. Under strict RUP or even under a more archaic “waterfall approach”, requirement engineering is a ceremonious process that documents, in writing, use cases (i.e. using UML diagrams, for example), functional and non-functional requirements, process flows, mock-up screens, etc. 
   
   It is not surprising that under RUP (or also under a traditional waterfall approach), the inception and elaboration stages take 40 to 50% of the total time spent in the software development project.
    
3. Testing “after the fact”, i.e. after code has been written. This is clearly a quality control, but often leads to detecting errors late, when correcting them is more expensive.
    
4. Strict controls to “requirement changes”. Processes for changing requirements are often so heavy and difficult, they seem to be designed more to prevent change than to manage change within a software development.

In essence, the software development methodologies proclaimed to be “best-practices” under CMMi and RUP, did produce significant gains in the quality of the code. However, this came at a significant price both in terms of:

a) Time (time was more predictable but the formality significantly increased the length of the software development vs. ad-hoc programming);

b) Functional relevance (a rigid process did not allow applications to easily adapt to changes in the business environment, so the ensuing application often was “what the client asked for but not really what the client needed”;

c) Frustration with cost (the ensuing application was not cheap to make, due to the administrative overhead of all the formality of the process, and the predicted cost of the application still did not match the initial “fixed price” estimate –yes, estimates where off by a lesser amount vs. ad-hoc estimations, but they were still significantly off.

This new series of posts tries to explore a question that clients often ask us… Why develop their software using Agile methodologies?

Many clients, even many software development practitioners, have heard about SCRUM or XP. But, as with many trends, it is difficult for them to tease how much of Agile software development is a fad, and how much is truly useful to develop more relevant software in less time and with less money. The following blog posts touch upon the experience of more than 6 years our company has had moving from the more traditional RUP world, to the Agile software development world in dozens of projects large and small, ranging from IT offshore initiatives to large IT outsourcing service deployments for our clients.

To understand Agile software development it is first necessary to understand the context amongst which it surged, so we will remount to a short history of software engineering as a practice, or science. Software development is a young science in comparison to its engineering peers. Take civil engineering, for example. In practice, humanity has engaged in civil engineering for thousands of years. Academically –i.e. in a formal classroom setting—it has been taught for over 300 years. Although advances in civil engineering still occur as structures become more creative, or larger, or ecologically friendly, it could be said civil engineering works, is reliable and is a mature science.

Nothing close to this occurs with software engineering, which came into existence with the appearance of the first computers in the 1940s. But computers were mostly the realm of defense departments of some highly sophisticated nations plus a cadre of select universities. Few people where exposed to them initially, so the science developed rather slowly at first. With time, of course, computers and software have become ubiquitous. However, software engineering still remains a young discipline, as is still far from reaching a stage where one could say that the way it is taught and practiced is standardized.

From our own experience as a software nearshoring firm, the way each software company manages complex developments is different from the next. Yes, a language, say Java or C# (.NET) is always the same and has a basic syntax. In this sense, everybody codes in the same manner. However, the complexities of software development are not only about coding –they involve the way a group of engineers interrelate to one another and to the client, coordinate each other, estimate their work, execute their tasks in an orchestrated fashion, and finally deliver a finished product. This set of blog posts will focus on the process of software development –or the software development methodology—rather than on technical aspects of engineering.

We believe a project’s method and coordination is what truly makes it successful, or sinks it. Many are the software development groups that can get the architecture right, that manage to recruit great coders, yet fail miserable when it comes to deploying a software development project that is delivered with a high quality standard, on time, with the right functionality, that is easy to maintain –all this done under a reasonable cost (continued in our next post...)

IT offshoring, whether it be to a distant continent in a completely different time-zone, or a nearby nation that goes to bed when you do, ultimately seeks gains in efficiency and costs.  Think about it?  If you could achieve developer rates as competitive as those of Asia by outsourcing your IT in your home city, would you send your work abroad? Probably not.  Gains in cost are so dramatic that IT outsourcing offshore is a reality that has come to stay, which begs the question: “with so many destinations to choose from, where should I go?” 

Countless studies still have India claiming the top-of-mind IT outsourcing spot.  After all, India’s solid technology education and relative availability of English-speaking engineers, coupled with very low costs, allowed the country to give birth to the IT outsourcing revolution in the end of the 1990´s and the early 2000’s.  However, many factors have changed since IT outsourcing’s early days, with alternate destinations becoming more relevant for many types of clients.  To begin with, demand for IT outsourcing to India has overheated the country’s market for talent, to the point that it is typical for clients to experience team churn-rates of 20 to 30% a year.  Furthermore, competitive pressure coupled with strong economic development has made India’s IT outsourcing rates increase dramatically over the last five years, reaching parity levels with those of Latin America as well as some Eastern European countries. 

The above factors, together with the rise of high-quality (i.e. CMMi 5) IT services vendors in other destinations, have increased the relative weight of other variables when it comes the time to make an IT outsourcing decision.  One aspect of critical importance in choosing a destination country, in our view, is whether one chooses to nearshore or offshore.  By IT nearshoring, we mean choosing to outsource one’s IT services to a location that shares a similar timezone.  For a US-based player, then, IT nearshoring might mean sending development work to Canada or Latin America.  For a European player, IT nearshoring might mean sending work to the Ukraine.  (Because PSL serves firms mostly in North America, we will speak from our perspective as a Latin American IT services provider).

To understand what is at stake, lets review what undergirds a traditional IT outsourcing process: Born out of the practices that arose in the fulfillment of large contracts (NASA, US DoD, etc), traditional software development tends to rely heavily on formal documentation and the production of Big Requirements Up Front (BRUF).  Predictive estimation methodologies such as RUP rely heavily on this approach.  Under predictive estimation, before software is coded, it is first fully documented via functional and non-functional requirements.  Working within such a software development model, when offshoring IT to a different time-zone, the software vendor would : 1) Send engineers to the client’ site who would compile all requirements; 2) Return to the offshore site to code the software development's requirements; 3) Deliver the code to the client (in medium to large projects, this cycle would be iterated several times). 

In the past, because software development requirements tended to be heavily specified (i.e. written down), it was easier for developers in distant time-zones to know what exactly they needed to do next, despite having little real-time access to the client, who was asleep when they were awake.

Today, traditional models for software development are highly questioned in their effectiveness.  Studies such as those produced by the Standish Group (The Chaos Report) show that a significant percentage (sometimes more than 45%) of projects that are attempted under BRUF deliver highly disappointing results, both in terms of cost, time to market and functionality.  Most companies now approach software development under agile methodologies, which seek to be adaptive rather than predictive (see more on the advantages of agile methodologies on our next post). 

Agile software development starts from the premise that software is an “idea constantly taking form”.  Therefore, changes will always ensue during the development process.  Instead of preventing change or trying to “outguess change from the get-go”, agile embraces smart change, and produces a process that incorporates improvements and modifications in an organized and risk-controlled manner.  In general, change can be brought about by different factors, including: 1) the market changes so the software has to change; 2) changes in the competitor’s business offerings require changes in the software; 3) new ideas arise in the user’s mind that justify changing the software; 4) changes in undergirding technology make new business models possible, therefore justifying changing the software. 

To manage change and promote adaptability, agile software development seeks to replace formal documentation about requirements–which takes a lot of time to write—for constant conversations about requirements.  Therefore, in agile SCRUM, say, the developers constantly interact with the client’s user groups to understand that nitty gritty of a functionality that was described to them at a high-level, rather than demanding air-tight documentation exist to describe the functionality in high detail.  Because conversations are not only faster, but provide for a give and take of ideas and concepts, the software that results in not only written in record time, but is also much richer and smarter in its functionality.  Serious empirical compilation show how Agile software development can produce the same code with at least 30% less cost, and in at least 30 to 40% less time than traditional approaches.

To undergo an effective Agile software development process, however, it is important that the developer and the client constantly interact, in real time.  Interaction does not imply a face to face presence; it can occur via chat, the telephone or video-conferencing.  But because agile groups do not have large backlogs of detailed information, if they are unable to reach the client at a critical juncture, the group might have to stop the software development temporarily.  For this reason, Agile software development demands that the client and vendor be in a similar time zone, so that questions are asked and responded in as close to real time as it is feasible.  Overlaps for an important part of the day (at least 60 to 70% of working hours or more) seem to work best. 

US or Canadian software companies that have previously outsourced IT services to India, China or the Philippines, are now approaching Latin America as a preferred destination for IT outsourcing partnerships, especially after agile software development projects with such destinations have gone sour.  Indeed, under agile, other weaknesses of former IT outsourcing vendors come to light, such as the inability to think critically about unforeseen circumstances, the inability to ask clarifying questions with little information to start from, or the complications certain cultural mores create in allowing for creative thinking “on the fly”. 

If you are a US or Canada company that is thinking about going agile with your IT outsourcing, and do not want to lose the benefits of working offshore, we heavily recommend that you look for software development and IT service vendors in Latin America that overlap your timezone to an important extent.