How to Acquire Enterprise Systems?

Much has been written about the problems with the acquisition of large DoD systems. An accepted view is that the existing acquisition processes are “broken”.

There are, however, useful lessons to be learned from cases where complex systems have been delivered on time and under budget. Here are a few rules:

Acquisitions Succeed Because of Sound Engineering.
Compliance with elaborate policies, guidelines and instructions that dictate how systems are built and operated are unlikely to give assurance that a system will be delivered on time, on budget and with all of the features that the users requested. There is a long list of GAO reports that attest to the consistent failures of programs, each of which followed the thousands of documents required to comply with practices dictated by DoD Directives 5000.1 and 5000.2.

Acquisition programs do not fail because they do not comply with practices dictated by DoD policies. Non-compliance is a consequence, not a cause. Failures are attributable to insufficient engineering leadership, to a lack of engineering expertise and the absence of good engineering practices.

The primary challenges of program failure are technical and the ways programs are organized. The primary skill of Program Executive Officers (PEOs) should not be the skill how to steer an acquisition through a maze maze of regulations and procedural restrictions.  The PEOs job is to make technology work and delivered with acceptable risks.

Program cost accounting and progress schedules are only indicators of how well the engineering and the organization of work is working. It takes more to measure of a PEOs success than what is accomplished during the acquisition phase. What matters are the follow-on operations and maintenance costs that will as well as the delivery of capabilities that, with enhancements and upgrading, will make a system viable for many decades after the PEO has moved on. Life-cycle operations and maintenance costs will always exceed the acquisition costs by a large multiplier.

Acquisitions Succeed Because Mistake Are Avoided
Acquisitions do not fail because of a few bad decisions. Nobody is perfect and mistakes will happen. They fail because major errors will add up and form a devastating failure-multiplier. It is the continuous aggregation of often well-intentioned measures that will ultimately add up to a mess. That can be corrected only sometimes with more money or degradation in performance but hardly ever through an improved schedule.

Programs will fail when multiple flaws accumulate. It is the creep in defective engineering that will ultimately cripple any program. Two to five fundamental flaws can be fixed with budget supplements, schedule slippage or by deleting features. More than ten flaws are hard to overcome. When there is an excessive accumulation of what I call “sins of commission” that will result in either redefining what needs to be done or letting the entire venture fade quietly into oblivion.

Here is a list of twenty-six things to avoid:

Eleven Sins of Program Managers:
- Never schedule the completion of a program investment past its financial break-even point. Break-evens should be less than two years for each development increment.
- Never plan a program that takes longer than a half of its useful technology life. Technology life these years is less than five years.
- Never assume that the completion of all of the acquisition cycle milestones will guarantee the delivery of originally planned operating results. The farther the milestones are spread the greater the probability of changes in original requirements.
- Never fail in the delivery of projected metrics for operation and maintenance, such as cost, latency, reliability and availability.  Performance metrics should be used to keep track of the expected results during the entire program cycle.
- Never assume that multiple organizations that depend on a system will reach agreement about the proposed features or the definitions of data.
- Never trust requests for major system changes to be paid by someone who has no accountability for results.
- Never commit to project schedules that take longer to implement than your customer’s average time to the next reorganization. In DoD the time between reorganizations is about two years.
- Never hire consultants to deliver undefined requirements subject to unclear specifications on a time-and-materials basis.
- Never consider a program complete unless you have acceptance from a paying customer who is also directly accountable for results.
- Never add major new requirements to a project after the budget and the schedule are fixed.
- Never test a pilot program by relying on excess support manpower to demonstrate that the system will function for a selected small number of senior executives who have limited requirements. Pilot programs should be always tested at locations where operating personnel can fall back on back-up solutions.
Eight Sins of Technologists:
- Never adopt a new technology unless the top management in charge of operations understands intimately how it works.
- Never design a program that will simultaneously deliver new communications networks, new data centers, new databases and new concepts of operation as an integrated package.
- Never introduce a totally new and untested technology for immediate deployment.
- Never deploy totally innovative software simultaneously with totally innovative hardware.
- Never code a system in a programming language that is unknown to most of your staff.
- Never give programmers access to the network control center consoles that manage computer services.
- Never automate a critical process that does not have a human over-ride switch.
- Never rely on 100% reliability of a single communication link to retrieve data from a critical database.
Seven Sins of Program Planners:
- Never consider a system acquisition program to be complete without with a full-scale pilot test.
- Never assume the recovery of any system from complete failure without frequent re-testing of fail-over readiness.
- Never depend on critical delivery dates without proof that a vendor has done it before.
- Never design a system that will not be able to operate under severely degraded operating conditions.
- Never consolidate data centers that operate with obsolete technology, dissatisfied customers and a management that is technically incompetent.
- Never convert an old application or database to a new one without being able to retrace your steps in case of failure.
- Never engage services of a contractor who has a program manager who lives in a trailer hitched to a pick-up truck (I did that, to my regret).

Build a Little, Test a Little
The above flaws are only a partial list. Systems managers will always discover new ways of how to stumble. In the DoD there is an unlimited opportunity for adding to an inventory of misfortunes.

The best way for preventing a persistent accumulation of flaws is not to engage in the development of applications that take much longer than two years to implement after a prototype pilot has succeeded. Therefore, the installation of applications, in progressive increments, must also fit into an overall design to assure interoperability.  This calls for mandatory compliance with enforceable standards.

All applications must be sufficiently small and modular so that they can fail.  When that happens most of the parts can be then re-purposed for restarting a system at a lower cost because the price of all technologies keeps dropping.

DoD systems programs should never pursue a monolithic approach for the construction of a new telecommunications infrastructure, for a new operating environment and for new databases. New applications should ne never be developed to fit into their own dedicated telecommunication network, a separate operating environment and certainly not with a self-contained database. Applications can be launched fast, but networks, operating environment and databases take a long time.

The telecommunications infrastructure should be an enterprise service. It should be designed incrementally so that that it will support applications when they demand added services. Building a global grid calls for long-term funding and for a completely different management structure than what is required for applications.

The operating environment, and particularly the data centers, should be an enterprise service that can be built incrementally, so that its capacity, security and redundancy can support applications when they come on stream. A mix of government controlled and commercially managed operations calls for financial arrangements that transfer most of the technology risks to equipment vendors.

The databases are the keys to achieving the interoperability and the security of DoD systems. These should be constructed based on shared standards as well as enterprise-wide metadata and not on application-derived definitions.

Finding Payoffs
The acquisition of enterprise systems by DoD must recognize that the total life cycle costs of preset investments into the DoD infrastructure can extend for many decades. What counts for such long-term investments is the discounted present net worth of cash flows for both the infrastructure as well as the applications.

From a financial analysis standpoint, the discount factor for investments for an application should be low if implemented rapidly. The discount interest rate would be a small multiple of short-term Treasury bonds.

The discount factor for investments in the DoD infrastructure must recognize the long time it takes to place communications and an operating environment in place while information technology takes place at a rapid pace. Consequently, the discount interest for infrastructure investments will be a large multiple of long-term Treasury bonds. On balance, that would favor spending money on infrastructure investments that speed up the implementation of applications at the lowest possible cost.

When viewed from the standpoint of life cycle costs the investments in new applications must be also traded off against long-term expenses for operations and maintenance. That will always favor spending money on how to accelerate the installation of applications.
Therefore, DoD policies should make enable the making of life cycle tradeoffs between acquisition costs and how systems are acquired and operated. DoD policies must also guide how investments are made on the infrastructure in comparison with applications.
The principal flaws in the implementation of current acquisition policies are:

1. The PEOs are not accountable for total life-cycle operations and maintenance (O&M) costs of systems. Systems acquisition budgets are funded separately from O&M. There is no visibility of what ongoing costs of applications may be.
2. The costs of the DoD infrastructure (currently consuming over 56% of total IT costs), which includes networks and the operating environment, cannot be traded off against the life cycle costs of applications. Infrastructure costs are widely distributed into enclaves such as DISA and DLA while each Service keeps building additions of their own infrastructure.
3. DoD IT costs are broken up between multiple organizations, each attempting to garner funds to attain self-sufficiency.
4. Only a small fraction of DoD IT life cycle costs is subject to a formal budgetary process where tradeoffs in investment vs. O&M are made. Meanwhile, individual project managers are left to their own resources to concentrate on keeping their acquisition costs below budget.

 Summary
The DoD systems acquisition processes can improve by signing up to the primacy of engineering over management by procedure. The acquisition of computer applications will gain from avoiding any of the twenty-six “sins” of how to manage.

However, the current acquisition policies will still fall far short in delivering what DoD needs.   In the absence of enterprise level tradeoffs how investments between acquisition and O&M costs IT is unmanageable. The acquisition of IT systems differs from the acquisition of weapons. O&M costs, inclusive of military, civilian and contractor labor (which do not show up as IT costs) overwhelm the acquisition costs, which are watched closely.

While the demands as well as the costs to support cyber operations are rising the budgets are shrinking. Cost cuts and additional outsourcing will be insufficient to shift of DoD from an emphasis on kinetic weapons to technologies that support information-based warfare.

The fundamental flaws are not Directives 5000.1 and 5000.2.   It is the funding mechanism that has allowed DoD to accumulate over 15,000 disjointed networks, close to a thousand of data centers and over 5,000 information technology projects that are not interoperable. The current method of funding projects has resulted in thousands of databases that are incompatible and therefore cannot support information warfare.

The DoD IT infrastructure has excessive costs while its performance is inferior as compared with commercial practices. Applications are redundant while their performance is inadequate.

The fundamental flaw in the management of DoD systems is not how acquisition is conducted but how money is spent on the DoD information infrastructure.  At present DoD operates an expensive and ineffective infrastructure. In FY10 it cost $19.5 billion, or 58% of total IT spending. This is expensive because it supports 1,900 projects that can be found in different self-contained budgets, which are managed by separate organizations. Such diversity does not permit the making of the making of tradeoffs between short-term low risk projects and long-term high- risk infrastructure investments. As result, all tradeoffs are made only locally on a sub-optimal scale. Project management is then burdened with elaborate restrictions that try to achieve economic trade-offs by administrative means that ignore sound engineering methods.