Virtual Data Storage

Virtual data storage technology provides a better way to manage storage resources for a virtual infrastructure, offering the ability to:

Increase storage resource utilization and flexibility.
Reduce management overhead
Increase application uptime.
Manage the diversity of the existing storage infrastructure.

Storage virtualization increases resource utilization by providing multiple virtual machines with shared access to a consolidated pool of clustered storage devices. It provides the foundation for distributed infrastructure services such as live migration of virtual machines and virtual disk files, as well as live storage migration, distributed resource scheduling, consolidated backup and automated disaster recovery.

As an automated file system, virtual data stores all the files that make up the virtual machine in a single directory. Optimized to support large files and also performing many small concurrent writes. With automated handling virtual machine files, the virtual data store provides encapsulation of the entire virtual machine so that it can easily become part of a disaster recovery solution.

As a logical volume manager, the virtual data store enables an interface to storage resources so that several types of storage (SAN, iSCSI and NAS) can be presented as datastores on which virtual machines can reside. Enabling dynamic growth of those datastores through aggregation of storage resources and dynamic expansion of disk space. As such, a virtual data environment provides the ability to increase a shared storage resource pool with no downtime. And it also provides a means for mounting a point in time copy of a datastore.

Conventional file systems allow only one server to have read-write access to the same file at a given time. By contrast, in a virtual data environment shared storage allows multiple instances of servers have concurrent read and write access to the same shared storage resources.
The virtual data store utilizes distributed journaling of its file system meta data changes to allow fast and resilient recovery across these multi-server resource pools.

The virtual data store can reduce or eliminate planned downtime by enabling live migration of virtual datastores across heterogeneous arrays with no disruption in service.

(text derived from http://www.vmware.com/technical-resources/virtual-storage/)

2010 Cybersecurity Malware Report

A Forbes 8/26/2010 briefing offers significant insights on current cybersecurity issues and warrants summarization (http://www.forbes.com/2010/08/25/cybersecurity-malware-spam-technology-symantec.html?partner=alerts):

1. In the first six months of 2010 an anti-virus firm created 1.8 million new malicious code signatures and identified 124 million distinct new malicious programs. Dependence on antivirus software that relies on the creation of signatures, or digital fingerprints, to identify threats cannot keep up with the current rate of malware creation. Malware is now able to morph its own code in real-time to evade antivirus software.

2. The popularity of social media is driving the increasing use of online URL-shortening services. Computer users are often unable to see the true locations where these hyperlinks are sending them, thus unsuspecting users can wander into a phishing scam or even malware infection. By April 2010, this number of such hyperlinks was 18% of spam.

3. Specialized malware is a rising threat, such as code that is designed to specifically compromise ATMs. Another example is Stuxnet, a Trojan horse that infects computers and seeks to steal SCADA-related documents.

4. In June 2010 Symantec tallied one in 387 Instant Messages containing some sort of hyperlink, and one in eight of those hyperlinks leading to malicious websites. Just as with links in e-mails and social networking messages, users should avoid clicking on links in Instant Messages.

Apps – the Future of Mobile Computing

Gartner has predicted that the total mobile phone installed base (smart phones and feature phones) will be around 1.8 billion and hence will exceed the PC installed base by 2013. The smart phone installed base will exceed the PC installed base before 2020. When we add to the smart phones the rapidly growing category of portable “net pads” (9.5*7.5 inches, 1.5 pounds, 64 GB) we can anticipate the demise of PCs as the dominant computing appliance in only a few years hence.

How are smart phones and net pads different? They are mobile devices that deliver universal computing ability and connectivity. They can be thought of as handheld computers with voice telephone features. They allow a user to run applications that satisfy the entire range of rapidly changing personal computing needs. They include operating systems that provide a platform for application developers.

Such mobile devices include powerful processors, abundant memory and screens that are sufficiently large to offer displays that are sufficient for most needs. However, the most important feature is found in the ways in which these devices accept “apps” constructed by individual developers.  The leader in this field is Apple.

As of June 2010 there were 225,000 apps available on-line from the Apple Store, from where over 5 billion software applications were downloaded. The apps were often developed by thousands of individual developers. The following is a small sample of what is shown in the Apple Store:

These finger-size touch screen accessible icons down loaded by a user to their iPhone or iPad. A touch on the screen will instantly retrieve the complete software application.
Shopping in the Apple Store is easy. Every icon has a detailed description, such as:

The price for the above app is $0.99 for a perpetual license. It includes a listing of technical specifications, application documentation, screenshot examples and the distribution of fifty-two customer ratings along with write-in comments from twenty-nine users.

This “Bills” application was authored by iBear, LLC, which is a very small company specializing in applications for mobile computing such as smart phones powered by companies such as Apple, Google/Android, Blackberry and Windows Mobile. These applications have been translated into eighteen languages where they are open for independent review and discussion over social networks such as Facebook and Twitter.

For example, only apps created by means of the iOS, the Apple's mobile operating system, are available in the Apple Store. Apple does not permit the iOS to use third-party hardware because it is closed and proprietary, which includes lock-down of any application. iOS also imposes a centralized approval process for any updates and can remotely disable all already published apps. Such restrictions make sense to assure warranted security as well as software quality.

There are other organizations, such as Google and Intel that have adopted idea that software should be made available as modular and ready-to-use  “apps”.  However, what matters is the business model which now makes mobile computing completely different from what has been the CIOs approach to application delivery over the past fifty years.

Summary

The availability of vast number of commercially available, fully tested, monitored, customer reviewed and inexpensive applications simplify the deployment of information systems in the Department of Defense. Mobile computing apps offer a different business model for the fielding of software. The “off-the-shelf” readily deployable apps offer the following advantages:

1. A standard DoD development methodology as well as a standard DoD computing environment makes it possible to produce result-focused software. Such a standard environment is not only technologically feasible but also offer large cost reductions.
2. Developers (e.g. contractors) would be able to reuse software components and apply standard systems development tools to the production of quality software. With much of the support infrastructure already included the amount of code needed for applications will be reduced. Bug defect elimination and software quality testing will be enhanced.
3. The modularity of applications enables small firms to compete for the DoD business. The placement of certified apps on DoD networks for experimentation (e.g. “beta” versions) will be encouraged.  Developers can be paid on a per use basis, with payments handled through the DoD acquisition organization.
4. Security assurance of computer code will improve since each app package will be small and modular for certification. Security infections can be limited to isolated apps and be remedied by issuing new versions.

DoD should change from building applications, which call for costly and time-consuming projects that deliver complete functionality that includes infrastructure, data and displays. Instead, DoD should view all computing as a platform that delivers applications in mobile man-machine interactions. Adopting the new business model for mobile computing should simplify use, increase security and reduce costs.

Contractor’s Intellectual Property

When IT contacts are awarded the resultant intellectual properties, (e.g. software code and application methods) are not owned by the government but remain as the contractor’s assets.

Acquisition contracts are currently written so that firms are encouraged to share their commercial technology with the Navy. The retention of the intellectual property rights by the contractor is also needed to assure the continuation of maintenance and support services because it takes advantage of a contractor’s entire commercial base for any innovation.

After a contractor wins a contract award whatever constitutes inclusion of a contractor’s own intellectual property is hard to discern. Whether a code is installed from the contractor’s own library or whether it is written as government work-for hire is practically untraceable. What constitutes the contractor’s proprietary know-how while implementing and operating applications (particularly when this also involves a proprietary network) is indistinguishable from what the government has already paid for.

When a contract expires and has to be rebid, the contractor will tend to make claims for additional compensation for the transfer of applications to others. Such a liability is a deterrent to the reopening of a project for further competition.  The new bidder will have to pay for what is claimed to be intellectual property to be taken over as a legacy.

When contacts are awarded the negotiated terms are full of traps. The government will end up with all operating and maintenance costs but without clarity how much of that is compensation for the contractor’s intellectual property. Such contracts are particularly difficult to construct if the terms require maximum contractor involvement and a minimum of oversight by government experts. This happens particularly in cases when the government is inadequately staffed while the contractor deploys huge staffs to deliver operating management, all hardware, all operating systems modifications, all applications, security and ongoing maintenance plus conversions from legacy systems. For instance, in the case of NCMI there would be few limits on the amounts of intellectual property that had to be contributed by EDS in order to deal with unplanned conditions.

One of the issues that must be resolved during contract negotiations is the determination of what share of government payments to the contractors is compensation for work done (such as delivery of computer “seats’) and what constitutes a paid-up license for the receipt of software by the government. Sorting out such situations calls for elaborate legal clauses, which are always hard to interpret. The contractual provisions can be further complicated if the contractor can claim that some of the software constitutes “trade secrets” which warrant premium compensation and often involve litigation.

The preferred way of dealing with contractors is to invoke the principle that the government would acquire “unlimited rights” to all systems and applications. This calls for using a “source of funds” test to prove ownership. In the case of NMCI this could be applied to the verification of the costs incurred by EDS over and above the approximately $8+ billion of NMCI billings to the government. Did EDS have to spend funds for which it was not compensated? Were some of these funds applied to compensate for the contractor’s own errors and inefficiencies?

Summary

The government should not enter into agreements where the contractor can invoke unrestricted claims for intellectual property. In cases where a contractor must provide proprietary software, the government should negotiate a license that will give it “unlimited rights” to all licenses. In cases when the contractor is installing off-the-shelf software, the government should always insist on using license-free “open software”. Any contract should be unambiguous to specify that it includes a full transfer (or assignment) of ownership of all licenses, regardless of source or origin. If there are any license fees, these will be paid by the government and not by the contractor. Contracts should stipulate, explicitly, that no intellectual
property claims will be made by a contractor.

However, the greatest challenge involves projects that call not only for the delivery of applications, but also for the furnishing of the entire supporting computing, telecommunications and data management infrastructures. Such costs are at least a half of all Navy IT costs. If a contractor is asked to create a special purpose infrastructure, such as in the case of NMCI, the opportunities for making intellectual property claims could be very large.

Instead, the Navy should proceed with the acquisition of generic and commercial Platform-as-a-Service (PaaS) infrastructures, which cannot make any intellectual property claims.  PaaS services are obtained on a pay-for-transaction basis, with each payment constituting full compensation. PaaS hides the complex technologies that produce such services and are managed by the vendor without government involvement. The government then pays only for the results and not for the delivery of technological complexity.

The NGEN RFI Should be Modified

The NGEN RFI of July 2010 states that the most effective means for achieving NGEN objectives will require the execution of the “ITSM Process Improvement and Implementation” processes (par 2.3.1.2). Par 2.4.3.1 also calls for a NGEN contractor to deliver the government prescribed IT Service Management (ITSM) framework.

In July 2010 the DoD Deputy CIO issued a Standard Process Guidance Document that defines ITSM as inclusive of ITIL v3 as well as the ISO 20000 specifications and as well as the Control Objectives for Information and Related Technology (COBIT) requirements.

The description of the Information Technology Infrastructure Library (ITIL v3) is contained in five books: Service Strategy - 257 pages; Service Design - 317 pages; Service Transition - 251 pages; Service Operation - 251 pages and CSI (Customer Satisfaction) - 215 pages.  These books call for an exhaustively detailed documentation of every aspect of NGEN.

The ISO 20000 requirement is an international standard for IT Service Management. It includes SO/IEC 20000-1:2005 specification and prescribes what an organization must do to deliver managed services of an acceptable quality for customers. The scope of comprehensive documentation includes: Requirements for a management system; Planning and implementing service management; Planning and implementing new or changed services; Service delivery process; Relationship processes; Resolution processes; Control processes; and Release processes.

The Control Objectives for Information and related Technology (COBIT) is a set of best practices for information technology management created by the Information Systems Audit and Control Association (ISACA), and the IT Governance Institute (ITGI). COBIT documentation is exhaustive and can be seen as of primary interest to auditors.

ITIL, ISO 20000 and COBIT are often inconsistent and frequently overlap. For instance, each requires formal mapping of features in order to assure coverage.  A contractor would have to reconcile conflicting interpretations of what documentation is required when trying to comply with ITSM processes as dictated by the OSD CIO. The burdens imposed for full ITSM compliance documentation for small IT programs (less than hundred million) would overwhelm their budgets.   In the case of large projects, ITSM will consume a large part of the budget and will generate untold man-hours of revenue to contractors.

Summary 

Applying ITSM only to NGEN will offer essentially program-specific oversight paperwork. That is not desirable under current budget-constrained conditions. The Pentagon is awash with reports that are not actionable. SECDEF Gates announced on 08/09/2010 that the overall number of oversight reports would be cut immediately by a quarter. This should lead to the shrinking of the need for many ITSM processes for large projects and result in a total elimination of ITSM in the case of smaller projects. [Note: The Navy has only 13 projects with budgets >$100 million, 100 projects with budgets >$10 million and 536 projects with budgets <$10 million. Only very large projects can support ITSM documentation].

However, the application of ITSM to NGEN will have to be re-examined anyway. In view of the mandated cost reductions for all information systems, the Navy should proceed to acquire its infrastructure and data management methods not as an NGEN project, but as the basis of the all-embracing Naval Network Environment (NNE) that is interoperable with other DoD components.

NGEN is planning to break this program into separate contracts with separate functions that are not connected to the implementation of a coherent Navy NNE infrastructure. NGEN should be planning for a Navy-wide data management environment, which includes war-fighting applications, logistics, personnel and financial applications and not only limited NGEN solution, especially if NGEN is further broken up into efforts managed by different contractors and dozens of subcontractors. The NGEN July 2010 RFI will have to be modified to de-emphasize ITSM and start addressing the total Navy Enterprise infrastructure design which should serve the Navy's Information Dominance objectives.

Commercial experience shows that an enterprise must have a coherent enterprise-wide design for its entire infrastructure. You cannot make a baby by contracting to different mothers for making the hands, feet, body and the head!

The current RFI cannot deal only with NGEN by using ITIL. It must become seen as a component part of much more encompassing NNE environment.

Army's RFP for Private Cloud Operations

1. The Army will reduce the number of data centers from over 200 to less than 20 by establishing the Army Private Cloud (“APC2”).

2.  There will be two contracts: A. Using commercial private cloud computing capacity.   B. Acquiring containerized data centers that can meet urgent needs where rapid or temporary cloud computing is needed.

3. APC2 will use pay-for-use Private Cloud capacity instead of acquiring equipment and paying separately for consulting services to operate the environment.

4. The APC2 will employ best of breed, commercially available services using short-term contracts.

5. Contractors will own and operate all facilities, including all hardware and software provisioning.

6. APC2 contractor services will include:  Assurance of network connectivity; Application migration; Security assurance; Provision virtual Operating Environments; Capacity planning and forecasting/trending for growth; Configuration and management of customized servers, storage, security and networking devices; Disaster Recovery and Business Continuity planning and execution services; Migration planning, scheduling, coordination and implementation; Support continuity of operations; System administration and monitoring services; Maintain network uptime and network availability guarantee;  Manage vulnerability and incident management; Perform access identification and authentication and many others.

7. APC2 contractor will also provide the following: Service Desk / Service Request Management; Incident Management; Problem Management; Change Management; Release Management; and Configuration Management.

8. Maximum recovery objectives are four hours, with average availability at 99.995% (though the method how this will be calculated was not specified – see Strassmann blog on the exponential characteristics of failures in computer networks). In a cyber warfare environment four-hour recovery is not acceptable. The RFP does not address fail-over requirements that would assure 100% uptime under critical conditions.

Summary

The Army is handing over to the APC2 contractor, in addition to hardware/software operations, an all-inclusive list of systems management functions. For all practical purposes the role of Army management, to be accountable for the total performance and the security computing services, is not visible. Whether any contractor can deliver everything that is required (a brief summary is in par 7 and 8 above) within a pay-as-you-use pricing structure is questionable.

The role of the contractor to provide the Army with reports and status checklists is inconsistent with the goal of making cyber operations an integral part of the Army's warfare roles. The way the RFP has been written the support of warfare networks is treated more like a back-office activity which has been increasingly outsourced in the past.

Though the dedication to proceed with cloud computing is long overdue and highly commendable, the working out of how the Army can be held ultimately accountable for operations remains unresolved. As a minimum the end-to-end network control of every device as well as the management of the security of the entire network should be Army organic and not be outsourced.

Network Downtime Metrics

DoD’s largest network (NMCI) is supposed to keep track of average monthly uptime metrics. The Service Level Agreements (SLA’s) call for an average 99.7% uptime, which results in 2.2 hours of downtime per month. The > 400,000 clients network would then have a total downtime of 876,000 hours per month.

How the downtime is calculated contains a number of provisions, which make the determination of the actual number of available network hours difficult to find. Scheduled downtime, preventive maintenance, bug fixes, hardware upgrades and software enhancements are excluded from downtime hours. It is also not explained whether the SLA uptime applies to end-to-end performance, e.g. keyboard to data center connectivity.

Hitting the target of 99.7% uptime is not difficult. The NMCI calculation is the average of the total measured population.  The larger the number of clients included the easier it is to meet uptime number.

An examination of the incidence of failure in a network will show that the probability of failures is exponentially distributed (see http://en.wikipedia.org/wiki/Failure_rate).

1. There will be always a small number of clients that will have failures greater than the average. A few of these will be out of service for an extended time period.

2. There will be always a very large number of clients that have failures substantially lower than average.  A large number of clients will not have any failures except for scheduled downtime.

3. The average downtime reported by NMCI will be related to the number of clients included in the average. The larger the population that is included for reporting purposes, the lower the reported average. The effects of a small number of excessive failures will be masked in the number of cases included.

SUMMARY

The calculation of network downtime using averages is misleading. In information warfare a small number of critical clients with excessive failure rates is unacceptable.

Information warfare network reliability metrics should not focus on broad averages but on the number of critical clients with failure rates in access of their time to restore abilities.

The time to restore is difficult to predict, especially on ships. The only solution to reducing downtime risks is to adopt end-to-end network redundancies for critical components of the NGEN network.  Automatic fail-over rates can assure near zero failures. This should be pursued because it is an economically feasible solution. The Navy is in an excellent position to adopt for NGEN a zero defect approach for critical parts of its future networks. The economics of virtualization makes that possible.