ATP Focused Program Competition 97-04
Digital Data Storage

NOTE: From 1994-1998, the bulk of ATP funding was applied to specific focused program areas—multi-year efforts aimed at achieving specific technology and business goals as defined by industry. ATP revised its competition model in 1999 and opened Competitions to all areas of technology. For more information on previously funded ATP Focused Programs, visit our website at http://www.atp.nist.gov/atp/focusprg.htm.

The ATP Digital Data Storage Focused Program will help industry to develop and improve information technology for a wide variety of business and personal applications. The objective of the program will be U.S. predominance in the high-performance digital data storage market over the next decade. The program will achieve its goal through industry-proposed, supported, and conducted research directed at the major components of data storage systems: the hardware typically associated with data storage such as disks, tapes, and the software needed for the efficient location and retrieval of the desired information.

The need for this program is underscored by two trends -- the increasing digitization of audio, graphical, and video information previously stored in analog format and the growing use of digital document storage and retrieval. As information technology becomes more pervasive and ubiquitous, demand for increased digital data storage capacity and performance is evident in both the consumer and industrial marketplaces.

The first trend is evident in the new standards developed for digital technologies used in audio, still and motion pictures, and television. This is underscored by the recent flurry of activity to establish the U.S. position in digital video and television standards at a various levels -- acquisition, production, control-room editing, transmission and reception. ⁽¹⁾ Digital signals can be transmitted, stored, and manipulated in a far more precise manner than the older analog technology. Digital representations of pictures, video, television and music are therefore far superior to their analog counterparts. It is apparent that the trend towards their adoption will accelerate. Consequently, despite the data storage industry's impressive past compound annual growth rate in storage density (26% on average for the past 40 years), emerging growth markets relying upon digital signal processing will require an ever increasing improvement in storage technology to enable their development.

The second trend is in digital document storage and retrieval. While this may be thought of as an extension of office automation, information processing integration has been taking place over the last several decades and promises to make increasingly accelerated demands upon data storage technology as well. Currently, the United States digitally stores more than 400 billion documents, with 72 billion new documents being added each year. Digital document storage and retrieval will become more and more prevalent. This is, in part, driven by a change in cost structure. The cost of digital data storage has decreased to the point where digital forms are the least expensive means to store most of the information which traditionally would have been printed or microfilmed.

In addition to cost, digital documents have many advantages over their paper counterparts. The most apparent is the retrieval of specific information when and where it is needed. As anyone who has used modern database systems appreciates, the quality and completeness of information available to the user is far greater using automated methods than was ever possible by perusing technical manuals, books, and periodicals. And the future will witness an explosion of modern information and database systems containing everything from medical information to the arts. ⁽²⁾

The technical challenges of the design of efficient software for automated document retrieval is much more complex than simply establishing a classification scheme and cataloging items by keywords. In the near future, 'browsing' through information files, as practiced by today's network surfers, will no longer be practical. Natural language query systems will need to be developed in order to take the fullest advantage of modern data storage technology, yet we have just begun to understand the fundamental issues surrounding such systems. An ATP program in Digital Data Storage provides a national R&D forum for the concurrent design of data storage hardware and software to help solve the underlying problem that will face future generations: how to expeditiously and efficiently find the information that is needed.

The program is designed to provide the research initiatives in data storage technology necessary to keep the United States manufacturers of electronic and computer equipment competitive in the world market. The United States currently holds a commanding lead in digital computer and storage systems. The nation's digital storage industry -- makers of the tapes, disks, and other gear that have become the archives and the retrieval tools of the information age -- achieved its world-leading status by doubling storage capacity about every three years. Now, with competitors matching that rate of progress and new storage-hungry services rolling onto the information highway, industry observers say that regaining lost market shares and pulling away from the global pack will require an annual improvement rate of about 60 percent -- or more than twice as fast as today's blistering pace.

By fostering new industrial alliances, the ATP focused program on digital data storage aims to build the springboard for that kind of leap in technological capability and marketplace performance. Since the electronics, computer and data storage industry are well established in the United States, the channels exist for rapid commercialization of new products developed by this program. Further, if the U.S. manufacturers fail to protect their investment in digital technology, they risk the loss of not only the data storage industry, but the computer industry as well.

In order for U.S. firms to take advantage of the opportunities presented by the move toward digital information and in the mushrooming of information needs, a focused commitment of resources will be required to achieve the following business and technical goals:

• Business Goals:
  
• Reduce costs of R&D and reduce R&D cycle time through effective collaboration among data storage industry firms and between industry, universities, and government. R&D intensity and synergy are critical to success in the international marketplace.
• Bring advanced technology into the marketplace faster. The life-cycle of products is short (typically less than 18 months). Investment in future generation technologies is crucial to the continuous roll-out of new products needed to compete internationally
• Build effective customer alliances by seeking and developing technology opportunities that help blur the boundaries between digital computer technologies and applications where U.S. firms have been most successful against foreign competition and consumer applications where U.S. firms have been less successfulExpand world market share for U.S. digital data storage products. The U.S. must retain and expand its dominance in hard disk drives. In addition, U.S. firms need to be aggressive in developing and capturing markets for new products.

ATP funding will assist industry in reaching these goals by:

• Stimulating R&D alliances among industry firms, universities, and government laboratories to increase the economic efficiency and productivity of the R&D effort.
• Stimulating customer alliances among storage manufacturers and storage users. Such alliances will: a) facilitate concurrent R&D and product design and development, which reduces time to market; b) exploit opportunities for product differentiation; and, c) bridge the gap from computer storage applications, where the U.S. data storage industry has traditionally been successful, to applications in entertainment, education, and other consumer areas.

It is critical that each R&D dollar contribute to advance the technology that will allow industry to proceed with both nearer-term product applications and longer-term strategic development. Market opportunities are enormous if appropriate alliances move technology into the marketplace faster than the competition and a focused effort is made to establish U.S. formats as standards.

Technical Goals:

This program will focus on six technical goals which support these business goals and will thus serve to ensure and improve competitiveness of U.S. digital storage companies in the world market. It is necessary to achieve these goals if the storage industry is to attain future growth and product opportunities. These goals were also chosen in cooperation with the major data storage companies, and their industry representatives, as the major component parts of a program to develop the research necessary for advanced data storage hardware and software for the next decade. ⁽³⁾

Hardware issues:

Media: Improvements in media are targeted for disks, tapes, and magneto-optical systems. For magnetic recording systems (disks and tape) this means materials that have small grain structures capable of 10 gigabits per square inch (Gbits/in² ) for disks and 1 terabyte per cubic inch (Tbyte/in³ ) for tapes. New magnetic recording materials are sought with high coercivities, available in thin films, that can be deposited uniformly and economically. For electro-optical disks, this is the development of materials that have high sensitivity to short-wavelength illumination and advanced multilayered materials that increase storage density per drive.

Heads: Development is necessary to produce high-performance heads for magnetic recording, and improved magneto-optical record and sense technologies. Research is critical to develop new materials for giant magneto-resistive heads that afford high output signals (have a change in resistance, dR/R, which exceeds 30%) and lower noise (high signal to noise ratio) for disks with very high data densities (track widths on the order of 0.5µm to 0.7µm).

Tribology: Closely associated with both heads and media are issues associated with the close spacing between the heads and the media. As the data density increases, the spacing between the disk and the head becomes only a few nanometers, less than the mean free path of air molecules. And lubrication, or tribological issues need attention in order to attain acceptable error rates. For storage densities greater than 10 Gbits/in², separation between the head and disc surface can no longer be assured, and this will definitely require the development of new lubricants.

Tracking: Micropositioning devices are required to position heads over data tracks. As the data density of both magnetic and optical disks increases, the positioning of the heads becomes critical to achieve the highest signal-to-noise ratio and consequently the highest data integrity (lowest bit error rate).

Channel Electronics: Signal processing electronics needs to be improved such that raw bit error rates (correctable errors) of less than 1 error in 10⁴ bits are achieved for the track widths of 0.5µm to 0.7µm. Interlinked with the electronics is the data coding issues that permit errors to be corrected in real time.

Software issues:

Data storage and retrieval software is written for specialized applications at various levels in an information system. Each software level is essential to ensure that the storage system performs correctly across various platforms (computer architectures) and with the user's end application, such as a database.

Embedded Control: At the 'lowest' level is the error-detection and correction (EDAC) and encoding software that is often embedded in data storage units or the disk controller. Improvements here, though invisible to the user, have provided dramatic advancements in performance.

Hierarchical Management: Hierarchical storage management (HSM) software permits the efficient use of various types storage media when large amounts of data are encountered. HSM software can reduce storage costs and management problems, by moving infrequently accessed or inactive files from expensive disk drives to less expensive storage devices, such as tapes. There is an opportunity for enhanced HSM software to improve the performance of large data storage systems. HSM is especially effective for the vast number of PCS sold worldwide last year that were linked to some type of network. ⁽⁴⁾

Potential for Economic Impact

Continued progress in digital data storage will impact the storage industry itself and those industries enabled by digital technology. These industries include the entertainment, education, medical information systems, software development, communications, business and scientific applications, and automated manufacturing. The potential for economic impact is vast. The data storage market alone is currently about $100 billion per year with projections to grow to one trillion dollars per year in 2004 as new applications emerge.

But ramifications are much broader when taking into account markets that will spur the data storage industry, and which , in turn, will be enabled by increased storage capacity and performance. The first is the prospect of delivering "video-on-demand" -- an almost unlimited selection of movies, entertainment options, and information services -- to the nation's 95 million television households. The exact mix of existing and new cable lines, data storage, switching gear, compression engines, satellites, servers, and set-top electronics needed for these prototype systems is still being defined. Nonetheless, several competing system architectures will probably emerge, each embodying a unique set of underlying standards for data storage, compression, transmission, and user interface.

Video storage libraries present a big hardware opportunity. A library of 30,000 hours of compressed video will occupy 27 TBytes of storage. At today's prices, the cost for a magnetic disk farm of this size would seem prohibitive -- $50 million. Optical systems, with considerably lower costs per megabyte, are a much better bet for this role. The video library application requires only a write-once capability. Tape-based systems which both archive and store programs and data, where they could be accessed by multiple users may also make sense. Whichever option is chosen, storage subsystems will be a significant expenditure for buyers of video-on-demand servers and thus a substantial market opportunity for the storage industry. The option that is most like the home equipment in use today is the smart set-top box. At a selling price of $250 to $300, which many think will be likely, the set-top video-on-demand market could approach $2 billion if digital video reaches only 10% of the homes currently wired for cable. ⁽⁵⁾ Video server revenues are predicted to hit the $1 billion mark in 1998. ⁽⁶⁾ Other sources are more optimistic about the whole visual communications market, which is growing at 40 percent a year. It includes such products as videophones, video storage and retrieval systems for cable TV, or phone companies to provide movies on demand.

Computers and data storage are essential to the future education of the adult work force in both industrial and military training. Typically, computer-aided instruction takes many forms from a traditional workstation to provide instruction and drill in specific skills in a classroom setting, to portable units that contain 'work place information' such as electronic versions of repair manuals used in the field. Computer-mediated instruction is proving to be a lucrative business in which computer and software companies are making heavy investments. It is very data intensive, requiring large amounts of digital data in inexpensive and portable form. ⁽⁷⁾

Good Technical Ideas

The technical goal of the program is to significantly improve hardware and software performance, reduce manufacturing costs, and help maintain U.S. predominance in the high-performance digital data storage market. Opportunities and high-risk but high-payoff technical challenges abound.

The degree of performance enhancement is highly dependent on improvements to media. For example, areal density (the density of data per unit area of media) of magnetic hard-drive technology has increased an average of 26 percent per year for the past 40 years, resulting in an order-of-magnitude-improvement in hard drive density every ten years. It is believed that with aggressive research, the rate of increase in areal density of hard drives can be roughly doubled to 50 percent per year over the next seven years. The solution to many challenging research topics will make this possible.

There are many technical approaches to increasing the areal densities to 10 Gbit/in² such as the development of new materials with smaller grain sizes (approximately 10 nanometer in diameter), methods to orient magnetic grains, or produce ordered arrays of magnetic grains. The grain size cannot be reduced beyond a certain limit (approximately 10 nanometers for data densities of 10 Gbit/in² ) since the electrical signal when reading the disk decreases in proportion to the volume of the grain. Presently, the storage industry is just learning the 'envelope' of acceptable and available magnetic materials, their precise magnetic and physical characteristics, and their manufacturing methods to attain the ultimate limits of magnetic recording density.

Closely associated with the media characteristics are the read/write heads and their spacing to the media. For example, the flying height of heads is typically 50 nanometers and is expected to be reduced to 25 nanometers in the next few years. It is expected that, by the turn of the century, the distance between the disk and the head will be so small that lack of contact cannot be assured. This will require the need for lubricants of unique formulations to prevent media/head wear. These materials do not presently exist in useable forms. As the data density of disks increases, substrates of unprecedented surface finish are required for the tighter tolerances that will be encountered in future disk drives. New methods of fabrication, probably based on micromaching, are needed to perform these fabrication steps economically and with high yields.

Coding methods and electronics can provide increased performance of disk drives as the data density increases. For example, one new method of recovering data from a disk is called partial-response maximum-likelihood (PRML), a sampling scheme that permits disk manufacturers to dynamically calculate the probability of a given bit of data being a 'one' or a 'zero' from a sampled electrical signal from the disk head. Alternative methods are ripe for exploration such as the use of multiple heads which allow more complex coding methods to be used to improve the error rates at the lower signal-to-noise ratios. Present codes are all limited to processing codes stored serially, whereas in future systems there would be advantages to using two-dimensional schemes to read tracks of data in parallel.

Magneto-optical (MO) disk drives could potentially double capacity with pulse-width modulation recording, but this requires the precise control of laser power to control the domain edges with sufficient accuracy, a procedure not currently feasible. In addition, it is possible, with greater control of materials and manufacturing technology to reduce the track pitch which will further increase the capacity of MO disks. Additional methods to eliminate cross-talk between tracks, employing improved materials and more sophisticated electronics, could potentially make MO units with a track pitch of approximately 0.6 microns. ⁽⁸⁾

Other research topics could include (but are not limited to) investigations of new magnetic and optical materials and components, microtribology, laser and related optical technologies, and interfaces to existing and future systems. Development of new technologies such as digital optical tape, holographic storage, and atomic force probe schemes are on the verge of becoming reality. The basic science and fundamental research has been done. But much work remains to turn these concepts into practical devices to be sold in the marketplace.

Lastly, the data storage industry needs improved software at various levels in the data storage system to wring out as much performance as possible from improved hardware. For example, for high performance data systems, the information to be recorded is usually 'striped' across several physical data storage devices to provide a more uniform and statistical predictable access to the data in a given time. High-performance systems will require better algorithms to ensure lower data latency (the time from the request for a given piece of data to when it is delivered to the processor) and innovative architectures to ensure high reliability in the event of the failure of one or more storage devices.

Industry Commitment

The data storage industry has several organizations that are instrumental in systematically identifying the needs of both manufacturers and users of data storage equipment. Furthermore, these organizations have a successful history of achieving collaboration, and promoting communications among their member companies. The National Storage Industry Consortium (NSIC) is dedicated to enhancing the competitiveness of U.S. information storage manufacturers in world markets. NSIC, which represents over 95 percent of the data storage industry (in terms of revenue), has established four major research programs, including two ATP projects. NSIC, in fact, credits the ATP as a driving force in fostering collaborations among its members. The National Media Laboratory (NML) is primarily concerned with government use of data storage equipment and the data reliability of media. The Optoelectronic Industry Development Association (OIDA) is dedicated to the promotion of optoelectronics in a wide variety of applications, including data storage. Both NSIC and OIDA have called for concerted efforts aimed at fundamentally new and better storage technologies, the expected outcomes of the ATP focused program.

Program Ideas have been submitted by these and other organizations, including:

• National Storage Industry Consortium
• 3M
• Advanced Recording Technology, Inc. and High Performance Storage, Inc.
• Dow Chemical Company + The National Media Laboratory, Advanced Development Corporation, Metrum, StorageTek, and Antek
• GPTech, Inc.
• Pequot Corporation
• Pinnacle Micro, Inc.
• Advanced Development Corporation

Moreover, each of these organizations were present at the 'Workshop on Focused Programs: Digital Data Storage' (Irvine, CA, March 10, 1994) and expressed their support for the objectives of the proposed program.

Why ATP Funds Would Make a Difference

The major economic benefits of digital data storage technology flow to its customers -- downstream manufacturers of the diverse products enabled by higher performance data storage components -- and users, customers who benefit from enhanced productivity in commerce and industry and from new consumer products. The companies supplying data storage to the marketplace, through such components as disk drives and tape drives, are experiencing severe price competition from both US and foreign competitors and shorter and shorter product life cycles. Revenues have been barely rising and profit margins have been squeezed despite the rapid growth in number of units and capacity shipped. As a result, R&D budgets are strained and increasingly focused on the next year's product rather than the next generation products that will be needed to maintain the US industry's edge in the longer term. The industry is not able, on its own, to fund R&D at a level adequate to meet the long-term needs and escalating market pull from emerging applications.

ATP can help by 1) cost sharing longer-term, enabling R&D in digital data storage; 2) stimulating R&D alliances among industrial firms, universities, and government laboratories which enhance efficiency and productivity of the R&D effort; and 3) stimulating customer alliances among data storage manufacturers and users, targeted to novel types of integrated systems solutions and new applications.

Many of the participants in ATP Joint Venture projects have said that the shared efforts concentrated on early-stage technology development can reduce overall R&D costs and R&D cycle time. With adequate R&D, the US data storage industry can remain a top performer in a worldwide market projected to grow tenfold, to $1 trillion, during the next decade and support the emerging applications markets.

Projects Funded in the First Competition

The first competition of the Digital Data Storage Program received 24 proposals in a wide range of technologies including projects for improved hard disk performance, new substrates and magnetic media for hard disks, novel methods for data storage involving molecular and micromechanical devices, and thin-film deposition tools. Relatively few proposals were received in the area of software, tribology and channel electronics. While this second competition is open to all of the digital data storage topics outlined in this program description, people with proposals in these particular areas are especially encouraged to submit proposals. A complete list of proposals that were funded for the first year, along with a summary of the project objectives, may be found on the ATP Web site: http://www.atp.nist.gov. The titles of the recipients follow. (SA) indicates a proposal received from a single applicant and (JV) indicates a proposal from a joint venture. In the case of a Joint Ventures, the participants of the venture are shown.

• A Revolutionary, High-Density, High-Speed, Low-Cost Optical Information Storage Technology -- (SA) Optex Communications Corporation

• Digital Data Storage Technology via Ultrahigh-Performance Optical Tape Drive Using a Short-Wavelength Laser -- (SA) LOTS Technology, Inc.

• Technology Development for Optical-Tape-Based Rapid Access Affordable Mass Storage (TRAAMS) -- (JV) Terabank Systems, Inc. (Lead), Carnegie Mellon University, Energy Conversion Devices, Motorola, NASA/Goddard Space Flight Center, Xerox, Polaroid Corporation, University of Arizona.

• Ultrahigh-Capacity Optical Disk: Multilayer Short-Wavelength Write-Once and Erasable Optical Disk Recording System -- (JV) National Storage Industry Consortium (NSIC) (Lead) and Eastman Kodak, SDL, and Carnegie Mellon University.

• High-Performance, Variable-Data-Rate, Multimedia Magnetic Tape Recorder -- (JV) Imation (spinoff of 3M) (Lead), 3M (original company), Seagate Tape Technology Inc., and Storage Technology Corp.

These projects funded in the Digital Data Storage Program build on the following projects funded in ATP General Competitions:

• Short-Wavelength Sources for Optical Recording -- (JV) National Storage Industry Consortium (NSIC) (Lead) with Carnegie Mellon University, Eastman Kodak Company, IBM Corporation, and Uniphase, Inc. (1990)

• Ultra-High Density Magnetic Recording Heads -- (JV) National Storage Industry Consortium (NSIC) (Lead) with Applied Magnetics Corporation, Carnegie Mellon University, Censtor Corporation, Digital Equipment Corporation, Eastman Kodak Company, George Washington University, Hewlett Packard Company, IBM, Quantum Corporation, Read-Rite Corporation, Seagate Technology, Stanford University, University of Alabama, University of California, San Diego, University of Minnesota, and Washington University. (1991).

• Electron-Trapping Optical Memory for Digital Recording Applications -- (SA) Optex Communications Corporation (1992)

• A High-Density and High-Speed Read-Only Optical Data Storage System -- (SA) Calimetrics (1994)

Acknowledgments

The author sincerely appreciates the efforts of Jeanne Powell, Joseph Mathias, and Ron Marquardt for their many helpful suggestions and comments.

References:

1. For a more complete discussion of how digital coding techniques will be used as the common language of visual communications for the world, please see the "Report of the Task Force on Digital Image Architecture" published by the Society of Motion Picture Engineers - SMPTE (595 West Hartsdale Avenue, White Plains, NY 10677) 1992. This report considers the factors that have the potential to fundamentally change digital image systems Included is the market situation, as well as technology and regulatory trends, that may reshape our communications systems in the next few decades.

2. Corcoran, Elizabeth, "Bit by Bit, An On-Line Collection" The Washington Post Monday, October 10, 1994, p. 1. Describes the multimillion-dollar National Digital Library Project, based at the Library of Congress, with plans to provide five million rare American artifacts in electronic form by the year 2000.

3. "Storage Technology Roadmaps" prepared by the National Storage Industry Consortium, San Diego, CA.

4. Sutton, Laurie, "Data-storage technology gets a boost: ECCS and Storage Dimensions add to the RAID-storage product market, LAN Times, July 26, 1993, vol. 10, no. 14, p. 99, © McGraw-Hill Inc. 1993.

5. "Pretty picture" (video-on-demand offers opportunities for computer-related companies), Computer Letter, May 3, 1993, vol. 9, no.14, p.1, ©Technologic Partners 1993.

6. Gengler, Barbara, "Out of the closet; storage management, which spans tapes, disks and CDS, is key to your overall network performance" LAN Computing, March 1994, vol. 5, no. 3, p. 10, ©Cardinal Business Media Inc. 1994.

7. Ubois, Jeff, Networking for knowledge (federal legislation to increase educational networking) MIDRANGE Systems April 29, 1994, vol. 7, no. 8, p. 44. ©Professional Press Inc. 1994.

8. Kayanuma, K., et. al., "High Track Density Magneto-Optical Recording Using a Cross Talk Canceler," SPIE Proceedings on Optical Data Storage, Vol. 1316, pp. 35-39.

Date created: November 1994
Last updated: April 12, 2005