The Digital Universe

A year ago this month, IDC produced a white paper for EMC entitled, “The Expanding Digital Universe: A Forecast of Worldwide Information Growth through 2010,” this month the forecast was revised. The digital universe that a year ago was merely expanding is now “diverse and exploding.” IDC computes the size of the digital universe last year at 281 exabytes or 281 billion gigabytes and reports that, “By 2011, the digital universe will be 10 times the size it was in 2006.” From another perspective: “The number of digital ‘atoms’ in the digital universe is already bigger than the number of stars in the universe.”

Largely accounting for the increase in size of the digital universe is the growing amount of visual data: both still and video images and their increasing resolution. Though the conversion from film to digital is almost over in the still image arena, this may serve only to drive the creation of more images (the resolution of which will continue to rise). In the surveillance world (which is now responsible for a growing amount of bytes) the conversion to digital from film is progressing rapidly from its infancy: “Most cameras are still analog. But shipments of networked digital cameras are doubling every year.” Generation of visual data is likely to continue.

Of course where there is data there is a want to store it and IDC points out storage and content’s symbiotic relationship. “Cheaper storage allows us to take high-resolution photos on our cell phones, which in turn drives demand for more storage. Higher-capacity drives allow us to replicate more information, which drives growth of content.” With storage capacity rising and prices dropping, the math is pretty simple. But dropping prices and rising capacity cannot keep up with data creation, 2007 marked the first time that, “the amount of information created, captured, or replicated exceeded available storage.” This divergence is predicted to continue: “by 2011, almost half of the digital universe will not have a permanent home.”

As the size of the digital universe increases, its composition is becoming more varied. The digital universe is now comprised of “images, video clips, TV shows, songs, voice packets, financial records, documents, sensor signals, emails, text messages, RFID tag transmissions, barcode scans, X-rays, satellite images, toll booth transponder pings, and the notes of ‘Happy Birthday’ coming from singing greeting cards.” The size of the constituent parts is varied as well. “An archived digital movie master kept at the National Academy of Arts and Sciences might be a terabyte. A DVD might be 5 gigabytes. An email a few kilobytes. An RFID signal only 128 bits.” While observing the size and variation, it is interesting to note how the pieces add up to make the whole. “The tiny signals from sensors and RFID tags and the voice packets that make up less than 6 percent of the digital universe by gigabyte,” states IDC “account for more than 99 percent of the ‘units,’ information ‘containers,’ or ‘files’ in it.”

IDC reiterates from last year’s report that at the core of the digital universe remains a critical dilemma: “While 70 percent or more of the digital universe is created, captured, or replicated by individuals — consumers and desk and information workers toiling far away from the datacenter — enterprises, at some point in time, have responsibility or liability for 85 percent.” This has implications for those in enterprise IT positions. Regulations will need to be instituted to address all phases of the data lifecycle: how is it created; is it worthy of being archived; how is it to be stored, searched for, and retrieved, how is it protected? These are the questions on the table, and more attention than previously necessary will be required of them.

Towards the end of the report makes an interesting aside, “Of that portion of the digital universe created by individuals, less than half can be accounted for by user activities — pictures taken, phone calls made, emails sent — while the rest constitutes a digital ‘shadow’ — surveillance photos, Web search histories, financial transaction journals, mailing lists, and so on.”

It appears that Big Brother is watching while the universe explodes.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: