The Challenge is Staying Afloat in a Rising Sea of Data

 

“A $40 million computer tells you you're chasing an earthquake, but you don't believe.” – Capt Mancuso, “The Hunt for Red October,” Paramount Pictures (1990)

 

One of the toughest jobs in any country, any company is gathering, hoarding, protecting information that may give them an edge against “them.” And we must be doing a helluva’ job because organizations and individuals are accumulating data and meta data at an ever-increasing rate.

The sheer volume that’s coming at us is causing us to experience information overload…emails to answer, virtual friends to contact, YouTube videos to watch, games to play, movies to sneak, reports to read/digest, stuff you have to find/do in the virtual world…just in case. Computers, smartphones, tablets are helping to accelerate our hunger for more data and our ability to create new/better information.

Does it seem as though there’s so much information, you can’t keep up with it? Can’t make sense of it?

You’re right!

As Jack Ryan said, “Welcome to the New World, Captain.”

The Next Big Number
EMC (a world leader in big storage stuff) recently sponsored a report by IDC (international research firm) that says we have finally cracked the Zettabyte barrier.

 

 

 

Piling Up – When it comes to storage capacity a MB isn’t much, a GB is barely tolerable and a TB is almost a necessity for families today. It wasn’t that long ago that we considered a PB as an amazingly big number; but today, we talk about a Zettabyte without blinking an eye. Storage requirements are more than doubling every year and there’s no end in sight -- as long as we can properly manage it and find the specific information we want and need. Chart - IDC

This year, we will have created, replicated more than 1.8 Zettabytes of data (images, video, music, graphs, stuff) which represents a 9x growth in just five years.

And we’re getting better at it.

Over the next 10 years, data centers/repositories – enterprise, cloud, personal – will grow by a factor of 50 while the number of files these locations will have to deal with will increase by a factor of 75.

As Admiral Painter commented, “This business will get out of control.’

Why are we capturing, storing, hoarding all of this data?

Industries of Knowledge
The global economy isn’t based on “things” anymore; it has become a knowledge economy and the countries/organizations with the smartest, best-trained workforces, best ideas and the ecosystem to support innovation will be the leaders.

The value of the data you and everyone captures was expressed by Stewart Brand, founder of Whole Earth Catalog, at the first Hackers’ Conference in a conversation with Steve Wozniak (co-founder of Apple) in 1984.

In his discussion with Woz, Brand said “On the one hand, information wants to be expensive, because it’s so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free because the cost of getting it out is getting lower and lower all the time. So you have these two fighting against each other.”

Viola!! Brand gave a whole band of Black Hatters (hackers) a rallying cry.

No good deed goes unpunished.

The innocent conversation also gave bands of super-bad, super-annoying Black Hatters like Wikileaks, Anonymous, Lulz Sec and others a rationalization for their activities.

But simply outing private data because you can doesn’t accomplish anything. In fact, most of the time, it does more harm than good.

As Capt. Mancuso observed, “The hard part about playing chicken is knowin' when to flinch.”

 

Opening Data Vaults
Whether they are being pulled or pushed, many organizations are opening their data vaults to countries, companies and individuals to develop even better information, better solutions.

 

The most valuable commodity companies and individuals have is the data they want to save as securely as possible. With knowledge becoming such a valuable commodity, some organizations have found that the more open they are with their information, the more solutions they find themselves. Art - NYTimes

The World Bank, long a treasure trove of information, is making that information available to democratize development economics.

Historically, the bank hoarded information and released bits and pieces in a choreographed fashion to a specific audience.

With the change, organizations and people everywhere have been tapping into and using the free data.

The information has been used to reshape government policies around the globe; and in some instances, to reshape governments when people use it with their smartphones, Twitter statements, Facebook/Google+ postings.

The new computing/communications tools create – you guessed it – even more data.

Graphic Illustration – There are almost limitless ways to describe how our data is increasing. By 2020, the digital universe will be 44 times larger than last year and will include the Petabytes of storage individuals will have in their homes. IDC

With the right tools, IDC points out that the overload of information is becoming increasingly valuable and available across all sectors of the digital universe.

Improved global network connectivity, lower cost computing power and streamlined/focused apps make it easy for companies – large and small – to tap into and use transactional data, warehoused data, metadata and data located virtually anywhere to respond to, assist and serve customers.

Data on Everything, Everyone

Created, Consumed – While producing data from data, about data, may seem like a whirlpool or black hole; it is our ability to analyze all of the data surrounding other data that delivers answers to unasked questions. From tomorrow’s infrastructure, apps, governance and consumption, big data is also changing the datacenter and its team. New opportunities appear with each new content delivery source that is launched. IDC sponsored by EMC

The big data that is getting bigger all the time isn’t just original content waiting to be used.

Improving with Age
It’s more valuable than ever because data is captured about the content consumers use, which is arguably even more valuable than the original content.

In other words, the data around the data.

As Jack Ryan said, “It is wise to study the ways of one’s adversary.”

Better Information, Better Assistance – Companies, both large and small, are increasingly turning to the broad range of internal/external information to manage their organizations, plan products/services for tomorrow, and speed/improve product and customer support. – SAP

As IDC explains, big data calls for new technologies to economically extract value from huge volumes of a wide variety of data so that it can be captured, analyzed and used.

Fortunately, while the mass of information is growing exponentially, the cost of storing it is steadily dropping.

Decreasing Data Storage Costs

Demand Increases, Costs Decrease – As companies and their IT activities change, there will be changes as the digital universe expands. It will be more complex, require more processing, more storage and increased management which will raise the levels of investment. Fortunately, storage costs will decrease to meet the exploding storage requirements. Source – IDC sponsored by EMC

That’s a good thing, at least in our household where we’re constantly adding TB drives.

Why?

Because even in a free information society, there are some data that must be safely/securely protected.

Privacy Still Vital
In the digital universe, there are key areas of security that must be maintained:

• Privacy – say your email address on a Sony account or YouTube upload
• Compliance – email retention in the event of litigation or legal compliance
• Account information – such as recent customer/user identity thefts
• Confidential – customer lists, trade secrets
• Lockdown – the highest security for medical records, financial transactions, military actions,

As Jeffrey Pelt pointed out, “I'm a politician which means I'm a cheat and a liar, and when I'm not kissing babies, I'm stealing their lollipops. But it also means I keep my options open.”

The bad Black Hats argue that all data, all information should be freely available; but somehow we don’t believe that will hold up in any court…including the court of public opinion.

Our desire to accumulate more and more information is insatiable. The key to your success though is how you filter, discard and use that information … and how often you’re right or wrong.

There’s still hope, as long as we review the data and rely on gut feelings.

As Jack Ryan said, “I had a 50/50 chance. I needed a break.”

Finding the data in an ocean of files is becoming a business opportunity for app developers and submarine drivers.

As Rich Wallace, publisher/editor of The Next Silicon Valley pointed out recently, there are increasing opportunities for entrepreneurs, start-ups and established firms everywhere to take advantage of today’s knowledge-based economy.

Andy Marken is CEO of Marken Communications

I will be giving a copy of the book, Social Media Judo, starring yours truly to three randomly-selected people who email me at john.obeto@absolutevista.com.

Should the winner prefer, a Kindle version of the book will be substituted once that becomes available.

This free offer ends on September 30, 2011 at midnight Mountain Time.