Dodging a dark data disaster

It turns out that I have a bad habit.

As the world becomes more digital, our need for processing power will only increase

I caught an acquaintance peering at my phone the other day. They looked perplexed. In fact, I would go as far as to say they looked dumbfounded.

“What on earth are you looking at?” I asked.

“How many unopened emails have you got?” they inquired.

I glanced down at my phone. It was just over 40,000.

The look of disgust said it all.

It turns out that humans are divided into two tribes: those that manage their inboxes and those that don’t.

Those who do cannot comprehend why those who don’t cannot keep up with such a basic thing.

For those who keep their inboxes to zero unopened emails, the rest of us are anathema. A mailbox full of unopened mail is the modern-day equivalent of running your fingernails down a chalkboard.

I didn’t dare tell the acquaintance that one of the reasons (among many others) that I left my last job was because I hit six figures in unopened emails. Hitting that number puts your inbox beyond redemption.

As I say, it is a bad habit. Actually, it turns out it’s a terrible habit.

Soon after this exchange, I found myself talking with a data centre company. I have become fascinated by data centres, their role in our ever-increasing digital lives, and the incredible efforts that some of these companies are putting into environmental strategies against all odds.

Data centres are the digital factories of the 21st century, consuming energy and water and generating lots and lots of excess heat while providing a service our digital lives demand.

Over the course of the conversation, we ended up talking about cloud storage and the vast amounts of data that we all have saved around the world.

Someone on the call mentioned a friend who had just returned from holiday. This friend had decided to keep only their 20 best holiday photos and delete the rest, reasoning that most photos lie dormant on the cloud, never to be looked at again, each with a digital footprint equating to energy and water.

Think about the last time you went on holiday. How many did you take? And how many did you delete?

It turns out that I have 73,592 photos and videos in the cloud. Each of these is stored in at least two places. And each is consuming energy and water.

I was then introduced to the concept of ‘dark data’.

Dark data is the extensive amount of unstructured or semi-structured data organisations collect but do not effectively use or analyse. This data often remains “in the dark” and is not leveraged for decision-making or business insights. It is the corporate equivalent of unopened emails or pictures sitting on your photo stream that you never look at.

Financial institutions are big culprits of this, often accumulating large volumes of data including customer transactions, emails, and documents.

This unused, unanalysed data could be valuable for improving user experience, developing new financial products, or enhancing security measures. Instead, it sits doing nothing other than consuming energy and water.

According to Slingshot Simulations, up to 52% of all information an organisation produces and stores is dark data, and this is rapidly increasing.

And its overall environmental impact is massive.

According to Veritas Technologies, dark data contributed a staggering 6.4 million tons of carbon dioxide in 2020 (about the same as Cyprus). According to IDC, the amount of data the world will store will grow from 33 zetabytes (ZB) in 2018 to 175 ZB by 2025. The implication is that unless people change habits, there will be 91 ZB of dark data in five years – over four times the volume we have today, with all the energy associated with powering the infrastructure in which it lives.

Individuals and corporations are sitting on vast amounts of dark data.

So, what can be done about it?

For individuals, a digital spring clean might be in order. I love the simplicity of being ruthless with photographs and only keeping a select few. Is this also an opportunity for some hardware innovation?

Someone has helped me set up an email filter on my inbox, looking for the word “unsubscribe”. I am now happily deleting spam emails comprising most of my unopened mail. In his brilliant book, “How Bad Are Bananas?”, Mike Berners-Lee tells us that a spam email’s carbon footprint is 0.3g of CO2. And that’s just sending it. Rapidly deleting means no footprint for storage.

For corporates, a digital spring clean is also likely to be required. And a ruthless approach is needed going forward, ensuring that only useful or regulatory-required data is stored.

My data centre friends have massive challenges on their hands. As the world becomes more digital, our need for processing power will only increase. Ridding the world of unnecessary bits and bytes won’t just help them. It will help us all.

About the author

Dave Wallace is a user experience and marketing professional who has spent the last 30 years helping financial services companies design, launch and evolve digital customer experiences.

He is a passionate customer advocate and champion and a successful entrepreneur. 

Follow him on Twitter at @davejvwallace and connect with him on LinkedIn.