Big Data…Panning for Gold

Publish date:

We often talk about finding the ‘golden nuggets’ in reference to the insight we hope to gain from our data…but is this just a blind faith we need or is there something more to it? In 1848, James W Marshall found gold in Coloma, California. In the space of just a few years, the population […]

We often talk about finding the ‘golden nuggets’ in reference to the insight we hope to gain from our data…but is this just a blind faith we need or is there something more to it? In 1848, James W Marshall found gold in Coloma, California. In the space of just a few years, the population boomed, with thousands of prospectors arriving hoping to find their own nuggets of gold.

The geology of California meant that the early pioneers were able to pan for gold in rivers and streams by simply sifting through small amounts of stone and gravel; sometimes coming up short, other times hitting the jackpot. Over time, their technology, their tools and their techniques improved to allow them to search and thus extract the gold at scale. The gold rush led to all sorts of social, commercial and financial benefits for the region. It was sparked by one individual finding gold…and then continuous experimentation and exploration for others to find theirs.

Looking for nuggets in data has followed a similar path. For a long time we’ve had user productivity tools that have allowed us to sift, to interrogate, to search out our own golden nuggets, but this has only been possible on a relatively small scale. The ‘geology’ of our data landscapes have changed massively in just a few short years, making the job of finding insight all that much harder. Of course now that the big data technologies have matured enough, it makes the job of analysing the volume and variety of data both possible and affordable.

Of course this doesn’t mean we have to dig up the whole hill to find out if there is gold…and this is important…we start small (prove the use case), do it quickly (succeed/fail fast) and then scale out once we’ve seen what we need to. Additionally, just because the ‘pan’ of stone and gravel doesn’t immediately turn up gold (the use case), it doesn’t necessarily mean it has no value…even fools’ gold had its uses for a while.

So, my personal view is that yes, there is a bit of ‘faith’, a bit of ‘belief’ in the premise behind big data, but then I think that exists in other fields too. Combine that with the right tooling, the right approach (starting small), the right level of agility…and of course, an understanding of what you’re looking for…and we could well find our own golden nuggets.

Related Posts

Insights & Data

Gesture recognition for a safer, more inclusive society

Date icon August 12, 2021

The emergence of hot tech: Gesture control and touchless user interfaces ~ for a low-touch,...

Insights & Data

Time to shift the gear with software defined vehicles

Date icon August 5, 2021

Software and Data Drive Change in the Automotive Industry.

Business Data Lake

Why data needs to be hands-free

Danny Centen
Date icon October 14, 2020

Data is being democratized, extending the audience for business analytics by sharing access...