We often talk about finding the ‘golden nuggets’ in reference to the insight we hope to gain from our data...but is this just a blind faith we need or is there something more to it? In 1848, James W Marshall found gold in Coloma, California. In the space of just a few years, the population boomed, with thousands of prospectors arriving hoping to find their own nuggets of gold.
The geology of California meant that the early pioneers were able to pan for gold in rivers and streams by simply sifting through small amounts of stone and gravel; sometimes coming up short, other times hitting the jackpot. Over time, their technology, their tools and their techniques improved to allow them to search and thus extract the gold at scale. The gold rush led to all sorts of social, commercial and financial benefits for the region. It was sparked by one individual finding gold...and then continuous experimentation and exploration for others to find theirs.
Looking for nuggets in data has followed a similar path. For a long time we’ve had user productivity tools that have allowed us to sift, to interrogate, to search out our own golden nuggets, but this has only been possible on a relatively small scale. The ‘geology’ of our data landscapes have changed massively in just a few short years, making the job of finding insight all that much harder. Of course now that the big data technologies have matured enough, it makes the job of analysing the volume and variety of data both possible and affordable.
Of course this doesn’t mean we have to dig up the whole hill to find out if there is gold...and this is important...we start small (prove the use case), do it quickly (succeed/fail fast) and then scale out once we’ve seen what we need to. Additionally, just because the ‘pan’ of stone and gravel doesn’t immediately turn up gold (the use case), it doesn’t necessarily mean it has no value...even fools’ gold had its uses for a while.
So, my personal view is that yes, there is a bit of ‘faith’, a bit of ‘belief’ in the premise behind big data, but then I think that exists in other fields too. Combine that with the right tooling, the right approach (starting small), the right level of agility...and of course, an understanding of what you’re looking for...and we could well find our own golden nuggets.