Big Data success needs Big Metadata transparency

Publish date:

  Eisenhower once said, “In preparing for battle I have always found that plans are useless, but planning is indispensable”. If the new battle is that of Big Data with revenue generation, cost reduction and/or risk mitigation the outcome, metadata may be integral to your plan but, Big Metadata will provide the adaptive planning mechanism to progress with transparency […]

 

Eisenhower once said, “In preparing for battle I have always found that plans are useless, but planning is indispensable”.

If the new battle is that of Big Data with revenue generation, cost reduction and/or risk mitigation the outcome, metadata may be integral to your plan but, Big Metadata will provide the adaptive planning mechanism to progress with transparency and confidence.

Am I in ‘cloud cuckoo land’ with my data?

With information quantities growing exponentially and, with the gradual adoption of cloud services become the norm rather than the exception, how can the CIO and the CRO move forward with increasingly distributed IT platforms with confidence?

CIO – how can I assure that the data on which we are basing critical decisions is traceable, audit able and understandable?

CRO – how can I assure that my information assets are not exposing my organisation to unnecessary risk or compliance failure?

CSO – how can I assure that our critical information assets and documents are protected to the same level inside and outside our firewall?

The answer is Big Metadata and if it’s not coming to a project near you soon, it should be.

It’s metadata Jim but not as we know it!

I know most of you out there sweat when somebody mentions the word ‘metadata’. It is not the most inspiring of subjects yet, it’s definition and importance is one of the most commonly misrepresented messages in the field of IT.

It is however breathtakingly simple and, you will be pleased to note, we do not have to resort to too much technical mumbo jumbo.

Metadata is the transparency of your IT systems and how they interact in the context of how your organisation operates.

Big Metadata is therefore, a more scalable and flexible construct incorporating increasingly unstructured and, externally distributed operations at the speed of Big Data interactions with, the additional context of how any given external 3rd Party is interacting with your business

The ‘Better Big Data = Bigger Metadata’ Paradox

Clearly, Big data focuses on rapid information consolidation, analysis and provision to guide our decisions at key points in our B2C and B2B interactions. It will enable our IT platform to become the equivalent of Walmart’s supply chain or, Amazon’s channel mechanism within our own industries.

Yet, it has a hidden impact that Big Data vendors do not want you to know…….

Big Data will increase fragmentation and diversification of existing systems, their interfaces and related quality assurance efforts, and all this at a time of increasing regulation and, in an era where lack of transparency is leading to unnecessary levels of risk and, potentially large fines for lack of compliance.

So, I conclude that, if…

Big Data = our ability to choose the next best course of action with context

Big Metadata = our ability to do this with confidence whilst mitigating risk.

Big Data Tipping Point. No metadata dumping!

So first things first: we are at a tipping point with Big Data and as such, toolsets and approaches to this challenge are not fully mature as of yet (as far as I am aware). So here is a stab and what I think a Big Metadata tool and approach needs to provide:

Big Metadata for 2013……..

  • Understands our IT landscape and how it relates to key business processes
  • Knows which cloud services are linked to your on-premise systems and, what data exchange is occurring between the two
  • Handled structured data and unstructured data standards with equal aplomb within a single corporate toolset (from SQL to Hadoop HDFS, XML to Video)
  • Has a toolset that is centralised and dedicated and is therefore unlikely to be the independent metadata components within your existing applications.
  • Is flexible, adaptive and, is able to integrate new data sources quickly and seamlessly on a configurable basis with limited (if any) coding required.
  • Provides a governance and scorecarding process that can continuously assess the health of your organisations data assets.
  • Embeds the risk or regulation mitigation process into its very fabric to ensure that you are regulating compliance in the terminology of your organisation rather than regulating the differences between technical IT interfaces and their data interchange formats. (Think: where am I on FATCA compliance rather than where am I on XSLT to XML interchange…..)

Oh, and the next time you are asked to justify a metadata foundation within your own business, just think of Eisenhower preparing for battle and you might be more successful!

Related Posts

analytics

Accelerating the customer experience (CX)

Goutham Belliappa
Date icon September 15, 2020

There is no single answer or recommendation on the preferred architecture. It depends on the...

Customer Experience

Accelerated digital customer experience possible only through “intelligent content”

Karthick Purushothaman
Date icon July 17, 2020

Excellent customer experience cannot be achieved if you do not think about “content” in the...