Data has always been an important asset in business, however the value of data was redefined with emergence of the big data. The combination of structured and unstructured data coming in data streams from multiple sources provides immense value to businesses reshaping their products, services, business models, revenue models, and much more.
The emergence of big data though is not taking place in isolation; it ties back to advances happening in other technologies. The characteristics of big data – volume, variety, velocity, and value – are either consumed or supported by these technologies. These technologies include data storage, cloud computing, blockchain, AI, IoT, and others. In many cases, this is a two-way relationship; i.e., advances in technologies improve big data processing and outcomes while the improved big data landscape either improves the outcomes of these technologies or pushes them to innovate further.
The relationship between these technologies and big data continues to evolve, making some of use cases that were previously considered unreasonable feasible. This relationship is explored further for some of the key technologies that are changing the global computing and businesses landscape:
- Cloud computing and big data – Cloud computing is making scalable big data possible. The distributed processing of big data requires a complex infrastructure and tools to be set up. Scaling big data infrastructure up or down based on the data processing needs has been a challenge which cloud computing has addressed with their elasticity. Beyond the scalable infrastructure and the storage, cloud computing offers extended capabilities inclusive of big data engine, databases, stream processing capabilities, and the services using big data effectively such as machine learning services.
- Blockchain and big data – As with any data solutions, big data also faces the challenge to build verifiability and trust. There are many real-world use cases, including fraud identifications, mortgage decisioning, and others that can consume big data effectively. However, lack of trust and verifiability can lead to far bigger issues limiting adoption. Blockchain, with its characteristics of trust as a network protocol through decentralization, immutability, and transparency across any type of blockchain network, solves these challenges effectively. The scalability and storage costs associated with use of blockchain networks can be addressed through integration of onchain and offchain networks while technolog advances to resolve these within the design.
- IoT and big data – This relationship seems simple compared to others. IoT produces a tremendous amount of data at breakneck speed. The devices are multiplying at an exponential rate and, with that, data is growing faster than ever. To make sense of this data, big data comes to the rescue, providing capability to stream, store, and process the data to meaningful outcomes. However, this is where complexity comes in. Within the adoption of these technologies, big data may also need to enable these integrations with capabilities to feed the data into blockchain or even empower the AI engine. Watch out for some exciting developments with these integrations.
- Artificial intelligence and big data – Artificial intelligence, specifically with machine learning capabilities, can produce extremely useful predictive analytics for businesses and governments. However, for the algorithms to be designed correctly and not produce biased outcomes with high variance, this requires data generally at a large scale. Once designed, data is again needed in order to test and validate these algorithms and then for continuous training. The data in production, where algorithms need to perform, also needs to be transformed and processed to work with the algorithm. This data is mostly big data, given the value that comes out of combining structured and unstructured data streaming though multiple sources.
Like any other technology, big data has two sides. It does need significant investments, with rapid increase in unstructured data. It is difficult to authenticate big data for not breaching any privacy aspects. It is difficult to gauge and ignore the changes it can bring to society, including the negative ones. We are already seeing the world getting more polarized, driven by the use of this technology. However, ultimately, it is all about doing the right thing and establishing a mechanism to promote using the technology for doing what is right and demoting what is wrong. I personally would have liked to see more defined policies framework for big data at a broader level, including national governments, the way we see it being done for technologies such as blockchain and AI.
Big data has tremendous potential. However, organizations and government entities trying leveraging this potential must also pay attention to the relationship big data shares with the other technologies. If leveraged effectively, the combined value delivered by the big data and the other technologies like IoT and Blockchain can be significantly higher. This does require investments to be made to combine the technologies and drive them collectively through the phases of ideation, and implementation.