Enabling text-based insights: infrastructure matters

As readers may have noticed, the techniques regarding data collection and analysis are already established and still developing. These “soft” assets, however, can only reach their full functionality when hardware conditions are met.

Perhaps the more practical question is why these use cases are not widely applied. Besides the path-dependence issue within the public sector and the lack of the awareness of big data before the wave of artificial intelligence, we argue that the nonexistence of modern data collection and analytical infrastructure restricts the full potential of the NLP techniques.

For one thing, the OCR and various data mining techniques require sufficient storage and adequate servers to digitize the documents accordingly. Without reliable data storage and pre-processing infrastructure, there is no way to employ cutting-edge analytical algorithms. As for analytics, the public sector needs to use cloud-based solutions or high-performance computing (HPC) to derive text-based insights effectively. Without an infrastructure for big data, there will not be any valuable evidence to derive.

In addition, I would like to mention the role of human intervention with regard to the NLP pipeline. Applying the quantitative method does not mean that qualitative endeavor is obsolete. Instead, for all supervised or rule-based training of the models, human labeling is essential for future success. Also, qualitative knowledge about specific administrative or political issues can inform the NLP solution developers about model-relevant issues so that the pipeline can be tailored to the needs of various tasks. For instance, particular word combinations such as “parental leave” are important features for some political fields. Dividing them into “parental” and “leave” is not informative.

To sum up, NLP techniques provide the public sector with a variety of new opportunities to analyze relevant data in unstructured text form. There are already many established use cases that use cutting-edge algorithms to amass large amounts of data and deliver politically and administratively relevant insights in an automated fashion. Nevertheless, an infrastructural pipeline is essential for the success of the analytic engine, and for valuable qualitative insights and labeling practices.

You can get in touch here.

Qixuan Yang

Expert in Data analysis, Machine Learning, Research

As a Data Scientist in the AI Garage of Capgemini Invent, I work with technical as well as subject matter experts to develop data-driven solutions and products. I am particularly interested in machine learning, especially its applications in the natural language processing (NLP). Being in-fluent in both Python and R and close to academic discussions, I attempt to identify the best strategies for enterprises to turn internal and external data into valuable analytical assets and distill the signals from noises.
Since I joined Capgemini, I have been working in the banking and the pharmaceutical industry for standardizing the analytical pipeline.
My recent work includes:
– Designing and implementing a reporting tool for a leading German bank, which significantly reduces manual efforts and saves cost
– Functioning as NLP expert in a pharmaceutical company in Germany to build a plausible ML/NLP pipeline
I previously worked in the academic field and had important operating role in algorithmic design of data-driven social science projects, incl. text classification and topic modeling. I am particularly interested in the intersection between social science and statistics.
Previously, I have studied at University of Konstanz and Yale University, and joined research teams and led training and tutorials in both institutions. For instance, I created a tutorial for NLP at Yale University and led a training at its graduate school.
I am fluent in Chinese, English, and German. In his spare time, he is a passionate swimmer and blogger.