The future of life sciences: Where multimodal AI meets personalized healthcare

Capgemini Invent examines how multimodal AI and digital tools enhance treatment monitoring, real-time adverse event detection, and patient safety

Vast amounts of data exist across the life sciences sector, but it is often siloed or too complex for traditional analysis. Multimodal AI unlocks this untapped potential, integrating research data, clinical records, and real-world evidence to transform how we understand diseases, develop drugs, and personalize care. This shift promises a more nuanced, data-rich approach to medical research and patient care. Multimodal AI in healthcare will lead to better outcomes for both patients and practitioners.

What is multimodal AI?

Multimodal AI is a category of deep learning that is capable of processing multiple types of input simultaneously (i.e., text, visuals, audio, gestures, environmental cues, touch, physiological sensors, and GPS). Such models move beyond early unimodal models, which can only process one type of input. By leveraging multimodal data, AI can model more diverse domain knowledge, make more accurate predictions, and tackle more complex challenges.

Pioneering companies are beginning to use multimodal medical AI for drug repurposing. The integration of research publications, clinical results, and even molecular structures can unlock hidden potential within existing drugs, potentially delivering life-saving treatments more rapidly. 

Leading the digital revolution, advancements in large language models (LLMs), such as ChatGPT, have transformed the way we process and generate text that mimics human conversation. These models have a wide-reaching impact, extending beyond text creation to influence personal and professional aspects of life. By automating tasks that were once manual, these technologies are changing how we interact with digital platforms and make business decisions.

Alongside LLMs, the rise of large multimodal models (LMMs) marks the advent of multimodal AI. This AI iteration processes diverse data types (text, images, audio, et cetera) to create comprehensive domain knowledge models. Within the life sciences, multimodal AI in healthcare offers significant improvements in patient care and operational efficiency across the pharmaceutical value chain and throughout the whole life science field. The reason for this is simple: the integration of diverse datasets reveals unprecedented insights, making AI for healthcare a true gamechanger.

Multimodal data integration is crucial for understanding complex systems and structures through data, mirroring the diverse ways humans process information. It enables artificial agents to mimic nuanced human-like communication and is particularly vital in the life science sector, where data from research, patient records, genomics, and real-world evidence are foundational. Traditional isolated analysis of these datasets often led to fragmented insights, but AI’s capability to synthesize multimodal data heralds a new era of integrated analysis. However, adoption of multimodal AI presents several challenges, especially within the regulatory sphere of healthcare data handling and privacy.

“Multimodal AI is the future of AI, because it allows machines to perceive and understand the world in the same way that humans do.”

Fei-Fei Li, Professor of Computer Science at Stanford University

The rise of generative AI (Gen AI) and its foundational transformer technology has generated substantial excitement. And yet, the full potential of the transformer technology, crucial for enabling multimodal AI and artificial general intelligence (AGI), remains untapped. The following is an introduction to the field to help business leaders successfully integrate this emerging technology and unlock its many vectors for growth.

Artificial intelligence terminology

The transformative impact of AI extends beyond processes and products. It has the potential to reshape business models and influence the global economy. With such broad reach, innovators and executives need clearly defined and categorized key terminology (see figure 1).

The terms described in Figure 1 are likely to become household names in the near future. And yet, despite having a vague notion informed by popular culture, few people truly understand what artificial intelligence is. Fewer still, even within the sector, know how this emerging technology can be applied within the life sciences. But before we can explore the many exciting applications and any potential challenges, we need to examine the nature of data.

Modelling knowledge with data

To fully appreciate the capabilities of multimodal AI in healthcare, it is essential to explore the nature of data, how information is derived from it, and the process of leveraging this information to address relevant problems.

Raw data, in its initial form, often possesses limited value in itself. It becomes significant when contextualized with relevant questions, transforming into valuable information. The true value of this information is derived from the actions informed by it. As data is contextualized and correlated, its value increases substantially. Effectively structuring data is key to unlocking its full potential, facilitating the extraction of significant insights.

Data’s inherent complexity can obscure the valuable information it contains and the relationships within. With its advanced processing capabilities, AI is particularly adept at analyzing and interpreting this complexity. Data represents information in a format suitable for machine processing, namely binary code. Multidimensional sensors, which are essentially complex vectors of binary data, enable the processing of more complex information (see Figure 2). Data models break down intricate domain knowledge into formats accessible to machines, establishing connections and providing context for more effective problem solving. Speaking of which, let’s now turn our attention to the problems multimodal AI can solve within the life sciences.

Multimodal data and AI in life sciences

In pharmaceutical research, multimodal data integration unlocks significant opportunities for deeper clinical and medical data analysis. Professionals can gain improved insights into drug-cell interactions and drug mechanisms by combining various data types (molecular and genetic information, electronic health records, diagnostic images from MRIs and CT scans, and patient interaction recordings). These biomarkers are valuable sources of medical information that can lead to the identification of disease. Additionally, advanced AI technologies incorporate secondary data sources, such as patient reported outcomes (PROs) and real-world data (RWD), covering patient feedback and insurance claims. These rich but unstructured secondary sources are transformed by multimodal AI for more effective analysis, offering insights that closely mirror patients’ real-world experiences, thereby enhancing research applicability.

Previously, data from different sources was analyzed separately, limiting the discovery of correlations and patterns and hindering drug development efficiency. Multimodal AI facilitates integrated analysis, leading to a fuller understanding of drug effects, patient reactions, and treatment outcomes. This marks a shift towards more precise and patient-focused research.

However, integrating diverse data modalities poses challenges, such as handling data heterogeneity, avoiding redundancy, and maintaining patient confidentiality. Multimodal AI faces such issues as data quality variance, alignment difficulties, and the risk of overfitting. As a result, it is vital to implement robust, scalable solutions.

The promise of multimodal AI lies in its ability to merge complex domain knowledge across data types, a boon for the data-centric pharmaceutical industry. Figure 2 illustrates essential multimodal data sources for informed medical decision-making, linking these to use cases that trace the patient journey from prevention to treatment and follow-up. These examples highlight the transformative impact of multimodal data on healthcare outcomes and efficiency.

By harnessing the power of diverse data streams and employing sophisticated analytical techniques, pharmaceutical companies can significantly improve their understanding of drug safety and effectiveness in real-world scenarios. The success of such initiatives, however, hinges on the ability to navigate the complexities of data integration, analysis, and privacy protection, underscoring the importance of robust data management practices and advanced analytical capabilities in the pharmaceutical industry.

To fully appreciate multimodal AI’s transformative potential, let’s take a look at a recent successful initiative led by Bayer Vital GmbH.

The VENTASTEP proof of concept: Industry example²

The VENTASTEP study, conducted by Bayer Vital GmbH, is an example of technology-driven Innovation. It demonstrates how integrating multiple data streams in a clinical environment, with the help of digital tools, can provide valuable insights into treatment impacts, patient adherence, and real-time detection of adverse events. By diving into the different stages of the pharma value chain, we can learn from the lessons of the VENTASTEP study and understand the transformative potential of multimodal data and AI.

The study aimed to evaluate the impact of inhaled Iloprost treatment on patients with pulmonary arterial hypertension (PAH) by integrating multiple data streams in a clinical setting. It utilized digital tools to collect and analyze data and tried to detect and record irregular heart rates with AI.

“VENTASTEP was a proof of concept that underscored the potential of digital technology in enhancing our understanding of patient responses to medication. Integrating multimodal AI could significantly magnify this potential, offering deeper insights and more streamlined processes. Learning from this work is invaluable and paves the way for more structured and financially sustainable research ecosystems in the future.”

Dr. Christian Mueller, Bayer Vital GmbH

To monitor daily physical activity, heart rate, and inhalation behavior, the study utilized smart devices like the Apple Watch Series 2 and iPhone 6s, along with a dedicated app. The Breelib nebulizer facilitated the digital monitoring of inhalation data, providing a comprehensive view of treatment adherence.

The insights and experiences gained from the VENTASTEP study go beyond simple data collection. They provide a first outlook into the potential for collecting multimodal data and detecting adverse events in real time. The possibility of automated reporting to health agencies could lead to better patient safety protocols in the future. Additionally, the study’s findings could spur the development of more efficient and cost-effective digital monitoring systems, which can better engage patients through real-time data feedback. By connecting traditional clinical evaluations to digital monitoring, VENTASTEP has shown the potential of a promising avenue for the industry to move towards a more integrated, data-driven healthcare paradigm.

Final thoughts: The future of multimodal AI in healthcare

In the life science sector, the integration of multimodal data with AI is transforming traditional practices. It leverages varied data types, from genomic information to patient interactions, fostering advancements in drug development and personalized patient care. Multimodal data is driving earlier and better targeted interventions. For example, merging genetic data with patient medical histories allows for the development of targeted therapies, increasing treatment efficacy and customization. This integration also enhances patient communication, leading to more informed healthcare decisions.

The future of AI in healthcare will be a new paradigm of patient care. The application of AI and multimodal data extends from drug discovery, where AI can predict interactions and optimizes compounds, to after-sale services, such as personalized medication apps. These technologies streamline processes, reduce costs, and expedite treatments from the lab to the patient. However, every new solution comes with its own challenges. In the case of multimodal AI, it is important to correctly manage data availability, privacy, and regulatory compliance.

While many of the applications for multimodal AI are technically feasible, the availability of suitable data is the overshadowing obstacle. Additionally, the need for high computational power and specialized infrastructure for processing complex datasets is essential. At Capgemini Invent, we believe end-to-end, holistic implementation of AI models in healthcare is the future of life sciences. Multimodal databases will transform entire R&D departments (e.g., in-silico trials). Digital twins of complicated systems (e.g., bioreactors) will reveal new insights. And AI-powered internal knowledge management tools will facilitate rapid and accurate access to quantitative and qualitative research data. As a result, life sciences professionals will be able to pursue exciting new avenues for both business growth and societal impact.

Reach out for support

Capgemini optimizes the performances of processes across the entire value chain. Our life science teams support clients with an end-to-end approach that incorporates cutting-edge technologies and forward-thinking methodologies. To reinvent your drug discovery, clinical trials, patient care, and personalized medicine, check out our Generative AI Strategy or reach out to one of our experts for support.