Google cloud platform & SAP datasphere: A powerful partnership for workers safety monitoring

Our Hack2Build use-case started from a willingness to focus on sustainability. This time, not especially from an ecological nor economical point of view, but more from social and human aspect, by focusing on safety of employees.

Reports from the European Agency for Safety and Health at work (EU-OSHA) highlight the high number of accidents in production companies in Europe. Occupational accidents reached 3.2 million in 2022. The most common causes of incidents in production companies relate to fall, contacts with dangerous objects or exposure to chemicals. What’s more, the BDO Industrial Accident Management Barometer reveals that the average cost of an industrial accident in the manufacturing sector is estimated at €32,000 including worker’s compensation and lost productivity. Beyond the financial and economic aspects, these accidents seem to have a real human cost.

These statistics demonstrate that workplace accidents in manufacturing companies are a major problem. Efficient preventive measures are needed to reduce the number of accidents and protect the health and safety of employees. Most existing measures require a high manual intervention to monitor health and safety in plants. Hence, the goal of this use case is to automate the whole safety process, using the power of Intelligent connected infrastructures, where AI, cloud and analytics are used.

Workers’ safety: A cloud-based solution architecture for a real world challenge

A manufacturing company gathers continuous snapshots taken by a camera in a production line through a live video stream. Snapshots include people wearing their full equipment, other people having only a helmet, or a mask with no glove, etc. Videos and pictures captured in this plant will be fetched through AI algorithms. The Personal Protective Equipment Detector (PPE) model from Google Cloud Platform is used to verify the presence of required equipment. The model output is returned in BigQuery, which will then be fetched in Datasphere using the Google BigQuery connector. Injected data will be enriched in Datasphere with HR master data. On top of that, a compliance report is foreseen in SAP Analytics Cloud (SAC), so HR or compliance service can take actions and keep an eye on the health and safety data.

PPE model on Google Cloud Platform: AI-Powered insights

Google Cloud Platform (GCP) offers a variety of tools and services that can help automate worker safety processes in manufacturing companies. In the context of this Hack2Build event, we have worked with the Vertex AI Vision Personal Protective Equipment (PPE) Detector model. This model can be used to identify workers who are not wearing the correct equipment, monitor compliance with PPE requirements and provide workers with real-time feedback on their PPE usage.

The input stream to this pre-trained model is a collection of images or videos of workers in a manufacturing company. The model identifies the workers and detects the PPE items they are wearing. In the parameters section of the model, you can select equipments you want to detect. You can choose from either head, face or hand covering. For this H2B usecase, we go for a full PPE detection, including all three items.

The output of this model is stored in a BigQuery table. The model outputs an annotation field in JSON format. It provides information about the identified person, the confidence score, the PPE detected, etc. This table will then be injected into SAP Datasphere for further modeling and enrichment.

We are aware that the model used has its limitations. It is only applicable to the production sector and can only detect 3 specific types of equipment. To extend this use case, but also to enhance the scenario to further cases, we recommend a combination of standard and customized models and algorithms in GCP.

Enriching PPE model output to create compliance reports

Data from google BigQuery is made available in sap datasphere via virtual tables with no need for replication. Only few steps were required to create the connection and make data virtually available. This highlights the recent partnership between SAP and google to harness the power of data and AI.

The virtual table is used in a view and then in stored procedure for some JSON conversion in HANA cloud. Indeed, although data is usually being stored in tables, with columns and rows, where for each column a strict definition or schema defines layout and data types, part of the output provided by the PPE model is stored as a JSON document. Hence, Conversion is needed to query the JSON text and present it as a flat table. We came up with an innovative solution to parse the JSON field into a flat table. For this proof of concept, JSON conversion has been performed through the Open SQL schema by means of the database explorer, since document store is not enabled in SAP Datasphere. After ingesting the JSON field as a large object into the Open SQL schema, we use table function JSON_TABLE to normalize the JSON response. The interesting point here, is that we have done all developments with SAP HANA Cloud or SAP Datasphere. No other tool nor coding with python was needed here.

Next, PPE data is enriched with some HR master data in the data layer. Since we had no access to a S4/HANA system, we decided to combine the generated parsed table with HR standard content available in the SAP Marketplace. The downloaded content provides a list of all employees currently working in the company. It comes with some dimension and text views. As a next step, we created graphical views on top of the PPE parsed table and HR master data.

Finally, models are consumed in SAP Analytics Cloud where insightful dashboards are built. The model output has been used to create custom visualizations, that can be used to analyze worker safety data. The report gives an overview of the most worn equipment. For each equipment, KPIs indicate whether it is being worn correctly. We can see here that, although people wear their equipment, in over 40% of cases the equipment is poorly worn. This could be a helmet worn sideways, a mask worn too low, or a glove missing.

In further steps, we want to create alerts, that would be sent to the appropriate personnel so that they can take real-time actions.

To conclude, we have clearly shown that SAP and Google Cloud customers can use Vertex AI together with SAP Datasphere to create AI solutions that accelerate companies’ sustainability programs. As a result, businesses will be able to create sustainability reports, automate AI-powered sustainability alerts and take real-time actions with deeper analyses on social and human impact. This specific use cases focuses on the manufacturing sector, but it could easily be broadened to retail or medical sector for instance.

As a team of four we got the chance to participate in a unique and rewarding experience. Together with some solution architects, software engineers and data modelers, we worked hard to create an MVP to automate workers’ safety. During the 5-days Hack2Build, we really pushed ourselves to the limit, learning new technologies on the fly, making quick decisions. We truly enjoyed a sense of community, where everyone could learn from others. The positive energy led us to a first, rewarding, place. We are looking forward to work towards a final solution build and go-to-market.

Capgemini is leading the world in SAP knowledge. Click below to learn more.