Edge AI that packs a punch

“I fight for perfection”

– Mike Tyson

“Do you achieve it?”

– Charlie Rose

“Nah! No one does, but we aim for it…”

– Mike Tyson

Put simply, deep learning is a type of machine learning that aims to mimic the way the human brain works to recognize patterns and make decisions. Deep learning on mobile and edge devices, like smart wearables with limited computational resources, can be challenging. However, running machine learning models on edge devices and keeping the data local has advantages for data privacy, sustainability and latency.

With one of our staff a keen boxer, we took on the challenge of working out what types of punches were being thrown with the constraints of:

• A need for real-time feedback
• Minimal hardware
• Low powered devices

How does it work?

Capgemini developed an end-to-end demonstrator, where data was collected using a custom iPhone and Apple Watch application, allowing sensor readings to be automatically labeled. A hybrid deep learning model was built to accurately classify each punch type. The model was converted to an optimal form to run efficiently on low powered edge devices.

The demo streams raw sensor data from the watch to display on a screen, along with the punch classifications, as close to real time as the network connectivity allows. In this case, we’re using a single Apple Watch on the left wrist to generate sensor data (specifically accelerometer, gyroscope and orientation data). This data is sent to an AI model that determines which types of punches are being thrown by either hand.

When we demonstrate this to the attendees at various events, the reaction is often “Wait – how is that even possible?!”. Indeed, how do you know what the right hand is doing with a watch strapped to the left wrist? While it’s certainly easier to track the lefts, boxing is a whole-body sport – there are small but characteristic movements that happen in response to throwing a punch. These movements are what the ML model detects, and uses to classify the type of punch.

In a world where seemingly everything is smart and sensors are everywhere, this may seem like an artificial constraint, but often you can’t put sensors right where you want them – for example, due a harsh industrial environment or because they would be too cumbersome or intrusive for the wearer.

To run on low powered edge devices and give near real time results, the machine learning model had to be as lightweight and efficient as possible. Through a careful selection of model architecture and the type of optimization techniques discussed in this blog post on model optimization, we produced an AI model that could interface in real time on a low end smartphone. 

Applications

Our technology has potential applications across many domains. For example, you could extend this application to give feedback to a novice boxer and help them avoid common mistakes – the minimal hardware would make this a very portable solution. For patient monitoring, perhaps when managing chronic illness, ensuring privacy is crucial and minimizing the number of sensors could help ensure compliance and guarantee that solutions are minimally invasive and cost effective.

In industrial settings, being able to classify in real time close to where data is being generated allows for rapid intervention. It can also help to reduce the cost and associated carbon emissions from the transfer of large volumes of data.

In disconnected applications, like drone operation in remote locations, this approach can be used to improve autonomy, for example allowing these drones to locate a safe place to land in an emergency, through on-board real-time video analysis.

Conclusion

Capgemini is exploring the applications of this technology in sports equipment, but we believe that this is just the beginning; this technology certainly isn’t limited to the complex movements of the ‘sweet science’. Whether based on incoming sensor, video or other data, being able to analyze inputs in real time on low powered devices has a crucial part to play in unleashing the potential of intelligent products and services.

Want to see in action? Watch our boxing demo video

Interested in finding out more? Take a look at our Intelligent Products and Services offer and follow David on LinkedIn.