HCM City University of Technology students make hand rehabilitation game

The video game was developed by a group of students, using artificial intelligence to recognize hand gestures and support rehabilitation for this part.

The research group (BK Hand Rehab), Department of Computer Science, Ho Chi Minh City University of Technology started building a hand gesture recognition system for video games from August 2022. The group gradually upgraded the system into a computer game to help restore and treat hand function.

Vo Ngoc Sang, a member of the research team, said that BK Hand Rehab's games are being developed to focus on supporting the rehabilitation of the wrist, hand and fingers.

The game is designed and operated through a computer and a Leap Motion Controller sensor device. Initially, players are required to perform hand gestures according to the instructions in each game. The sensor is capable of receiving and recognizing information about the location of certain points on the hand. From there, the team uses artificial intelligence (AI) algorithms to generate corresponding hand shapes and gestures, creating actions in the game.

Students design video game to restore hand function — The game helps users have flexible hand reflexes. Video: *Research team*

The game system is designed with a hierarchical structure by level, meaning that for each game, players will experience from easy to difficult levels, creating more challenging experiences. The game is suitable for many different subjects such as the elderly, disabled or those with hand injuries who are in the physical therapy stage. Thanks to hand gesture recognition to interact with the game, patients undergoing physical therapy can practice and relax to create a comfortable feeling.

Typically, patients who need to recover hand function will practice simple mechanical exercises, or have the support of nurses or doctors. The team's solution is based on video games, making the recovery process more interesting and less boring.

Currently, BK Hand Rehab offers 3 basic games. Players can practice through static games (choose shapes and colors) or dynamic games (move objects to catch falling items or control to avoid obstacles).

The games are developed by the team following a five-step process: ideation, design, refinement, testing, and deployment. As for the hand gestures selected for each game, the team consults research from prestigious organizations such as Harvard University and the Scottish Government's National Health Service.

To practice through games, users need a hand stand, a computer, and a Leap Motion Controller sensor. Users are guided through specific movements depending on the game genre and based on the situation, they will decide which movements to perform to interact with the corresponding context.

Users experience the game in real life. Photo: Research team — Users experience the game in real life. Photo: *Research team*

Some users who experience the product find the exercises in the form of games to be compact, comfortable, easy to install and use. The team plans to research other methods to increase the accuracy of hand gesture recognition and more difficult movements. At the same time, the games need to be improved in terms of graphics and mechanics to add to the user experience.

Associate Professor, Dr. Quan Thanh Tho, Deputy Head of the Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (VNU-HCM), instructor, assessed that BK Hand Rehab offers an outstanding solution compared to traditional hand therapy methods, thanks to creating an enjoyable recovery experience, reducing boredom, helping to save costs and not requiring patients to travel to medical facilities.

Associate Professor Tho said the project has great potential, although the use of video games to support hand rehabilitation is not a new idea, and has been mentioned and realized in many previous studies. However, previous studies still have many limitations such as choosing inappropriate movements for the game, and the hand gesture recognition step has not reached the expected accuracy and speed. "BK Hand Rehab has solved many problems that previous studies encountered," he said.

He also added that the game needs to be adjusted to suit patients through real-world user feedback. Second, the team needs to get validation from physical therapists about the effectiveness of the solution. “These are the areas that need to be improved,” he said, adding that he is working with the Department of Computer Science and Engineering to help the team connect with medical centers and orthopedic hospitals to conduct the necessary research.

The Hand Rehab project recently won first prize in the 2023 Community Initiative Competition organized by the non-profit organization SonTa Foundation.