A research consortium around the Cologne Game Lab at TH Köln is developing a learning environment with artificial intelligence to support the independent learning of psychomotor skills.
MILKI-PSY aims to create AI-powered, data-rich, multimodal, immersive learning environments for independent learning of psychomotor skills. In doing so, a cross-domain approach is emerging that enables multimodal recording of expert activities and using these recordings as blueprints for learners. With the help of AI-supported analyses, learning progress is to be supported through automated error detection and generated, individual feedback. This creates holistic, innovative learning environments for learning psychomotor skills, in which personalised, AI-supported learning support enables individual learning processes based on complex data analyses.
The COVID-19 pandemic has shown that many teaching/learning activities can be carried out without physical presence. This is hardly true for psychomotor skills: their development, as they are necessary in many disciplines (e.g. medicine, engineering, chemistry, artistic activities, sports) requires hands-on practice, direct feedback and reflection. In order to achieve the desired learning successes, personnel supervision and material input are therefore indispensable. Both increase costs and limit the scaling possibilities of the study programmes concerned: experts are rare and expensive, and the use of materials causes further costs.
Anyone who wants to learn a sport trains new psychomotor skills. Until now, this has required role models such as teachers on site who explain, demonstrate and assess certain procedures. As part of the joint research project MILKI-PSY, a consortium led by the Cologne Game Lab at TH Köln is developing a learning environment with artificial intelligence (AI) to support the training process.
Artistic, athletic and manual activities require psychomotor skills in which physical movements are linked to mental processes. “Learning these requires hands-on practice, direct feedback, and reflection. Everyone is familiar with video tutorials, but they lack any feedback,” explains project leader Prof. Dr. Roland Klemke from the Cologne Game Lab.
The research project “Multimodal Immersive Learning with Artificial Intelligence for Psychomotor Skills” (MILKI-PSY) designs an innovative environment for independent learning of psychomotor skills. For this purpose, the correct movement sequences are recorded by trainers using cameras and sensors. A virtual avatar generated from this recording will then serve as a model for the learners. This can be displayed on a large screen, in an augmented or virtual reality environment, for example. With the help of artificial intelligence and automated error detection, the learning progress is analyzed and individual feedback is generated.
“The learning environment will be created for basic sequences and fundamental movements of sports. There, it can be a useful complement to the trainer by allowing skills to be practiced and immediate feedback to be provided with the highest precision in real time. In this way, it will be easier for dancers, for example, to improve their psychomotor skills and adapt their own movement to the optimized movement of the expert,” says Klemke. In collaboration with the Institute for Product Development and Design Technology (IPK) at TH Köln, the approach is also being investigated for psychomotor training in the field of human-robot interaction.
In the project, the Cologne Game Lab is responsible for constructing the immersive environment with which users will interact. Immersion refers to diving into another world. “Using technologies from video games, modern instruments from sensor technology and artificial intelligence, we want to test the extent to which independent learning of psychomotor skills can be better supported,” says Klemke.