Zum Inhalt springen

MILKI PSY Prototypes

The MILKI-PSY consortium has developed a range of research prototypes, including the MILKI-PSY Cloud, an ML-based cloud solution for processing multimodal sensor streams and delivering psychomotor feedback. The IMPECT framework serves as an immersive training environment for various psychomotor skills like dance and exercise, featuring a virtual teacher, feedback cards, and a cloud server for session management. Another prototype is a multiplayer exergame that tracks body movements in real-time using Human Pose Estimation technology. We’ve also developed visual feedback systems using AR and XR, such as a virtual golf club for real-time skill comparison and training, as well as an augmented reality application that supports human-robot collaboration in assembly tasks. Additionally, we’ve researched methods like prototype networks and attention mechanisms for few-shot learning to identify and evaluate movement discrepancies in student training sessions.

MILKI PSY Cloud

Partner / Lead: RWTH

MILKI-PSY Cloud is a collaborative ML-based cloud solution for processing multimodal sensor streams and delivering feedback in psychomotor learning scenarios. Structured in three phases — data ingestion, machine learning, and data exploitation — it applies MLOps approaches to provide both direct and long-term feedback.

Read More (doi, ref, URL, …)

Slupczynski, M. P., & Klamma, R. (2021) Vorschlag einer Infrastruktur zur verteilten Datenanalyse von multimodalen psychomotorischen Lerneraktivitäten. Technical Report. http://dx.doi.org/10.13140/RG.2.2.23664.58886/1

Slupczynski, M. P., & Klamma, R. (2021). MILKI-PSY Cloud : Facilitating multimodal learning analytics by explainable AI and blockchain. MILeS 2021: Multimodal Immersive Learning Systems 2021: Proceedings of the 1st International Workshop on Multimodal Immersive Learning Systems (MILeS 2021) at the 16th European Conference on Technology Enhanced Learning (EC-TEL 2021): Bozen-Bolzano, Italy, pages 22-28. https://doi.org/10.18154/RWTH-2022-01582

Slupczynski, M. P., & Klamma, R. (2022). MILKI-PSY Cloud: MLOps-based Multimodal Sensor Stream Processing Pipeline for Learning Analytics in Psychomotor Education. MILeS 2022: Multimodal Immersive Learning Systems 2022: Proceedings of the 2nd International Workshop on Multimodal Immersive Learning Systems (MILeS 2022) at the 17th European Conference on Technology Enhanced Learning (EC-TEL 2022): Toulouse, France, pages 8-14. https://doi.org/10.18154/RWTH-2022-09814

Immersive Multimodal Psychomotor Environments for
Competence Training (IMPECT)

Partner / Lead: CGL

A training toolkit designed for psychomotor skills training, consisting of feedback and instructional components. It includes the IMPECT front-end, a client application that serves as an immersive learning environment for training psychomotor skills, and the IMPECT-Server, a cloud server used to navigate learning sessions and generate feedback cards for the client application. Multiple front-end client applications can connect to the IMPECT-Server to support a range of psychomotor skills (e.g., dance, exercise, human-robot interaction, etc.)

Read More (doi, ref, URL, …)

Slupczynski, M. P., Mat Sanusi, K. A., Majonica, D., Klemke, R., Decker, S. (2022). Implementing Cloud-Based Feedback to Facilitate Scalable Psychomotor Skills Acquisition. MILeS 2023: Multimodal Immersive Learning Systems 2023: Proceedings of the 3rd International Workshop on Multimodal Immersive Learning Systems (MILeS 2023) at the 18th European Conference on Technology Enhanced Learning (EC-TEL 2023): Toulouse, France, pages 36-43. https://doi.org/10.18154/RWTH-2023-09769

Mat Sanusi, K. A., Majonica, D., Handwerk, P., Biswas, S., & Klemke, R. (2023). Evaluating an immersive learning toolkit for training psychomotor skills in the fields of human-robot interaction and dance. In MILeS@ EC-TEL (pp. 70-78).

Dynamic Movement Primitives

Partner / Lead: DFKI

DFKI investigated and implemented several methods to efficiently represent and predict human motion. We outlined a machine learning supported approach to provide feedback. For the walking scenario, we provided an evaluation of how movements can be compared to show deviations between student and expert movement. DFKI also provided initial software modules for motion representation and motion comparison for the partners to support prototype development.

Read More (doi, ref, URL, …)

Paaßen, B., & Kravcik, M. (2021). Teaching psychomotor skills using machine learning for error detection. In Proc. of the First International Workshop on Multimodal Immersive Learning Systems (MILeS 2021) at the 16th European Conference on Technology Enhanced Learning: Technology-Enhanced Learning for a Free, Safe, and Sustainable World (EC-TEL 2021). CEUR Vol-2979.

Few-shot keypose detection

Partner / Lead: DFKI

We investigated methods to automatically recognize student trials for poses with only a single correct teacher demonstration. We investigated relevance learning, prototype networks, and attentional mechanisms to achieve a robust few-shot approach that generalizes to all students. In an experiment with one teacher and 27 students performing a sequence of fitness and dance movement elements, we showed that prototype networks work best in combination with an attention mechanism.

Read More (doi, ref, URL, …)

Paaßen, B., Baumgartner, T., Geisen, M., Riedl, N., & Kravčík, M. (2022). Few-shot Keypose Detection for Learning of Psychomotor Skills. In Proc. of the Second International Workshop on Multimodal Immersive Learning Systems (MILeS 2022) at the 17th European Conference on Technology Enhanced Learning – Addressing Global Challenges and Quality Education (EC-TEL 2022). CEUR Vol-3247.

Motion visualization techniques

Partner / Lead: RWTH

This application investigates and researches comparative visualization techniques for human movements, providing different visualization types to display body movements and error information. It supports showing one action at a time while considering changes over time to analyze factors influencing motion visualization techniques and aid in movement correction.

Read More (doi, ref, URL, …)

Shenyao Li (2024). From Still to Dynamic: A Comparative Analysis of Visualization Techniques for Human Movements. Bachelor Thesis, RWTH Aachen University. Supervisors: Prof. Dr. Stefan Decker, and Prof. Dr. Roland Klemke. Advisors: Michal Slupczynski, and Khaleel Asyraaf Mat Sanusi. April 2024

Fitness Trainer

Partner / Lead: DFKI

We present a system for monitoring exercise performance and immediate feedback on posture during training. This real-time guidance facilitates self-correction and boosts motivation, especially without professional supervision. Fitsight is a reliable and sophisticated tool for self-guided fitness journeys.

Read More (doi, ref, URL, …)

Kotte, H. Kravčík, M., & Duong-Trung, N. (2023). Real-Time Posture Correction in Gym Exercises: A Computer Vision-Based Approach for Performance Analysis, Error Classification and Feedback. In Proc. of the Third International Workshop on Multimodal Immersive Learning Systems (MILeS 2023) at the 18th European Conference on Technology Enhanced Learning (EC-TEL 2023). CEUR Vol-3499.

Duong-Trung, N., Kotte, H., Kravčík, M. (2023). Augmented Intelligence in Tutoring Systems: A Case Study in Real-Time Pose Tracking to Enhance the Self-learning of Fitness Exercises. In: Viberg, O., Jivet, I., Muñoz-Merino, P., Perifanou, M., Papathoma, T. (eds) Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. https://doi.org/10.1007/978-3-031-42682-7_65

Kotte, H., Daiber, F., Kravcik, M., & Duong-Trung, N. (2024, June). FitSight: Tracking and Feedback Engine for Personalized Fitness Training. In Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization (pp. 223-231). https://doi.org/10.1145/3627043.3659547

Samanta, A., Kotte, H., Handwerk, P., Asyraaf Mat Sanusi, K., Geisen, M., Kravcik, M., & Duong-Trung, N. (2024, June). IMPECT-POSE: A Complete Front-end and Back-end Architecture for Pose Tracking and Feedback. In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization (pp. 142-147). https://doi.org/10.1145/3631700.3664865

Motion skeleton superimposition in dancing/fitness

Partner / Lead: DSHS

Visualisation method to provide real-time feedback for learning a movement sequence (e.g., in the field of dance and fitness). The targeted performance (teacher) is virtually and proportionally superimposed with the actual performance (learner) via a large screen during learning.

Read More (doi, ref, URL, …)

Geisen, M., Baumgartner, T., Riedl, N., & Klatt, S. (under review). Innovative motor learning with real-time feedback for video-based training [Manuscript submitted for publication in Educational Technology Research and Development]. Institute of Exercise Training and Sport Informatics, German Sport University Cologne.

Psychomotor Feedback Engine

Partner / Lead: RWTH

A rule-driven feedback engine for psychomotor learning to create rules input by domain experts to provide real-time, personalised feedback. The system dynamically applies these rules based on continuous analysis of user movements, timing, and accuracy, without relying on hardcoded rules or labelled datasets.

Read More (doi, ref, URL, …)

Slupczynski, M., Nekhviadovich, A., Duong-Trung, N., & Decker, S. (2024). Analyzing Exercise Repetitions: YOLOv8-enhanced Dynamic Time Warping Approach on InfiniteRep Dataset. In iWOAR 2024 – 9th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence.

Feedback/Error Prioritisation

Partner / Lead: DIPF

An implementation with REST API to set up and schedule feedback pipelines based on preset priorities.

Read More (doi, ref, URL, …)

[WIP] Dot sensor application

Partner / Lead: DIPF

Mobile Application that uses human keypoint estimation and senors to implement velocity based training method for psychomotor skill improvement

Read More (doi, ref, URL, …)

GolfPutt AR Tool

Partner / Lead: DSHS

Virtual golf club as a role model for learning the golf putt. It is superimposed with the real golf club of the learner in real-time via Microsoft HoloLens glasses.

Read More (doi, ref, URL, …)

Geisen, M., Nicklas, A., Baumgartner, T., & Klatt, S. (2023). Extended Reality as a training approach for visual real-time feedback in golf. IEEE Transactions on Learning Technologies, 17, 642-652. https://doi.org/10.1109/TLT.2023.3322660

IMPECT robots escape room

Partner / Lead: CGL

This study investigates human-robot interaction within an educational setting, emphasizing trust and collaborative decision-making facilitated through augmented reality (AR) technology. Participants were tasked with repairing robot components and navigating a game-like environment with an AR device. The study aimed to understand how trust is established and maintained in interactions with autonomous robots.

Read More (doi, ref, URL, …)

Extended Abstract published in EduRobotX Workshop of EC-TEL24
Not published yet. There will be a full publication extending on this

Majonica, D., Fanchamps, N., Iren, D., & Klemke, R. (2024)

IMPECT-Dance – Wizard of Oz

Partner / Lead: CGL

One of IMPECT’s front-end client applications, an immersive learning environment for a full-dance routine consisting of five exercises with a Wizard of Oz study where the expert navigates sessions using the IMPECT’s backend platform locally. It features multimodal feedback delivered through feedback cards, a virtual teacher with recorded movements from a motion capture suit, and a live camera view using a webcam

Read More (doi, ref, URL, …)

Mat Sanusi, K. A., Majonica, D., Handwerk, P., Biswas, S., & Klemke, R. (2023). Evaluating an immersive learning toolkit for training psychomotor skills in the fields of human-robot interaction and dance. In MILeS@ EC-TEL (pp. 70-78).

Space Striker: Exercise Odyssey

Partner / Lead: RWTH

This prototype application is a multiplayer exergame where two players cooperate to control a spaceship through body movements, navigating an asteroid field and escaping enemies. The game uses Human Pose Estimation (HPE) technology to track body movements in real-time, requiring no specialized hardware.

Read More (doi, ref, URL, …)

Yorck Degens (2024). Enhancing Physical Activity and Social Interaction through Peer-assisted Exergames. Bachelor Thesis, RWTH Aachen University. Supervisors: Prof. Dr. Stefan Decker, and Prof. Dr. Roland Klemke. Advisors: Michal Slupczynski, and Khaleel Asyraaf Mat Sanusi. July 2024

Mathkinetics

Partner / Lead: DIPF

A video game that employs pose and speech recognition, challenging users to navigate obstacles while solving arithmetic problems.

Read More (doi, ref, URL, …)

Scarcella, D., Schneider, J., Kiesler, N., & Schiffner, D. (2024). Mathkinetics: Solving Arithmetics While Running out of Breath. Proceedings of the 16th International Conference on Computer Supported Education, 250–256. https://doi.org/10.5220/0012536900003693

Motion anticipation in basketball

Partner / Lead: DSHS

Superimposed videos are used to help learners anticipate shots and fakes in basketball. In these videos, a superimposed shot and fake are shown, stopping just before the ball is released or at the peak of the fake to conceal the outcome. After viewing, learners identify whether the player in a specific coloured shirt performed a shot or a fake. The superimposed motions aim to facilitate easier comparison and identification of differences between the actions.

Read More (doi, ref, URL, …)

Bernhardt, F., Riedl, N., Geisen, M., Meyer, J., & Klatt, S. (in preparation). Difficulty in distinguishing motion differences between shots and shot fakes in basketball: An exploration using an innovative approach.

Golfpitch visual and tactile feedback tool

Partner / Lead: DSHS

A novel learning method for golf pitch training includes augmented visual information through XR. A virtual hand and wrist of an expert is shown in real-time through Microsoft HoloLens glasses. Learners try to follow the optimized wrist motion both in slow motion and in normal speed.

Read More (doi, ref, URL, …)

Geisen, M., Braun, T., Mat Sanusi, K. A., & Klatt, S. (in preparation). Augmented visual information through Extended Reality: A novel method for golf pitch training.

XSens Motion comparison (DTW/Kabsch)

Partner / Lead: DSHS / RWTH

A novel approach for identifying motion differences in sports training using sensor suit data to visually compare joint positions as skeletons against reference values. By calculating root-mean-square deviation (RMSD), it quantifies differences in body positions. After manually aligning recordings at key motion points, the Kabsch algorithm adjusts orientation and translation to minimize RMSD. Frame-by-frame analysis of minimal RMSD reveals motion dissimilarities.

Read More (doi, ref, URL, …)

Geisen, M., Seifriz, F., Fasold, F., Slupczynski, M., & Klatt, S. (2024). A novel approach to sensor-based motion analysis for sports: Piloting the Kabsch algorithm in volleyball and handball. IEEE Sensors Journal. https://doi.org/10.1109/JSEN.2024.3455173

Human ↹ Robot object handover

Partner / Lead: IPK

An approach to explore how a robot nonverbally communicates motion intentions in a collaborative work environment. The proposed setup involves enhancing a collaborative assembly cell, featuring the two-armed robot Yumi, with the use of Microsoft HoloLens.

Read More (doi, ref, URL, …)

Keller, T., Götze, F., Varney, V., & Richert, A. AR robot motion intent communication in collaborative assembly: Presentation of a study concept. In 5th International Workshop on Virtual, Augmented, and Mixed Reality for HRI (2022).

Collaborative assembly with robots in AR/VR

Partner / Lead: IPK

An approach for designing a framework for AR-based and VR-based training systems aimed at acquiring psychomotor skills in a collaborative assembly scenario.

Read More (doi, ref, URL, …)

Keller, T., Varney, V., & Richert, A. (2021). WIP: Development of a design framework for the provision of multimodal content in an AR-based training system for the acquisition of psychomotor skills. In MILeS@ EC-TEL (pp. 37-45).

Hololens Hand Capture

Partner / Lead: IPK

An extended version of the AR application for the handover task now includes hand tracking. The goal is to transmit the human hand’s coordinates to the robot controller, enabling the robot to synchronize its end effector accordingly.

Read More (doi, ref, URL, …)

IMPECT-Sports – Wizard of Oz
(Exercise routine)

Partner / Lead: CGL

This prototype utilizes IMPECT components, an immersive learning environment for basic exercise movements where a Wizard of Oz study was conducted to allow the teacher to navigate sessions from the local machine by triggering multimodal feedback (e.g., visual, auditory). It features a virtual teacher with recorded movements using a motion capture suit and real-time motion recording of participants via Kinect.

Read More (doi, ref, URL, …)

Mat Sanusi, K. A., Slupczynski, M., Geisen, M., Iren, D., Klamma, R., Klatt, S., & Klemke, R. (2022, October). IMPECT-Sports: Using an Immersive Learning System to Facilitate the Psychomotor Skills Acquisition Process. In MILeS@ EC-TEL (pp. 34-39).

Edutex

Partner / Lead: DIPF

The Edutex software infrastructure allows learning analytics stakeholders to leverage data related to learners‘ physical contexts. Edutex accomplishes this by utilizing sensor data from smartphones and smartwatches, along with response data from experience samples and questionnaires collected via learners‘ smartwatches.

Read More (doi, ref, URL, …)

Ciordas-Hertel, G.-P., Rödling, S., Schneider, J., Di Mitri, D., Weidlich, J., & Drachsler, H. (2021). Mobile Sensing with Smart Wearables of the Physical Context of Distance Learning Students to Consider Its Effects on Learning. Sensors, 21(19), 6649. https://doi.org/10.3390/s21196649

IMPECT robots for
assembly and handover

Partner / Lead: CGL

This study explores immersive learning in the field of Human-Robot Interaction (HRI). We examine four categories of HRI cases: robot-led, human-led, autonomous, and collaborative. For each case, we evaluate the potential risks and human learning outcomes. Our findings include interaction styles that can be applied in HRI as well as the directionality and (multi)modality of these interaction styles.

Read More (doi, ref, URL, …)

Majonica, D., Fanchamps, N., Iren, D., Klemke, R. (2024). Exploring Immersive Learning Environments in Human-Robot Interaction Use Cases. In: Dondio, P., et al. Games and Learning Alliance. GALA 2023. Lecture Notes in Computer Science, vol 14475. Springer, Cham. https://doi.org/10.1007/978-3-031-49065-1_26

3D Pose Estimation in Sports Broadcast

Partner / Lead: DFKI/DSHS

DFKI and DSHS have proposed a method to estimate the global geometry of athletics stadium images using track boundaries. By back-projecting estimated 3D skeletons into the image using this global geometry, we were able to show that current state-of-the-art 3D methods for estimating human posture are not (yet) accurate enough to be used in kinematics research.

Read More (doi, ref, URL, …)

Baumgartner, T., Paassen, B. & Klatt, S. Extracting spatial knowledge from track and field broadcasts for monocular 3D human pose estimation. Sci Rep 13, 14031 (2023). https://doi.org/10.1038/s41598-023-41142-0

Pilates Correction

Partner / Lead: DIPF

A game which enables users to perform a basic Pilates exercise while playing, with their score determined by how accurately they execute the exercise.

Read More (doi, ref, URL, …)

Meik, A., Schneider, J., & Schiffner, D. (2021). Get your back straight! Learn Pilates with the Pilates Correction Game. DELFI 2021, Die 19. Fachtagung Bildungstechnologien Der Gesellschaft Für Informatik e.V.
https://dl.gi.de/items/472f9a00-b1db-45b1-b3c2-d47812259dde

Mental Training: Running VR

Partner / Lead: DIPF

A Cognitive Behavioral-influenced VR application that integrates two key mental aspects of running—strategy and motivation—in the context of long-distance races. The application, designed for head-mounted displays, allows users to simulate typical racing scenarios without the need for physical running.

Read More (doi, ref, URL, …)

Cardenas Hernandez, F. P., Schneider, J., Di Mitri, & Drachsler, H. (Under review). On-your Marks, ready? Exploring User Experience in VR-Cognitive-Behavioral Training. IEEE Transactions on Learning Technologies.

Yu & Mi

Partner / Lead: CGL

An AR application for training human-robot interaction, featuring a step-by-step assembly case completed in collaboration with a virtual robot. Instructional and feedback components are provided by the virtual robot to assist the user throughout the assembly process.

Read More (doi, ref, URL, …)

Mat Sanusi, K. A., Majonica, D., Künz, L., & Klemke, R. (2021, October). Immersive training environments for psychomotor skills development: A student driven prototype development approach. In Multimodal Immersive Learning Systems 2021 (pp. 53-58). CEUR.

Mat Sanusi, K. A., Iren, D., & Klemke, R. (2022, November). Experts’ evaluation of a proposed taxonomy for immersive learning systems. In International Conference on Games and Learning Alliance (pp. 247-257). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-22124-8_24

Mat Sanusi, K. A., Majonica, D., Iren, D., Fanchamps, N., & Klemke, R. (2024). MILSDeM: Guiding immersive learning system development and taxonomy evaluation. Education and Information Technologies, 1-34. https://doi.org/10.1007/s10639-024-12479-4 

Flowmotion
(Yoga Training Environment)

Partner / Lead: CGL

An immersive learning environment designed to teach basic Yoga postures, featuring a camera-based system that tracks the user’s skeleton using ThreeDPose UnityBarracuda for 3D pose estimation. A virtual teacher demonstrates the postures, with corrective feedback provided for errors and positive feedback given as score points.

Read More (doi, ref, URL, …)

Mat Sanusi, K. A., Majonica, D., Künz, L., & Klemke, R. (2021, October). Immersive training environments for psychomotor skills development: A student driven prototype development approach. In Multimodal Immersive Learning Systems 2021 (pp. 53-58). CEUR.

Mat Sanusi, K. A., Iren, D., & Klemke, R. (2022, November). Experts’ evaluation of a proposed taxonomy for immersive learning systems. In International Conference on Games and Learning Alliance (pp. 247-257). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-22124-8_24

Mat Sanusi, K. A., Majonica, D., Iren, D., Fanchamps, N., & Klemke, R. (2024). MILSDeM: Guiding immersive learning system development and taxonomy evaluation. Education and Information Technologies, 1-34. https://doi.org/10.1007/s10639-024-12479-4