VR Piano Learning Platform with Leap Motion and Pressure Sensors

Aiming at helping novices to learn and practice playing piano without a real piano teacher, we propose a virtual reality (VR) piano system which can record the user's piano fingering data. Our system uses two Leap Motion sensors for wide-range hand tracking, and we developed a method to calibrate and compose the tracking results of two Leap Motion sensors. However, Leap Motion sensors fail to accurately detect whether a finger presses a key because the precision of the estimated depth is not high enough. Hence, we employ multiple pressure sensors attached on the user's fingertips to more accurately check whether the user is pressing the piano key or not. An HTC Vive headset is used to display the immersive VR environment while the user practices. Moreover, a transparent acrylic board is mounted above the Leap Motion sensors so that the player can directly play on the board, which enhances the immersion and presence by touching a physical object. The experimental results show that our VR piano system provides users a great piano-playing experience and is helpful for piano learning.