Historical context: This article was originally published in China on July 6, 2020. The features, usage figures, comparisons and invitation below reflect the Violy team’s account at that time.
1
Most parents who have supervised a child’s instrument practice will recognize these experiences:
- Learning an instrument feels mysterious: you only find out whether something is wrong with practice when you visit the teacher.
- Practice is so difficult to quantify and describe with data that supervising it often leads to arguments with your child.
Unlike the established homework and assessment systems for Chinese language, mathematics and foreign languages, instrument learning has long lacked a relatively simple, transparent way to evaluate progress.
This is an important reason why learning an instrument becomes a painful experience for many children and parents.
More and more parents decide that their children should learn an instrument from an early age. At the same time, many parents and children struggle to persevere and give up soon after they start.
The underlying issue is clear:
- Things would be easier if children and parents could assess how practice is going each day at home, promptly identify basic problems and resolve them.
But how can parents and children with limited musical knowledge assess practice themselves?
The old approach is to rely on exceptional parents, exceptional teachers and exceptional students.
There is a new approach: rely on the latest technology and the app in your hand.
In more and more fields, computers are taking over relatively simple, basic tasks that are highly repetitive and involve straightforward logic.
In instrument learning, the most basic problem is wrong notes!
For teachers, detecting them is also a basic, repetitive task.
Computers should take care of this work.
And it should be addressed during practice at home, rather than left for the teacher.
2
For the piano, quite a few companies were already working to solve the problem described above.
At the time, the most common approaches were:
- Smart pianos.
- Piano sensor strips: hardware devices placed on conventional pianos to detect key presses.
The basic idea behind both approaches is to use hardware to detect key presses and determine whether the performance is correct.
Digital pianos can also detect key presses, making it possible to identify performance errors.
However, these approaches are relatively costly, especially the first two, which involve additional equipment.
Existing solutions depend on hardware, but the bigger problem lies in the software. (We will spare you 20,000 Chinese characters of complaints here.)
3
People are increasingly recognizing the importance of software. “Software changes the world” is becoming more and more of a reality!
The conventional software approach to detecting errors during practice generally works like this:
- You practice while looking at a digital score in the app.
- Once detection starts, a cursor moves through the score, one note at a time.
- You have to play each note at the moment the cursor reaches it, and the score shows whether you played it correctly.
In this process, the person practicing is passively and mechanically “pressing keys”, rather than playing musical phrases.
In our view at the time, most smart instruments and apps designed to make instruments smarter used this software approach.
It is relatively simple to implement technically.
This method has some value for beginners, but it is certainly not a lasting, human-centered approach.
A human-centered approach should let learners take the initiative and play, with a practice coach beside them:
First, choose the passage to practice.
- The student practices and performs as usual, reading their own paper score.
- After the practice session, the coach comments and explains where the mistakes were, so they can be corrected promptly.
Good software should provide “intelligent practice coaching” like that coach: the app should follow the person, rather than make the person follow the machine.
4
Three years before this article, Violy had introduced intelligent practice coaching for the violin, which we described as a world first.
Shortly before this announcement, we had also rolled out support for viola, cello, clarinet, flute and digital piano.
As the parent of a young musician with Violy on your phone, imagine this scenario:
- You find the score in the app and select a passage to practice.
- Your child reads their usual paper score and practices as they normally would.
- In the app, you can see exactly where they are in the performance: which measure and which note.
- The moment your child finishes the piece, the app gives you a score for the performance and shows the mistakes: missed or wrong notes, and notes played too early or too late.
Notice what happens? Your child plays normally throughout, while you can understand their performance at a glance. When you identify a problem, you can discuss it with your child promptly and ask the teacher how to improve.
With the app’s help, parents become able to assess how practice is going.
In other words, instrument practice becomes more like Chinese language, mathematics and foreign-language study: it can be better quantified, represented through data and made transparent.
By the time of this article, Violy’s intelligent practice coaching had been widely welcomed by parents and teachers. We reported hundreds of thousands of registered users and more than 100,000 assessment records generated each day.
We also reported receiving recognition and encouragement from many experts in the field:
- Ding Zhinuo, violin educator at the Shanghai Conservatory of Music.
- Wing Ho, professor of viola at the Central Conservatory of Music.
- Gu Yinglong, violin educator.
- Shao Guanglu, violin educator.
- Huang Jun, young violinist.
- Huang Futang, violin educator in Taiwan.
There is another key reason so many people used Violy’s software-based assessment:
- It requires no additional hardware: you simply install the app on your phone.
- In other words, detection and analysis are based entirely on the recording captured by the phone’s microphone.
- Downloading and installing the software is enough to try it, keeping the barrier to entry low.
5
But sticking to this software-only approach for the piano is difficult: the sound of a piano is too complex!
We started intelligent practice coaching with the violin because its sound is relatively “simple”—though beautiful.
Three years earlier, we had already begun preparing for this day: intelligent piano assessment.
We first built up our technology using instruments with relatively simple sounds.
Now we had reached that point and finally completed an initial version of piano assessment.
We worked for three years to reach this day!
6
Dr. Wey also summarized the challenges of assessing piano performance through sound. This part is rather technical, so feel free to skip it.
- Accurately identifying the pitch of every note when several notes are played simultaneously
At any given moment, there are often many notes sounding at once. These include both the notes being played at that moment and the sustained sounds of notes played earlier. An extreme example is both hands playing chords—pressing eight keys simultaneously—while earlier notes continue to sound because the sustain pedal is held down.
- Maintaining robust polyphonic note recognition across the wide range of tunings and timbres encountered among users
In Dr. Wey’s account, string players generally tune their instruments before each practice session, whereas most piano users have their pianos tuned only every six months or longer. Some have not had them tuned for years and pay little attention to routine maintenance. As a result, each user’s piano has different tuning characteristics. The degree of deviation varies from note to note on a single piano; even the two or three strings for one note can differ in pitch. Timbre also varies enormously between pianos.
- Keeping the performance-following cursor stable
A cursor that follows the performance is one of our distinctive features. The piano’s wide pitch range, the rich combinations of notes, the flexibility of performance and the complex, varied ways users play all make stable, reliable, real-time cursor following more difficult.
- Keeping assessment results consistent across many different devices
Microphones on users’ devices have different frequency responses, and manufacturers apply different amounts of noise suppression to microphone recordings. Noise-reduction algorithms can weaken piano reverberation, suppress sustained sounds and distort some components of the audio. Different manufacturers use different algorithms; some apply noise reduction and others do not. Microphone hardware characteristics and noise-reduction algorithms mean that the audio signals we receive vary enormously.
7
After all the effort we put into intelligent piano assessment, how well did it work?
Our assessment at the time was:
- It was usable at a basic level.
- We considered it the best implementation available.
- There was still considerable room for improvement. We needed to keep working and take on another three years of development.
We will not elaborate on that assessment in this article.
At the time, we warmly invited everyone to try it!