For over 130 years since its foundation, Yamaha has stood by its brand purpose "To deliver the joy of music equally to all" and expanded possibilities of musical expressions supported by data, including development of an auto-accompanied piano.

In this project, as an extension of the AI technology that takes in a large amount of performance data, Yamaha made it possible for the piano to compensate for the pianists' individual physical disabilities, allowing them to play the piano as a soloist in an unprecedented, highly challenging concert joined by one of Japan's leading orchestras and choirs.

Context 1: While many Japanese corporations are committed to tackling the issue of DE&I, the progress is lagging behind in the world. About 60% of the Japanese public say when meeting with people with disabilities, they do so consciously. Comparing it to the situation in the US and Europe (roughly 10%), we must admit that there still is a tendency to view physical disabilities as unusual. It is, therefore, no surprise that when it comes to disabled people wanting to enjoy music, opportunities for them to play musical instruments and to play with others are still quite scarce. Considering the situation, we thought this was a challenge that Yamaha, as a leading musical instrument manufacturer, was best positioned to tackle.

Context 2: Listening to Beethoven's Ninth Symphony has become a year-end tradition in Japan, with as many as 23 concerts per day are held in December every year. Numerous orchestras and choirs play the symphony at the end of the year, wishing for people to reunite in the hope for peace. Holding a concert of the Ninth at this time of the year naturally attracted attention but it was also necessary to make it a truly unique one to stand out among many other concerts.

The joy of playing a musical instrument, and doing so with others, is beyond compare. But it is also true that it takes a huge amount of time and effort to attain that joy, and many people give it up halfway through the process. People with physical disabilities are all the more likely to do so, because however passionate they may be, they face great difficulties with playing a musical instrument on stage together with other musicians due to their physical limitations. That is why Yamaha, a leading company in musical instrument manufacturing, decided to rise to the challenge and make the joy of music equally accessible to anyone.

Data from physically disabled pianists contain irregularities such as notes omitted due to fingers missing a key, or uneven sound caused by unstable keyboard touch. As a result, when playing with others, they may not be able to synchronize with other players or generate sound that can be heard over other instruments.

Yamaha upgraded the technology used in the "AI-Assisted Piano" that automatically plays the part for unmovable fingers in synchronization with the pianist's performance. By collecting performance data of the three pianists with different disabilities, Yamaha applied three individually aligned AI technology and enabled the pianists to express themselves in a performance of the Ninth joined by Japan's leading orchestra and choir.

Usually, AI means a technology "that creates a standard from massive amount of data", but in this project we introduced an AI technology that is customized to meet the different needs of individual players.

Performance data was gathered from the practice sessions that spanned nine months (total 18 hours/month for three pianists) to develop individually aligned AI technology for each pianist. Data gathering continued as the music arrangement was repeatedly revised with consideration to the pianists' skills and balance with orchestra and chorus.

AI is used to assist the pianist born with a three-fingered right hand by playing for the missing two fingers, to help the bed-ridden pianist who strikes the keyboard at the back of a finger by detecting the faintest push on the keyboard and generating other required sounds and pedal movement, and to support the weak-fingered pianist by adjusting the sound volume. This high-quality concert was realized by adopting such AI technologies in order to harmonize the performance of disabled pianists with the orchestra and choir.

Data gathered in advance were analyzed to calculate the parameters of performance assistance. During the concert, the physical data of the player's touch on the keyboard is read by sensors real-time to move the hammer, realizing an output intended by the pianist. The following three technologies contributed.

1. Power-Assisting Technology: When the keyboard is struck too weakly to generate properly audible sound, it adjusts the strength of the hammer in synchronization with the pianist’s performance to ensure that all sounds reach the audience.

2. Ultra-Low Latency Sound Generation: A technology that prevents accompaniment part from lagging behind the piano, when both should be heard simultaneously. Even the faintest touch on the keyboard is detected and transmitted to the system in order to minimize the inevitable delay.

3. Performance Continuation Technology: It adjusts to the pianist's irregular movements, maintaining seamless music flow.

Praised by many including a university dean of the School of Global Studies and Collaboration and a business leader promoting AI in Japan, saying "Yamaha has evolved AI to a new dimension that will contribute to entertainment and culture creation", this project is also expected to have an impact in business and education in future.

In addition, this AI system will be made into an app to be released in 2024 for use with over 10 million pianos worldwide, aiming not only to broaden piano users by making it easy to start playing but also to provide even better user experience to existing piano customers. The challenge of providing an enabling environment to every person who wants to play the piano has been successfully achieved.