For my HCI course project in Fall 2022, I decided to study the role of visual aids such as piano roll and music notation in aiding the memorization and reproduction of note onset timings in a jazz improvisation with respect to a backing track. The motivation for this kind of study is two-fold. Firstly transcription and imitation are essential while learning jazz improvisation. The task we present in our study is a form of rhythmic transcription and imitation which would be very beneficial for students of jazz. Secondly, previous studies, such as Hilebrandt et. al. have shown that for temporal information (such as the position of note onsets) is better captured in the audio modality and in the case of multimodal input (audio-visual), the visual modality doesn’t seem to play an important role at all. However, my hypothesis was that in the context of music, visual inputs must have some significant role to play in memorization/understanding of temporal information given the popularity of visualizations such as score notation among western musician performers, piano rolls among digital audio workstations etc.

For this project, I developed an interface . Users had to listen to (and in some cases, see) the snippets of improvisation, memorize the position of the notes with respect to the bass and drums, and tap out the note positions first along with the improvisation playing in the background, then without the improvisation (i.e. only with the bass and drums). Through quantitative and qualitative analysis, I show that visualizations do indeed help with a more accurate memorization of the note onsets.

For more detail, please read my report . This was preliminary work and I would love to hear feedback if you have any!