This weekend I was at MHacks 11 in Ann Arbor, Michigan. We had an awesome time building a game called Recorder Hero, and got to use some cool technologies that I’ve been wanting to try out for a long time!
The Project
Recorder Hero is like Guitar Hero, but for the Recorder. You know – that instrument that every kid in America learned to play in elementary school. The game has a playlist of hit songs, ranging from Hot Cross Buns to Party Rock Anthem, that you can learn to play on a handheld digital recorder (your iPhone).
Architecture
Recorder Hero, as a complete system, runs on two devices at once. The handheld digital recorder is an iPhone app with on-screen buttons that mirror the layout of an actual recorder. Meanwhile, the Gameplay interface is a Python app that loads up a serialized sheet music representation, renders the track timeline, and plays the recorder notes.
One of the big challenges of a complex multi-device setup like this is, of course, getting the devices to talk to eachother. One potential implementation could do this wirelessly — either by setting up a local Bluetooth connection, or by putting a web-server in the middle. Hackathon Wi-Fi is notoriously spotty, though, and we’ve gotten burned before by trying to rely on a stable connection for our hacks. Latency is also a big issue for a game like this, because the user experience would really deteriorate if there was a networking delay.
Instead, we opted to communicate over USB via a lighting cable plugged in to the laptop and the iPhone. There’s a wonderful Objective-C library called PeerTalk that makes it really easy to set up a TCP server that communicates over USB between an iOS app and a companion macOS app. Using that library, the iOS app sends a one-hot-encoded bitstring of the pressed buttons (the simulated recorder holes) to a macOS helper app running on the connected laptop.
The only problem, though, is that PeerTalk only gets the data onto a macOS app. The gameplay interface is running in a Python app, so we needed another communication layer. To solve this problem, we set up a pretty rudimentary one-way Socket between the macOS helper app and the Python app. The macOS app uses this Socket to forward any messages it receives to the Python app. With all of those pieces in place, we have a real, wired, handheld digital recorder with almost no latency. Pretty cool!
Gameplay
After spending the first half of the hackathon working on the input architecture, we got to spend the second half building out the actual gameplay experience.
The biggest piece of a game like Recorder Hero is the music itself. We scoured the internet for Recorder sheet music that’s simple enough to play without much experience, but recognizable enough that you could still jam out to it. A recorder doesn’t have a very wide range (and can’t play sharp or flat notes, either), so that limits the songs that you can play. We settled on a playlist of six songs including recorder classics like Hot Cross Buns, pop hits like Party Rock Anthem, and classical score like the Pirates of the Caribbean theme.
We set up a mapping between recorder button combinations and musical notes, complete with audio files generated from a recorder synth. One interesting problem that we ran in to is that the iPhone can only detect five touches at a time. This is a limitation of the hardware itself, so it’s not something that can be hacked around with some clever software. Unfortunately, that meant there would be some notes that couldn’t be correctly represented by our digital recorder (lower C requires seven fingers, for example). There are also some note combinations that are only distinguished by either covering or not covering the back hole (which our digital recorder doesn’t have), so we have to come up with our own button combinations for those notes.
After we had the game up an running, we ran into some practical limitations. Songs with small ranges like Hot Cross Buns and Party Rock Anthem were fairly easy to play, but songs with wide ranges turned out to be unreasonably difficult. We tried to stay as true to a real recorder button layout as possible, but button combinations for notes in the top octave get really complicated really quickly. It may be possible to switch between these quickly when holding an actual recorder with physical holes to guide your fingers, but it was pretty impractical on a smooth slab of glass.
The game as a whole, though, is very fun! Most of the songs can be played accurately with some practice. And, for the most part, you’re learning real recorder skills! Video Demo
The Team
I went to MHacks with my friend Jake. This was our fourth hackathon together! Others included DerbyHacks 2017 in Louisville, UB Hacking 2017 in Buffalo, and MinneHack 2018 in Minneapolis. On this trip, we also got to spend time in Downtown Detroit and Windsor, Ontario (across the river).