Thursday, April 12, 2012

Distributed Laptop Orchestra

So, this semester I've been involved with this really interesting class at Carnegie Mellon: 15-323 Computer Music Systems and Information Processing. Taught by one of the creators of Audacity and Nyquist, Roger Dannenberg, the course aims to give the students a broad background in how music is treated programatically, to the point were at the end of the semester we can all co-operate to build a full-blown laptop orchestra application!
On Monday April 16, it is going to happen; the world largest distributed laptop orchestra is going to perform as a part of "Symposium on Laptop Ensembles & Orchestras" at Louisiana State University. Apart from LSU and Carnegie Mellon, five more universities in the US and UK will participate

Stanford University
Texas A&M University
University of Colorado
University of Huddersfield (West Yorkshire, England)
Queen's University (Belfast, Northern Ireland)

It will definitely be a late night for our friends on the east side of the Atlantic, as the concert starts at 20:30 EDT (New York -time). That equates to 01:30 BST! And at that time, I will sit on the stage at UC McConomy, eager to hammer notes and sounds out of my keyboard. I expect my fellow europeans to be the same.

The previous CMU laptop orchestra
I also hope the laptop orchestra will actually work. That piece of software right there is running my code - I do not want it to crash with such great universities watching and participating.

If you can't come to McConomy to hear the Carnegie Mellon -version, you should tune in to the live stream from LSU!

If you are interested in the source code for the project, it is available as open source. In fact, you can even download the code and participate yourself! There are instructions on how you should use the system online as well, so I will not go into details here; but I'd still like to give a brief technical overview of the project, and discuss some of the interesting challenges:

(At least to me, they are. But I take no offence if you stop reading now :-)

A performer's computer is connected physically to a live PA system in each of our 7 locations via mini-jacks on the headphone port; our software contains sampled sounds it will loop or play once, depending on the nature of the sound, and it can also generate a few waveforms.

(In order to do this, we must know how to generate, process and play back music in real time on the computer. The music is generated in blocks of bytes, each representing a very short time of sound, by the standard of an ear; this block is then pushed into the computer's music playback buffer, and then the next block is generated. This must be done on a high priority thread, to prevent any lagging. That kind of stuff is what you learn in this course.)

After the music is pushed out to the analog mixer, it is mixed and sent two places: a) To the local speakers, and b) as a soundstream to LSU.

In the local mixer, there is then one input channel for each performer, as well as a stereo input coming from LSU. The stream from LSU contains all the sounds from LSU as well as all the sounds from the 5 remaining locations. Thus, there will be some pretty significant latency issues; speed of light is only so fast, and there is lots of network jitter as well. The sound from LSU will then have some latency, and the sound from the other universities will have an average of twice that latency. This also has the consequence that the performance will sound different at each location, depending on the local mix and also on the varying latencies.

Roger Dannenberg, conductor
(play picture as sound)
So how do the performer know what to play? It certainly can not be very rhythmic considering the constraints discussed in the previous paragraph. But it should be somewhat synchronised, to create at least some sense of structure. To do this, we have created a conductor interface.

The conductor (Roger) will be able to send various conducting messages through a special mode in the software. These messages will appear in the window for all the performers, so that the we can play according to the conducting; variable parameters controlled by the conductor include

* Intensity
* Dynamics
* Pitch (high/med/low)
* Play/stop/fade in/fade out
* Music language/Sound architecture
The performer interface; top half is determined by conductor
Note that the conductor can send different messages to different universities, or "orchestras" as we cleverly named them in the software. There is also a chat system based on an hierarchal form, so anyone can send chat messages to anyone, everyone in a single orchestra, or everyone in the federation of orchestras.

To make all of that work, there is a lot of network logistics going on. The network is organised by nodes and supernodes; all nodes are connected to a supernode, and all supernodes are interconnected. When sending a message from a node (say, the conductor node), the message is first sent to the supernode, then from the supernode to the correct supernode(s) (if applicable) and then from the supernode to the individual node(s).

But, how do the conductor node know at what IP address its supernode is? To do this, each node will register with a special server (currently running at CMU) when they start up. This server will inform the node of what IP its supernode has, or if there is no supernode yet. If there is no supernode yet, then the supernode will initiate a connection when it is set up. In this fashion, as soon as a node is registering with the server, the server will provide information about all the nodes with which it should connect; if the node is a supernode, this includes all nodes in his orchestra, as well as all the other supernodes.

In addition to this, there is a clock synchronisation protocol running, and there is also a performance monitoring system that keeps track of how much data is being sent to and from each node. My contribution to the code has mainly been on this performance monitoring system. It basically collects information of all messages going in and out of a node, as well as the latency to the closest "parent" node (where the node containing the original clock is the uppermost parent; this should be the supernode at LSU).

In sum, this is going to be pretty awesome, and I'm looking forward to celebrate the anticipated huge success of our concert. Wish me luck, and tune in on Monday!

CMU School of Computer Science: Laptop Orchestras in Seven Cities Unite Via Internet for First-of-its-Kind Concert
15-323: Computer Music Systems and Information Processing
LSU: Symposium on Laptop Ensembles & Orchestras