Although relatively new, the digital conference "Interactive Cologne" is already well known among coders and digital tinkerers.

The problem: To break the conference free from its "special interest" image and open it up to a wider audience.

The solution: We hosted an event where advanced technology was transformed into an emotional experience, bringing those involved closer to the issue. By using their own smartphones, visitors became part of a process in which art, technology and a cathedral merged to create a unique concert.

We hacked a work of art by Gerhard Richter, the most expensive living artist in the world and played the colours of his glass window in Cologne Cathedral on the audience’s own personal smartphones via the 10,000 pipes of the organ.

Tools:

• Frontend: HTML5 WebSockets, Javascript

• Backend: Java, Netty, webbit, MIDI

• Server: MacBook Pro, i5 2.4GHz, 8 GB RAM, Mac OS 10.8, Chrome 34.0 Wi-Fi: Wi-Fi: 6 Ruckus APs, ZoneDirector controller, DHCP (Cisco switch), DNS (Win server)

Team:

• Christian Neuls (Consultant Software Development), Henning Schenk (Consultant Information Architecture), Anja Kirsch (Web Developer), Alexander Barth (Project Manager), Mohshiour Hossain (Art Director), Alina Schlaier (Creative Director) and Gregor Kuschmirz (Motion Director).

• Artists: Prof. Dr. Winfried Bönig (Composer), Gregor Schwellenbach (Composer) and Damian Dziwis (Digital Artist)

Due to the extraordinary nature of the location, we were faced with several challenges.

The building was not networked. We had to set up the complete technological infrastructure independently. In order to translate the organ’s MIDI signals into the 72 colours of the Richter window and transfer them to the smartphones, we had to develop a technological system that was oriented especially to the location and the event. This was only used once for this event.

The Cologne Cathedral organ already had a MIDI interface, but we first had to gauge its limits before we could connect it to our system.

We were only able to carry out tests and meetings during the night when the cathedral was closed to visitors. We could therefore only plan one test run before the final event.

Our goal was to make the visitors as directly involved in the event as possible by using their own smartphones. As it was impossible to predict which device models and operating systems the concertgoers would bring with them, we were faced with the challenge of developing an independent and stable technical solution for all models. We needed to read the IP addresses of the devices and send them simultaneously to the 10,000 pipes of the organ, while at the same distributing the hacked colours of the Richter window, based on the MIDI signals, to the smartphones.

Since HTML5 and WebSockets in particular are relatively common now, we decided to implement our idea using a combination of HTML5 in the smartphone browser and a Java-based WebSocket server in the backend.

The connection then worked roughly as follows:

• The smart phone connects to an internal Wi-Fi network

• The user enters the address in the browser (dom.int)

• The HTTP server returns an HTML page

• The user enters the seat number

• In the browser, the HTML page uses Javascript to directly open a WebSocket connection

• The server can now “remotely control” the screen, i.e. change the colour and the interface elements

• The server thus administrates a list of open WebSocket connections and can send commands, such as colour changes, to all or any number of individual clients. All the clients’ connection data is stored in RAM in order to ensure maximum performance.

There were no standard solutions on which we could base our set up. Therefore, the whole interactive system is an innovation that makes it possible to transpose the notes of traditional musical instruments to hundreds of smartphones in the exact in real time and in the exact position. It was only used once and for a fixed number of guests at this event.

The bidirectional WebSocket technology is well-suited for all kinds of projects where the intention is for dispersed systems to communicate with each other in real-time. Simple, but scalable systems can be set up very quickly without complicated proprietary protocols. The technology can also be used, for example, in a native App or in a live application using other output media.

Due to its simplicity and minimal latency, the technology is also well-suited for the quick prototyping of completely new ideas. The combination of various technologies (WebSockets, HTTP, MIDI parsing, Threading, NIO) enables the flexible coupling of components of the most varied kinds (in this case: organ – server – smartphones) that may initially appear incompatible.