For some time now, artists, programmers and marketers have been seeking innovative ways to collaborate, to blur the line between art and technology, thereby creating complex systems that merge the real world with the digital world. To launch Nike’s new Hypervenom football boot collection in Southeast Asia we created an experience that transcended digital mediums and allowed consumers to physically take part, turning public performance into a spectacle. The ‘House of Deadly’ was born, a mini-stadium with projection mapped walls and tracking spotlights that acted as real-time virtual defenders that players had to be quick enough to dribble past before shooting at a moving target. The half a million litres large interactive installation was set up in Bangkok’s Siam Paragon, one of the biggest shopping centres in Asia, over three days. We worked with a team of software engineers to create the first computer vision system for real-time tracking of a football and the first depth camera tracking system to detect point of impact for a football. A team of 3D designers, motion animators and a sound designer developed interactive 3D landscapes that changed and evolved based on the players’ reactions and a team of more than 50 people to erect the whole installation.

The main objective was to allow fans to experience a whole new way of playing football, to make fans feel as if they were immersed in an epic video game that preserved the physical skills required when playing football in real life, then take the player through an emotional journey culminating in a ‘deadly’ twist, giving him or her a sense of empowerment from the game.

Authenticity is key for Nike. Not only did the experience have to feel real for football fans, but it also had to reflect the key attributes of the product - deception, agility and accuracy - and fully encapsulate Nike’s Hypervenom concept of fierce beasts, represented in the different landscapes and targets designed for the game.

To create such an experience we made the best possible use of existing open source software tools, such as openFrameworks, Arduino and Kinect to develop proprietary technology that we own. To simulate the skills necessary to play football like in the real world we had to overcome several technology challenges: hack two ball-feeders throwing balls on each level, develop a computer vision system for real-time tracking of a football, develop software to make mobile light heads behave as real defenders by engineering in a human factor following the ball and adjusting to the player’s reaction time, warping and synchronising three HD projections to control the content based on the player’s performance, synchronization of tracking, visual and gameplay processes and wireless DSLR cameras over an intranet to automatically share the players’ performance on social networks.

The event’s reception was overwhelming. The game operated at full capacity over the 3 days of the event – allowing everyone from young football fans to experience a football boot in a totally different way, to professional players from FC Barcelona such as top striker Neymar, who absolutely loved the game due to its realistic features and interactive elements.

Given the great feedback given by the participants and professionals, the future looks bright for ‘The House of Deadly’. We are planning to package and bring this experience to several other markets. We are currently in discussion with one of the largest international shopping mall companies to develop a portable experience based on the House of Deadly. We are discussing a license agreement with several professional top football teams to adapt the installation as a training facility given the positive feedback given by FC Barcelona players and management team. For that we are developing a new set of features that focus on measuring the ability, performance and accuracy of a player adapting the challenges based on the skills that need to be improved.