AR in real places
Six years spent on one problem: how do you put a digital thing in a real place and have it stay exactly there? It took a wayfinding character, a stadium, a 15th-century palace and a band playing over Times Square. Each one solved the thing the last one couldn't.
GPS gets you to within three to five metres, and it needs you to start walking before it can work out which way you are facing. That is fine for a blue dot on a map and useless for AR. At that margin a character stands inside a wall and a hologram performs ten feet off the stage.
Everything below follows from that one fact. To place something precisely on a real location you have to know the location precisely: map it in 3D, then locate the phone against that map using what its camera can see rather than where a satellite thinks it is. The four projects here are that idea getting progressively larger: one character on a street, then a stadium, then 110 acres of protected heritage site, then whole city centres.
HotStepper
The world's first AR character-based wayfinding app. Type any address and a character walks the real streets ahead of you until you get there. He reads the neighbourhood as he goes: a new haircut outside a barbershop, a reaction to whatever he passes. A rendering system we wrote samples colour straight from the camera feed, so he picks up the light of the street he's actually standing on and casts a shadow from the real position of the sun.
It also ran straight into the ceiling. With only GPS underneath him, we held him in place with creative hacks. The best of them was a confidence score acting as an invisible dog leash: strong signal and he strides ahead, weak and he sticks to your heel. It shipped, and people liked him. It still wasn't what we wanted, and closing that gap became the next five years of work.
AT&T Stadium
Nearly two years with Samsung produced a platform for geo-located AR delivered over 5G, and AT&T Stadium became the first venue in the world prepared end-to-end for it: the Samsung/AT&T 5G Fan Experience for the Dallas Cowboys.
The answer to GPS was to stop using it. We built a dense 3D map of the building with Scape Technologies and located the phone visually against that map, giving centimetre accuracy from a standing start with no walking in circles to calibrate. Because the map is real geometry, content behaves: a hologram walks behind a pillar instead of through it. And because the capture opens directly in Unity, the creative team designs on top of the actual building rather than a floor plan and a guess.
5G carried the other half: volumetric holograms captured at Metastage, streamed live through a compression format we wrote to get them to around 3% of source size with no visible loss. We productised the whole pipeline as a service for making any location AR-ready.
Changdeok Palace
Changdeok ARirang took the same approach to a UNESCO World Heritage site in central Seoul: 110 acres, 13 buildings and 28 pavilions, covered with Google Cloud Anchors and delivered over SK Telecom's 5G network.
Haechi, a mythical guardian, meets visitors at the gate and leads them through twelve activities across the grounds: archery with a Joseon prince, a traditional dance, court performers streamed in as holograms in period costume. It is the HotStepper idea three years on, now precise enough to walk someone to one specific pavilion inside a protected site where nothing can be installed, marked or drilled into.
The terrain forced a design answer. Across steep steps and shifting elevation a walk cycle either floats or clips, so Haechi jumps instead. The most convincing part of a location-based experience is usually the moment it admits the location is real.
Gorillaz Presents
By 2022 the map no longer had to be ours to build. Google's ARCore Geospatial API opened up Street View's global localisation, so a phone could place itself accurately in tens of thousands of cities with no bespoke capture at all.
Gorillaz Presents used it for a free AR concert. Murdoc, 2D, Noodle and Russel appeared at building scale over Times Square on 17 December and Piccadilly Circus the day after, performing Skinny Ape against the real skyline for anyone standing in the square with the app.
Same question as HotStepper: where is this phone, exactly, and what belongs there. Answered at the scale of two of the most photographed places on earth, live, in front of a crowd.
- Cannes Lions, Digital Craft winner
- Webby Awards, double winner
- Emmy Awards, 2023 nominee
- Tribeca X, 2023 winner (Immersive)
The parts of this work that are hard
The demo is never the difficult bit. These are the things that decide whether a location-based experience survives contact with a real venue, a real network and a real crowd.
Surveying and 3D-mapping a real location fast enough to be commercially viable and clean enough to design against inside a game engine.
VPS, cloud anchors, geospatial APIs, marker fallbacks. Each one fails differently indoors, outdoors, in crowds and after dark, and picking correctly is most of the risk.
Occlusion, elevation, sightlines, scale. Content that ignores the building it is standing in reads as a sticker on the camera.
Streaming volumetric and high-fidelity content over the network you actually have, inside the battery and thermal budget of a phone held up for ten minutes.
Onboarding hundreds of people at once, in daylight, in public, with venue staff who need to explain the whole thing in one sentence.
Built with and alongside Google, Samsung, AT&T and SK Telecom, including pre-release access to platform tech before it shipped.