news from the intersection of branding & technology

Posts Tagged ‘Augmented Reality’

The Augmented Eye of the Beholder

In brand-building, mobile & tablets, Shopper Marketing on January 8, 2011 at 12:59 pm

by Jeff Sandgren, Technology Editor

After that C in high school Creative Writing, I swore I’d never write another story that asked the question “what is reality?”  Yet here we are, me breaking my oath, and you hopefully a bit too curious to look away.  Well, technically, we’re going to examine in this story what is not quite reality, by looking at the ingenious efforts of three companies who find that a bit of augmentation makes the reality of brand marketing and messaging at retail more engaging to shoppers.

Some of the most familiar forms of augmented reality (AR) are Sportvision’s yellow “First  and Ten™” line on football fields and the “K-Zone” over home plate in baseball games, as seen on network television.  In fact, Sportvision’s many augmentations and mediations in sports broadcasting have become interwoven into our video fabric of perception – a model of solution maturity to which retail might do well to aspire.  Overall, AR is augmenting its market size in the real world, too.  ABI Research predicts a market size of $360 million by 2014.

At another corner of the Milgram Continuum (a graphical representation of the ‘map’ of augmentation and mediation) is Lockheed Martin’s F-35 Lightning Helmet Mounted Display System, which is too complicated for me to even attempt to describe, but represents a major innovation over the previously innovative Heads-Up Displays, standard gear for most recent fighter aircraft.  No, this is NOT the future of brand marketing.  We’re looking somewhere between yellow lines and F-35s.

One of my first surprises, in interviewing three hot players in this field, is that none of them is of U.S. origin, although all play eagerly in this market.  A sad but increasingly resonant reflection of the state of innovation at home, I fear.  At the leading edge of cool retail AR implementations I found European and Japanese innovators: Metaio, Total Immersion, and LM3 Labs.


Founded in 2003 and headquartered in Munich, Germany, Metaio GmbH is a pure-play AR shop dedicated to “enhancing the real world with 3D animation”.  Metaio’s founders, Thomas Alt and Peter Meier, met while mechanical engineers at Volkswagen.  They won a contest with a purse of 25,000 euros, threw in another 100 euros each, and never looked back.

Noora Guldemond, Metaio’s head of sales and marketing here in the US, notes the significance.   “Here in the Valley,” says Noora, “the perennial question is which round of funding your startup is on.  We’re proud to be a bootstrap company, completely self-funded, and already profitable over the last few years.”  Today Metaio has additional offices in San Francisco and South Korea.  According to their website, they have already completed projects for over 340 customers.

According to Noora, one of their most successful projects has been an interactive kiosk for Lego’s.  Shoppers can hold a box of Lego’s in front of the kiosk, and a superimposed display shows them how the assembled toy will appear, in 3D.  After starting in a few stores in 2008, sales lift surpassed expectations, and the kiosks are now being rolled out worldwide to all the company’s branded stores, and to a few additional locations including selected Walmart and Toys R Us stores.

Other prominent Metaio projects in retail and brand marketing include Ben & Jerry’s MooVision ice cream lids and projects for JC Penney, Time Out New York, Adidas and many others.  My personal favorite is Zombie ShootAR, a smart phone app that lets bored guys out shopping with their significant others see an image of their retail environment with zombies crawling up out of the ground that you have to shoot before they chew your face.

“No, honey, don’t hurry.  Take your time.  I think you should try on one of EACH color.  Accessorizing is much too important to rush.”  Zombie slaying: a tough job, but somebody’s gotta do it.  The REAL zombies, of course, are the shrewd marketing beasties who know your purchase size is getting bigger with each demon you destroy.  Good news: nobody ate your brains; bad news: your paycheck just became an entree for the undead.

Although they have a strong in-house development team – over 80% of their employees are technical engineers – Metaio is nurturing the mobile ecosystem with Software Development Kits (SDKs).   Their technology centers on their proprietary Unifeye™ platform, which expanded this year to include SDKs supporting iPhone, Android, Symbian OS and WinMobile.

Metaio’s mobile AR browser, enabling zombies and other graphic and informational augmentations to your smart phone, is branded “junaio™”.  Like most other mobile and internet-connected AR apps, it can be downloaded for free, and most of the content provided is also free.  With all that “free-ness”, you know the zombies of finance have to be just below the surface, demanding their just desserts.  One recent addition to enable this is junaio`s new “Ad-Inject” service, which enables ads to be served up on your mobile browser in a similar manner to the way Google AdSense works on your traditional one.

An example of this is the recent mobile ad campaign of the Hermes parcel service available in all major German cities. Whenever a junaio user opens the Wikipedia channel and points his onboard camera to check out interesting sites around him, he is being made aware of the nearest Hermes parcel shop in the vicinity. “This isn`t just plain vanilla advertising”, says Jan Schlink of junaio, “Hermes sees this campaign as part of their dedication to customer service.”  Good thing: zombies hate plain vanilla.

Total Immersion

A second leader in the field is Total Immersion (TI), which began eleven years ago as a French startup and today has offices in Paris, Los Angeles, Hong Kong, Tokyo and London.  We recently spoke with Max Polisar, VP Sales here in the US, following their announcement of strategic alliance with Adobe.  TI’s proprietary platform D-Fusion™ will now be more accessible to the “more than three million” developers and designers actively using Adobe Flash for desktop, mobile and other consumer electronic devices.   Total Immersion also plans to leverage the new Flash Player and Adobe AIR 3D future releases.  The business model includes providing developers with an integrated “tracking library” for Adobe Flash Player and AS3, which will generated “protected” software; developers can then pay a licensing fee to TI in exchange for a runtime key.

“The big difference with our solution,” says Polisar, “is marker-less tracking … that works.”  The point here is that some AR solutions use ‘markers’, like barcodes, quick response codes, or similar symbologies, to trigger recognition by the software application and generate the corresponding imagery.  In some cases, packaging redesign is necessary to properly display the markers, which can then create a domino effect of packaging and distribution problems, making a campaign much more difficult to design and launch.  By developing a ‘marker-less’ solution, the original object, without re-design, can be recognized and actuate the AR experience – streamlining campaign launches. [Editor’s note: some other companies also claim to have ‘marker-less’ solutions.]

Polisar says that Total Immersion focuses on making the AR “a feature of the product”, rather than just a promotional gimmick.  As an example, one of last year’s successful projects was Hallmark’s Webcam Greetings, which allowed recipients of traditional print greeting cards to interact online, using their webcam, and seeing a moving 3D animation of their card.

TI’s projects include marketing-at-retail solutions, but that’s just part of a much broader spectrum including toys, publishing, trading cards, amusement parks, and more.  Another favorite of Polisar’s was a project with the Olympus 10 camera.  “It helped push forward the technology,” says Max, “beyond just the cool and wow.  It really helped us open the door to the consumer electronics category.”

Other Total Immersion clients include Mattel, Coca-Cola, McDonalds, P&G, Kraft Foods, Nokia, BMW, Boeing, Sony Ericsson and many more.

TI is focusing to push the D’Fusion software onto “as many platforms as possible … we want to be platform agnostic.”  The broad net strategy seems to be working: at their recent open house, over 400 attendees from around the world paid them a visit to find out more about TI and AR trends in general.

LM3 Labs

While not pure-play Augmented Reality in the usual description, some of the most exotic – and exciting – work in interactivity solutions is being done by LM3 Labs, which spun out of France’s Centre National de la Recherche Scientifique (National Center for Scientific Research) in the same way that E Ink displays and Electronic Product Codes spun out of MIT here in the US.  The founding scientists were working on computer vision-based tracking algorithms originally, and went to Japan in to engage the video projection manufacturing sector which is so prominent there.

“It really started with a big mistake,” explains Nicolas Loeillot, co-founder and COO, “we were just trying to license the technology to manufacturers, but couldn’t generate much interest.  The feeling back then was that people didn’t need interaction. ”

Following this initial discouragement, the team went back to France, until in 2004 they began to work with Japan’s NTT DoCoMo.  This led to the epiphany, as Loeillot describes it, that “… clients wanted to buy products, not technology. And of course it is so easy to make products in Japan.  So we wound up becoming as much a ‘hardware company’ as a technology company.”

Their first product was the Ubiq’window™, which allows people to interact with digital content or real life objects just by using natural gestures.  Over time, their portfolio expanded to include Catchyoo™, a family of interactivity solutions for floor, wall, table and 3D spaces, and Airstrike™, a touchless solution which lets users interact with any type of display, distantly, with natural gestures.

As their clients continued to encourage them to provide more ‘whole solutions’, they developed a network of partners to deliver just that.  “We were selling color commentary to black & white guys,” reflects Loelliot, “we couldn’t just sell interactive tables and chairs.”

Nicolas admits that their solutions aren’t cheap.  “We tried going to a convenience store chain, but found our sensors were more expensive than their budget for one shop for a whole year!”  Undaunted, they shifted their focus to ‘higher-end’ clients, and have successfully delivered projects to a prestigious list including Audi, Calvin Klein, Cartier, Chanel, Chivas, Dior, and more.  (And yes, Mr. Liljenwall, they did a well-received project at Disney Tokyo Resort)

As they’ve grown their portfolio and successes, less glamorous but powerful ‘everyday’ brands have also adopted their solutions, including Amstel Beer, Heineken, Levi Strauss, Nestle, Nokia, Pepsi … again the list goes on and on.

Being involved in installations around the globe, Nicolas notices significant differences in cultural patterns in getting people to interact via touch and gesture.  He cites an installation for Ito En (a Japanese tea retailer) in New York City that LM3 Labs did during a virtual blizzard.  “The chill factor was -15°F,” recalls Nicolas, “yet there were people all over it!  We can do an installation in busy area of Tokyo, and initially no one touches it!”  Generally speaking, Nicolas observes, it’s a question of passive versus active interactivity; “Asians expect the content to come to them.”

So, armed with these insights, they adopted a different approach.  First they tracked bodies moving by windows on the street, but found the content changed too much.  Instead, they are using face-tracking instead of gesture recognition in some recent installations.  At an installation in Ginza for Seiko, the “Seiko Flame” follows passers-by along the window as they walk along, but only switches to more meaningful content if the viewer’s gaze shifts to the flame and lingers.

Looking Ahead

Looking ahead, Guildemond sees technical advances in Flash, notably with more native 3D support, as being a key upcoming development.  “Better 3D will enable a more engaging experience,” says Noora, “Games will drive creative, and that will accelerate adoption.”

So when should marketers use AR?  “It depends on the brand and audience,” suggests Guildemond, “Look for opportunities to extend engagement after the purchase, or to close the deal.  Look at what’s been done by others so far.  Focus on audience experience.  And call us!”

Polisar also sees the enhancement of the consumer experience as paramount.  “We see AR moving into commerce akin to online shopping.  Ubiquitous image recognition is the future … in and out of the retail store.”

LM3 Labs has big plans in store for 2011, with the launch of their newest offering, MoovAction™ starting this quarter.  They’ve got some very cool table top stuff coming, tracking individual fingers for special effects literally “at your fingertips”.  And as if that’s not enough, Nicolas hints at a 3D field project, 3DFeel™, that will launch this year and goes beyond even what Microsoft’s Kinect™ can do.

“It’s really not Augmented Reality as much as it is ‘Mixed Reality’,” Loeillot maintains, “where there’s a device, something between you and the reality, that brings digital content into the real world in a wide variety of forms that allow you to interact naturally in ways that enhance your experience.”

Between that Zen-like view and the more graphic images of attacking zombies and stalking flame-balls, it can all get a bit mind-melting.  But don’t be put off by the descriptions.  Readers are encouraged to go online and view some of the sample projects of these leaders.

Augmentation and mediation are coming, as sure as zombies, and you’ll want to have your head on straight if you don’t want to get eaten alive by your competition.                  JTS

  • junaio launches Ad-Inject mobile advertising offering (
  • ARCchart: Mobile augmented reality app revenues to reach $2.2 billion by 2015 (
  • You: Readers Choice Awards 2010 Winners (
%d bloggers like this: