AI Creates Digital Twins For A New Age Of Internet

By John Jeffay, NoCamels March 05, 2023 5 minutes

Yoom produces volumetric images, or digital twins, for a wide range or immersive experiences. Courtesy

Say hello to your digital twin. Deep tech experts in Israel can capture a highly realistic 3D image of you using multiple cameras and AI.

They then bring your twin to life either doing things they’ve actually filmed you doing – playing guitar, taking a golf swing, jumping out of a plane – or things you haven’t.

Your digital twin can be programmed to mimic the actions of a library full of stuff that other people have done while you sit at home and enjoy a coffee.

Yoom captures a photorealistic volumetric image using multiple cameras and powerful AI. Courtesy

Yoom, founded in Tel Aviv, is one of the go-to companies for what’s called volumetric video capture – creating a 3D image with endless opportunities.

The internet is taking another leap into the future with “immersive experiences”. That means we won’t just see a character in two dimensions. As we watch a performer sing, for example, we’ll move the mouse on our device to swing from left to right, zoom in or out, go low or high, and make the experience unique and personal.

Volumetric video capture is the technology that makes this possible. Yoom have been working on it since 2016 and say the AI that powers their system is among the best, because it has learned from thousands of captures, and has been tweaked and refined over the years.

It’s an expensive and complicated process, so the main clients currently are sports stars, entertainers, games developers and retailers.

As the “real” and virtual worlds become more and more blended – that’s what the Metaverse and Web3 are all about – the demand for immersive and interactive content will grow exponentially. Consumers will come to expect a 3D experience, rather something that’s flat and 2D.

“We allow people to create without any physical boundaries,” says Bonnka Lim, Chief Marketing Officer at Yoom.

“We work with a Canadian artist called Riell. She had a creative vision about doing a music video about her childhood and the struggles she had.

Montage images shows the AI capturing Riell as she moves in the studio. Courtesy

“But she didn’t have the resources to actually do a shoot in Europe and other places with the team. So we captured her in the studio and created the whole world that she had in her mind.

“We produced a music video clip tied to the vision that she had as a creator. Our technology lowers the barriers of content creation, ultimately, because once we have your 3D model, we can create any environment you want.”

Images then go through a complex post-production process. Courtesy

Riell was filmed using a set of eight cameras, and her movements were processed by Yoom’s highly sophisticated neural networks – a series of algorithms that mimic the workings of the human brain to perfect the rendering, texture and lighting.

Sign up for our free weekly newsletter

The images they captured resulted in her digital twin on an apocalyptic city street and a blazing house for her single End It – in scenes that would have otherwise been impossible to create on a budget.

Cameras capture Riell from all angles to produce a 3D or volumetric image. Courtesy

“I didn’t know anything about volumetric capture before this,” she said. “I did not know this existed, nor did I understand the first like three times it was explained to me.

“I’m really amazed that it’s a camera with an algorithm that creates a 3D version of you. It took me a bit to wrap my head around that. It recreates you, which I think is sick!”

The Yoom team also captured Dustin Brown to mark his retirement from the LA Kings ice hockey team in the USA after almost two decades.

Ice hocky star Dustin Brown is filmed for an augmented reality tribute to mark his retirement. Courtesy

His digital twin appeared on a huge billboard, opposite the team’s stadium, appearing to smash through a window and hold aloft the Stanley Cup, the National Hockey League’s championship trophy. That was the result of capturing his volumetric image in the studio, then enhancing it and adding effects.

Paul Oakenfold, the godfather of electronic music and legendary British record producer and trance DJ, is another of Yoom’s celebrated volumetric captures. He appears in a PerfectoVerse, a 72-minute paid-for concert film and immersive experience that combines his lyrics and music with computer-generated visual wizardry.

The digital twin of Paul Oakenfold, godfather of electronic music, in the film and immersive experience PerfectoVerse. Courtesy

“He came to the studio and we captured him, and created a whole experience where people can actually see him performing, they can be around him and interact with the content. If you click, there are sparks and other effects,” says Lim.

The capture process can require as little as 30 minutes of actual filming, though there’s a lot of technical work that takes place afterwards.

Lim, who was at Warner Bros for 12 years, joined Yoom because he believes the technology it has developed represents the future of content creation and production.

“The magic is that once we have your digital twin we can then put you in any type of digital environment: we can put you in a game, and we can put you in an AR (augmented reality) experience.

“If you look at the history of content on the internet, we started with text, then we moved to image, and then we moved to video. But videos are a very passive experience. The only interaction you have with video is play, pause, skip forward, go back, share. And that’s it.

Volumetric capture will become increasingly in demand with the advent of the Metaverse and Web3. Courtesy

“I believe that the future is content that is more interactive, so your experience will be different from mine. And this is possible because of the technology we have developed. Because you interact with it, you’re more immersed into the content which means that you’re more engaged.”