Top Podcasts
Ewan Johnson and Andy Stack of Arcturus Studios on NFTs in Volumetric Video and AR, Plus: Quincy Jones NFT Platform, Mark Haines CNBC NFT, Warhol NFT Authenticity and More...
Are volumetric videos the video tech of the future? The founders of Arcturus Studios seem to think so! Co-founders Andy Stack and Ewan Johnson sit down in this information-packed conversation to tell us all about it. They talk about the origins of volumetric video, its practical applications, and how they came about the space as it connects from their past experiences working in the creator economy. They then discuss the intersection of NFTs and volumetric videos in today’s developing NFT space, shedding light on the cryptocurrency ecosystem as well as the future of ownership and asset transfer. Needless to say, this is an exciting time for video technology. Tune in as Andy and Ewan take you deeper into how volumetric videos bring human performance and interaction together.
—
Listen to the podcast here
Ewan Johnson and Andy Stack of Arcturus Studios on NFTs in Volumetric Video and AR, Plus: Quincy Jones NFT Platform, Mark Haines CNBC NFT, Warhol NFT Authenticity and More…
We aim to bring you not only the top 1% of what’s going on with NFTs but also what will stand the test of time. We explore the nuts and bolts in the business side and also the human element of how NFTs are changing the way we interact with the things we love. This show is for the futurist and dreamers, the disruptors and creators, the fans and connectors, and the makers and doers that are pumped about this ecosystem and driving where it goes next.
—
This episode features guests, Ewan Johnson and Andy Stack. Ewan is the Cofounder and Chief Product Officer of Arcturus Studios, the leading global software leader in the post-production and streaming of volumetric video. He has decades of experience telling stories with computer animation and has touched many movies we’ve all seen and love. Ewan was one of the first employees in the film group at Pixar and built their cinematography department. He later spent over a decade at DreamWorks architecting their animation, data pipeline and tools.
Andy is also a Cofounder at Arcturus Studios. He is a serial entrepreneur with three exits and a focus on creator tools and the creator economy. Before Arcturus, Andy spent six years at YouTube leading the teams that built YouTube Analytics and the monetization platform for creators. He produced collaborations between top YouTube talent and Google’s emerging technologies including the first VR 360 videos. Welcome to the show.
Thank you.
It’s great to be here.
Welcome. It’s awesome to have you guys. For all of our readers, what the heck is a volumetric video? Explain this to us.
Sometimes the easiest way to explain it is I tell people it is a three-dimensional photocopy of reality. If you’re looking at the nature of what it is, it is a performance from a person that has been captured from simultaneous viewpoints. That is turned into a full three-dimensional model moving through time that can be played back and viewed from any angle.
Can we bring Tupac back this way? Is Tupac alive again now?
You can construct volumetric videos using existing footage to create these meshes. Our teams specialize in exactly creating a three-dimensional representation of a person from an existing video. Volumetric video created in this stage relies on multiple cameras and shooting performance at the same time. It is a full reproduction of the performance. Whereas when you’re recreating a performer like Tupac from a previously recorded video that doesn’t have viewpoints that are surrounding him, there are massive portions of his performance that don’t exist in those recordings. The computer is making it up using machine learning or Deepfake algorithms. As a result, it isn’t truly his performance. Yes, you can but not as faithfully as when you capture a live performer on a volumetric stage.
Pictures have pixels and videos have voxels. What do volumetric videos have?
That depends upon the technology being used to capture and reconstruct the performance. The most common solution is a result that’s similar to the output of photogrammetry. You have a set of polygons that describe the surface and a set of textures that describe the visual characteristics of that surface. Those are correlated in time so that you have a temporal string of this. As a result, it’s akin to a three-dimensional model that would have been created for an animated film like Toy Story. Instead of being manually created by an artist where they’re painstakingly trying to recreate the appearance of a person, it is the person. It is all of the nuances of the movement, the behaviors, even the subtleties of how their muscles move under the skin.
Can we creep out kids by having their parents be holographic renditions watching them while they’re at work?
Absolutely. I was recorded early on in our life cycle for some sample data because I’m the easiest person in the world to gain the rights to. The funny thing is, that has become one of our core training materials. Anybody who starts to use our software or learns our software gets volumetric versions of me. Whenever we meet with the capture companies that are creating the hardware, every time they see their clients, they run into me because they’ve always got me up on their screen and working with them and testing them. Digital presence everywhere.
I saw the folks at Google are doing something as it relates to conferences where they’re sitting in front of a mirror that looks like you’re sitting in front of the person on the other side of a window or something like that. Is it drawing on similar technology?
It is. This is Project Starlight that they announced. They have started to dive into it. From what I’ve seen from their announcements and papers, it is a combination of volumetric capture techniques where they’re capturing the three-dimensional asset, the textures, and then streaming and distributing that. Also, the display system that helps give you that true sense of presence gets interesting when you’ve got volumetric performances. This is even beyond Starlight. It also dives into a little bit of what we do with our technology. When you can adjust that performance after you’ve recorded it, it can track where the viewer is and then respond to it. That gets fascinating for any type of console.
Andy, you’re rocking the hockey jersey here. I’m thinking about sports, watching sports, and having that interaction, having the same view regardless of where you’re sitting. I’m envisioning a volumetric display that I’m watching these games on.
Sports is an easy vertical to imagine a lot more growth with volumetric video. Imagine not being able to go to an NFL game. Instead, you’re at home and either on your tablet, phone, or even better, with a headset, imagine being able to fly around and choose the point of view wherever you want to be. Like Madden or a video game, you can be up and behind the quarterback. That is now a virtual camera, which has been created by the processing of the ring of cameras that is around the stadium.
There are some exciting things that not only can you as a viewer be able to watch your favorite clip from any angle but there’s also great training capabilities. Imagine Patrick Mahomes. He can relive some particular plays but this time, he can watch how the defenders were coming. He can see that again with a headset on and realize, “This guy was open. This is how it felt when I was getting rushed. I chose to move around this particular time.” He’s in a headset and he can choose to do something else. There are some exciting things with volumetric video working in that direction and a lot of challenges because it’s a lot of data.
That sounds like referees’ worst nightmare, having ten million backseat drivers.
Volumetric Video: We see teams like Facebook Reality Labs really focusing on getting a human capture that can then be turned into an avatar because, the reality is, the nuance of human performance is so subtle and so rich that you really want that connection and interaction.
It’s also a great training program for referees at the same time.
The question is, does the referee get that volumetric 3D view of the whole scene of things to make their calls? Do they have to rely on their traditional senses to fix things out?
They do go to the tape sometimes in 2D. Now, imagine they could put on the headset and they could walk around a frozen moment from a particular second where they had made the call and check from any angle, “Is he out of bounds? Is he in bounds?” To key off of Andy’s example going back to reviewing your play afterward for training, it’s not just a matter of thinking about where the defenders are in lining up but playing with your position on the field and exploring like, “If I had taken two steps to the right, I would have seen that perfect angle, an opening, to be able to get through.”
There’s a great health and safety angle to this as well that illustrates the power of volumetric video. You’re creating digital assets by the photo-real capture of what’s going on. You, as a player, don’t need sensors, in theory, on them because what gets processed then are a series of polygons that are moving around in 3D space and they’re colliding with each other. You can determine the force of impact by the weight of each player and how fast these digital assets are colliding with each other. You can more easily track how someone’s moving around, the force of impact, and be able to study and learn more without sensors. It’s all distilling what the reality captures down to CG assets.
It takes sports analytics to a whole other level. I think about NBA and NFL, to your point, determining the optimal play. It’s amazing.
I was going to say digital assets colliding with each other. That’s only to be learned on the show. Speaking of NFTs, how do you see what you’re doing in the realm of volumetric display and impacting the world of NFTs?
Let’s stick with the sports analogy. We’ve already seen in NFT’s popularity with the NBA Top Shot where it’s capturing moments. What if you’re able to capture the moment in three dimensions and seeing and explore that in that particular play? There’s another interesting element with volumetric video and with holographic video in relationship to this when it comes to also this concept of streaming media. What’s unique about our platform is you don’t have to have the digital assets present on your phone for you to download them. They can be streamed to you but they’re streaming three-dimensional representations. You can look at them in AR and move around them and view them.
Imagine you’re a LeBron James fan and you picked up an NFT of his favorite smack talk moments. You’re watching the game at home. You see him make a great maneuver. You pull out your phone and you send to your friend your NFT LeBron James moment. A notification pops up on his phone, he clicks on it, and then he’s got a full-scale augmented reality LeBron James standing in your living room going, “Gotcha,” and then he’s gone. It was a streamed moment that you own the right to distribute and share with a friend over a short-term temporal instance. It opens up a whole new world of sports smack talk, for sure.
How easy is it to create these and how expensive does it get? Is this mainstream? Is this something where Disney has to write the check for you guys to do your thing?
What’s been exciting over the past few years has been the commoditization of capture technology. There are over nineteen companies producing the hardware to capture volumetric video. In nearly every major city, there’s a capture facility that can go capture them. Prices are still outside the price range for the average person going down to the local mall to grab a volumetric capture but that’s changing. It’s definitely within the price range of teams, elements, and brands. It will be dropping quickly over the next couple of years.
What file type is created that you’re uploading into an NFT platform for example?
That’s another place where we believe we offer a unique solution. Each of the companies creating volumetric capture studios has developed its file format specific to the error stage. It’s like the error of the digital camera back in the early days where you got the digital camera and you got a piece of software that could open the images and you couldn’t do anything without that software. There was no Photoshop. That’s where the volumetric video was before. We offer the ability to work with any of the capture stage’s source media and we provide coding on top of all of our editing tools into what we’d like to become the universal format for volumetric video. It’s called OMS and it stands for Open Mesh Sequence. That is our distribution media for volumetric video whether we’re delivering it embedded into an application or streamed.
We had the guys from SuperWorld on the show and they work with Solidity if I’m not mistaken. There are other platforms we’ve had on the show or we’ll have in the future and ones that our readers are familiar with. Engine is an episode coming up. Is this product compatible with these platforms from the jump? Are there certain platforms that you’re prioritizing in terms of integration? How does this world look if someone wants to collaborate with you guys?
We have Player SDK for incorporation into their platform. If their platform is web-based, then it’s a couple of lines of JavaScript to add. They have the ability to stream volumetric video into web or web AR. We also have plugins for Unity and Unreal if they’re based on game engines. We have native integrations as well. We have an SDK that can be plugged into their application. We’ve specifically designed our streaming protocol as a solution that can be integrated into other people’s platforms. We provide the core backend for storing and distributing through standard content delivery networks as well as security but then allow and enable it to be built into other people’s platforms. Andy, you might have some additional insights.
The SuperWorld example is a good one. They’ve done a fantastic job of creating rich locations. They’ve created the stage on which the X can transpire experiences and so forth. We’re like Central Casting. You’ve set up a wonderful world that allows for some interaction and some transactions. Now you need the cast. Since we help enable the processing and delivery of human performance, it’s an amazing way to then bring in celebrities, your friend, or sports moments into any of these environments.
You guys have such intensive and deep creative backgrounds. One of the themes that we hear about time and again in our episodes is community, story, branding, marketing, bringing those things together and that being the epicenter of these projects that are doing well in the NFT space. You mentioned Top Shot out of the gate, Ewan. It’s a great example. As a user, you don’t even know that you’re operating in the realm of NFTs. You come in, you can use your credit card, you can buy your stuff, you have a marketplace, and you have a community. There’s Discord. How do you guys view that aspect of your business, community, branding, and marketing?
Andy, you should start with this one. You’re such a community person.
A bit of history behind Arcturus is that we started as a young Pixar because we were doing story content creation in a digital realm and also building some technology tools. We all come from strong creative backgrounds in different areas but we had a goal of being able to help establish a rich connection between the viewer and digital characters. How can we bridge that in a story that impacts and affects folks? Through our journey, we’ve experimented with 360 VR in a variety of content that we’ve made in addition to seeing that there’s this opportunity with volumetric video. We started to build out this particular angle because we saw that to be the future.
There’s a growing community around volumetric video and it’s been steadily growing both in terms of awareness and what can be done with that. What’s also interesting is that you can create crowds or communities within virtual worlds by placing a lot of these individual time-based performances. Imagine going to something like The Wave. The Wave has a lot of CG-generated avatars and so forth. They’re coming at it from the CG side to make it look as quite real as possible but it’s still caught in that uncanny valley. Imagine taking the actual photo-real capture of Bruno Mars and then putting him in that environment in which you can draw in a huge community around that particular performance.
Are we all going to have our digital representation of ourselves? Is that where we’re going?
Absolutely. There’s no doubt about that. We’ve already got it with Animojis in many ways and everybody’s custom avatars. We see it with volumetric performance and the distribution of it. We see teams like Facebook Reality Labs focusing on how you get a human capture that can then be turned into an avatar. The reality is the nuance of human performance is subtle and rich that you want that connection and interaction. For us, that’s what drew us to volumetric video. It’s an interesting technology but it’s that nature of human performance and interaction that’s crucial to it.
A story dropped about a new photo app, Paparazzi, that is famous for banning selfies. It goes to a broader point around spicing it up for creators. There are only so many selfies and pictures that people can look at of one individual even if they like them a lot. What are the conversations you’re having with top influencers? What can you offer them when you combine your technology with a concept they might already have in mind in terms of doing an NFT?
What drove me to see the utility in this for influencers was my experience at YouTube working with a lot of top creators but also seeing the burnout that it was causing them, especially at events having to be on all the time. If there was a way to scale their presence with their fans in a way that didn’t drive them crazy or work them into the ground, that could be a phenomenal opportunity. By being able to capture their performance, fans can have an augmented reality version and take a selfie through the AR app or as a short little video. Maybe the fans themselves could then create their own little unique experience with Bruno Mars or with another influencer. Therefore, maybe they can have that NFT locked away. What if that influencer says, “Here’s my volumetric video asset and that’s available only to 100 fans?”
That sure beats the wax museum.
Maybe you have to visit the Rock and Roll Hall of Fame to unlock access to that virtual selfie of Bruno Mars, which can then travel with you and be shared. That starts to get interesting.
All I know is volumetric Jeff is going to have Eathan’s hair, I’ll tell you that.
What we haven’t been led on to yet is that Ewan and Andy are transmitting AI versions of themselves for this podcast interview. It’s still a great interview but they’re off doing something else.
I will borrow that technology.
I wanted to bring this idea that I got excited about. I own a small corner of Washington Square Park in SuperWorld. I own a little piece of the Brooklyn Bridge in SuperWorld. I own a little piece of Millennium Park in Chicago. I’m thinking about capturing breakdancers that you would see on the streets in New York and volumetrically filming a routine that they love to do or whatever. Those guys are always hustling. They got to run around. They’ve got to go from subway car to subway car to do their gig. What if they could always be on for anyone in that park and you could see what they did? That could be something that’s in SuperWorld.
If you can reanimate Josh’s dance moves.
That’s a million-dollar NFT right there.
Sure.
Absolutely.
A week later.
Ewan, how about you?
Does bartering count?
Sure.
Andy, how about you?
He’ll be surprised, I’m sure. Ewan?
How did you go about it?
Craigslist.
How about you, Andy?
It’s Dogecoin.
What’s your most prized possession?
Ewan, how about you?
Come to Puerto Rico with Josh and me.
That would be a trip.
I’d love that.
How about you, Andy?
Ewan, how about you?
It can be the worst. Ewan, how about you?
I use a combination of Monday, and Evernote.
Not in a microwave.
No.
Andy, how about you?
Probably not.
No.
We’ll be standing by.
We leave you on the hook for something here.
It’s our pleasure.
Thanks. It was fantastic.
Important Links:
-
SuperWorld – Previous episode
-
Emma-Jane Mackinnon-Lee – Previous episode
-
@ArcturusXR – Twitter
-
ArcturusXR – Facebook
-
iTunes – Edge of NFT Podcast
About Andy Stack
Andy is also a co-founder of Arcturus Studios.
About Ewan Johnson
He has decades of experience telling stories with computer animation and has touched many movies we’ve all seen and love. Ewan was one of the first employees in the film group at Pixar and built their cinematography department. He later spent over a decade at Dreamworks architecting their animation data pipeline and tools.