A Framework for the Metaverse from Hardware to Hollywood and Everything in Between

The Metaverse is a thing. It is being built around us and though its final, fullest manifestation is decades away the fight for a piece of the trillions of dollars it is expected to yield has begun.

There’s no better way to get to grips with the Metaverse — where it’s come from, where it is now and where it’s going to — than reading the essays of Matthew Ball, a venture capitalist and former global head of strategy for Amazon Studios.

“The Metaverse Primer,” a series of nine blog posts co-authored by Jacob Navok, co-founder and CEO of Genvid Technologies, outline the state of knowledge about the Metaverse around topics like hardware, content and behavior.

This article provides a primer to Ball’s primer and offers a way to think about the emergence of the Metaverse.

Definition: What is the Metaverse?

Often mis-described as virtual reality, the Metaverse is a sort of successor state to the mobile internet. Decades from now it will be defined by characteristics such as infinite persistency, ubiquitous synchronicity, unlimited concurrency and wide interoperability but the way we will get there is through sustained and interconnected, improvements in numerous areas.

Based on precedent, however, we can guess that the Metaverse will revolutionize nearly every industry and function, writes Ball.

“From healthcare to payments, consumer products, entertainment, hourly labor, and even sex work. In addition, altogether new industries, marketplaces and resources will be created to enable this future, as will novel types of skills, professions, and certifications. The collective value of these changes will be in the trillions.”

READ MORE: A Framework for the Metaverse (Matthew Ball)

Hardware

Physical devices are used to access, interact with, or develop the Metaverse and the capabilities are getting better all the time. Ball cites a couple of examples. Apple’s Object Capture enables users to create high-fidelity virtual objects using photos from their standard-issue iPhone in a matter of minutes. Many new smartphones also feature ultra-wideband chips that emit 500,000,000 RADAR pulses per second and receivers that process the return information.

Both technologies enable users to map their world (objects, a street) and transplant it into other virtual environments, “thereby reducing the cost and increasing the fidelity of synthetic goods, or overlayed into real environments for the purpose of art, design, and other AR experiences.”

The latest XR headsets also achieves up to 120hz which is the minimum threshold for avoiding disorientation and nausea in some users. Ideally this would be achieved without needing a gaming-level PC and tether, but we’re getting there.

The hardest technology challenge of our time may be fitting a supercomputer into the frame of normal-looking glasses. But it’s the key to bringing our physical and digital worlds together.

— Mark Zuckerberg (@MarkZuckerbelg) May 2, 2021

While humans can see an average of 210 degrees, Microsoft’s HoloLens 2 display covers only 52 degrees (up from 34). Snap’s forthcoming glasses are only 26.3 degrees. To take off, we likely need far wider coverage, Ball thinks. These are primarily hardware challenges, not software ones.

Outside of consumer-grade hardware, the ability to map the real world is becoming a major source of IP. Leica’s $20,000 photogrammetric cameras have up to 360,000 laser scan set points per second, which are designed to capture entire malls, buildings, and homes with greater clarity and detail than the average person would ever see in person. Epic Games’ Quixel, meanwhile, uses proprietary cameras to generate environmental “MegaScans” comprised of tens of billions of pixel-precise triangles.

“This dynamic explains why companies such as Epic and Unity choose to buy and build up real world scans, rather than build from zero. In the coming years, it’s likely we’ll see intense competition in the category, with the likes of Nvidia, Autodesk, Facebook, Snap, and Niantic all choosing to build up their databases — just as Apple and Uber eventually moved off from the Google Maps platform.”

— The Metaverse Primer

“These devices make it easier and cheaper for companies to produce high-quality ‘mirror worlds’ or ‘digital twins’ of physical spaces, as well as use scans of the real world to produce higher-quality and less-expensive fantasy ones.

“This dynamic explains why companies such as Epic and Unity choose to buy and build up real world scans, rather than build from zero. In the coming years, it’s likely we’ll see intense competition in the category, with the likes of Nvidia, Autodesk, Facebook, Snap, and Niantic all choosing to build up their databases — just as Apple and Uber eventually moved off from the Google Maps platform.”

Networking

Ball acknowledges that the three core areas of networking — bandwidth, latency and reliability — are likely to be least interesting Metaverse-enablers to most readers. Nonetheless, as the complexity and importance of virtual simulation grows, the amount of data that needs to be streamed will increase.

“If we want to interact in a large, real-time, shared, and persistent virtual environment, we will need to receive a superabundance of cloud-streamed data,” Ball says. “Cloud data streaming is also essential if we want to seamlessly jump between different virtual worlds.”

The biggest challenge in networking is also its least understood: latency. The most immersive AAA online multiplayer games require low latency today. This is because latency determines how quickly a player receives information and how quickly their response is transmitted to other players.

“If we want to interact in a large, real-time, shared, and persistent virtual environment, we will need to receive a superabundance of cloud-streamed data. Cloud data streaming is also essential if we want to seamlessly jump between different virtual worlds.”

— The Metaverse Primer

“Latency, in other words, determines whether you win or lose, kill or end up killed. Yet while the Metaverse isn’t a fast-twitch AAA game, its social nature and desired importance means it will require low latency. Slight facial movements are incredibly important to human conversation — and we’re incredibly sensitive to slight mistakes and synchronization issues (hence the uncanny valley problem in CGI).”

Unfortunately, latency is the hardest and slowest to fix of all network attributes. Part of the issue stems from how few services and applications need ultra-low latency delivery. This constrains the business case for any network operator or latency-focused content-delivery network — and the business case here is already challenged and in contention with the fundamental laws of physics.

Ball says we should think of the Metaverse as raising the requirements for all aspects of networking — latency, reliability/resilience, and bandwidth — to that of AAA multiplayer games. It doesn’t matter how powerful your device is, he argues, if it can’t receive all the information it needs in a timely fashion.

Compute

Just as the virtual world is augmented, so too will be the “real” one. Every year, more sensors, cameras, and IoT chips will be integrated into the physical world, many of connected in-real time to a virtual simulacra that can interact back. Meanwhile, our personal devices will serve as our passports to, and part-time generator of, many of these experiences. In short, much of the world around us will be continuously interconnected and online. Including us.

In totality, the Metaverse will have the greatest ongoing computational requirements in human history. And compute is, and is likely to remain, incredibly scarce. As a result, the availability and development of computing power will constrain and define the Metaverse. It doesn’t matter how much data you can receive, or how quickly, or why, if it can’t be used.

Edge compute is considered a key infrastructure, either in servers at the edge of the network, or in the increasingly powerful mobile phones in our pocket. But even if we improve the computing power of consumer devices, move more enterprise computing power closer to consumers, and build out more centralized infrastructure, we’re still likely to fall short.

“In totality, the Metaverse will have the greatest ongoing computational requirements in human history. And compute is, and is likely to remain, incredibly scarce. As a result, the availability and development of computing power will constrain and define the Metaverse. It doesn’t matter how much data you can receive, or how quickly, or why, if it can’t be used.”

Ball’s idea is that a form of peer to peer networking will take place in which the available compute power of every local PC and device will be used to sate demand. Owners will be paid for the use of their compute power. He thinks this possible because the transaction will be conducted by blockchain.

In this conception, owners of underutilized CPUs and GPUs would be “paid” in some cryptocurrency for the use of their processing capabilities, perhaps by users located ‘near’ them in network topology. There might even be a live auction for access to these resources, either those with “jobs” bidding for access, or those with capacity bidding on jobs.

“Every computer, no matter how small, would be designed to always be auctioning off any spare cycles. Billions of dynamically arrayed processors will power the deep compute cycles of even the largest industrial customers and provide the ultimate and infinite computing mesh that enables the Metaverse.”

Virtual Platforms

For the foreseeable future, most of us will interface with the Metaverse via consumer-facing, interactive and immersive virtual platforms. These will be accessible from a variety of devices, such as a web browser and VR headset, and it’s here that we’ll hang out, buy things and show them off, collaborate and build, learn and perform, work and relax, consume in and experience the Metaverse.

The most popular current examples are Roblox and Minecraft, but a number of virtual platforms aspire to displace them through the use of blockchain. These include Decentraland, The Sandbox, Cryptovoxels, Somnium Space and Upland, and dozens more are in development.

A virtual Metaverse platform must have the technical ability for (relatively-unbound) creation (engine + studio + tools), services to support it (prefabs and asset marketplaces, voice chat, player accounts, payment services), and operate a multifaceted economy (i.e. consumer spending that’s shared with on-platform creators/developers, as well as creator/developer-to-creator/developer revenues). In success, these platforms generate a virtuous circle.

“As the world shifts to mirrorworlds and simulation technology, it becomes possible to interconnect previously independent simulations. Imagine interconnecting the Hong Kong International Airport to the local highway to scenario-test the flow of traffic. Then to the streetlights system that manages that traffic. Potentially with precise information on every car on the local grid.”

— The Metaverse Primer

Crucially, these requirements are not limited to games. Niantic, for example, is working to develop a “planet-scale augmented reality platform for current and future generations of AR hardware.” Facebook is working on its own VR and AR platforms, while Microsoft continues to pursue the same opportunity via its HoloLens platform (which integrates into its Minecraft virtual world). Nvidia’s Omniverse is another great example, suggests Ball. The service helps businesses bring together various digital assets, irrespective of their formats or engine, into a single virtual environment.

“As the world shifts to mirrorworlds and simulation technology, it becomes possible to interconnect previously independent simulations. Imagine interconnecting the Hong Kong International Airport to the local highway to scenario-test the flow of traffic. Then to the streetlights system that manages that traffic. Potentially with precise information on every car on the local grid.”

What’s key to Omniverse is that it can do this irrespective of the file formats and engine/simulation technologies being used. In other words, everything doesn’t have to be on Unity, or Unreal, or AutoCAD. And while Omniverse is, today, intended for design and testing, one can imagine Nvidia using this technology, plus its own industrial computing power, to operate much of the overall mirrorworld live.

Tools + Standards

The very premise of the Metaverse presumes that the “next” platform shifts from hardware-based operating systems and hardware-centric experiences to persistent and ubiquitous virtual simulations. We already see this today. The most popular games in the world, such as Fortnite, Roblox, and Minecraft, are designed to run on all endpoints, and none requires nor succeeds because of specific hardware (nor are they optimized for any specific devices, either).

Interchange standards and tools, protocols, formats, and services which enable are perhaps the most important aspect of the entire Metaverse framework.

“Without them, there will be no Metaverse — only a more virtual and immersive version of today’s mobile internet and app stores. What’s more, this pale imitation will be far less lucrative, dynamic, and healthy. It will make it harder for new platforms to emerge — and, frankly, for the Metaverse to be built.”

“All of this creates a burden on developers, and potentially a vicious circle, whereby no platform has enough users to develop for, and no platform has enough content to attract users. There’s a reason you can’t export an experience from Roblox to Minecraft or Fortnite, but just as you can’t easily import all your Instagram photos and likes into Twitter or TikTok or Snapchat.”

— The Metaverse Primer

For example, we want as much of the world to integrate into the Metaverse as possible. This means interconnecting the many devices and platforms around us today, from your car and home-security camera, to VR and AR headsets, projection cameras and screens, wearables, and more. Such development will require, or at least benefit from, the use of proprietary standards.

Facebook is investing heavily into XR specifically, so that it can establish its own OS, Ball says, to fight back against the gatekeeping of today’s mobile operating systems, and avoid using a standard created by its direct competitors, such as Snap.

“All of this creates a burden on developers, and potentially a vicious circle, whereby no platform has enough users to develop for, and no platform has enough content to attract users. There’s a reason you can’t export an experience from Roblox to Minecraft or Fortnite, but just as you can’t easily import all your Instagram photos and likes into Twitter or TikTok or Snapchat.”

But there is one major interchange technology that, while not open in the traditional sense, retains most of the values and benefits of an open standard, and also looks likely to thrive in the Metaverse: blockchain.

Payments and Blockchain

For many, the idea of the Metaverse is not just intertwined with blockchain, it fundamentally requires it. To others, this is absurd. A decentralized database isn’t needed to prove or manage asset ownership (if the NBA or Disney said who owned an image file or virtual currency, everyone would believe them). Nor is it needed to safely and quickly transfer money (Alipay and PayPal move billions per day via purely digital networks).

“What matters is whether blockchain standards can more effectively enhance developer profits over time, and in turn, grow the Metaverse economy.”

Today, only a tiny fraction of online users and gamers even have a crypto wallet, and almost no brands and games issue NFTs (non-fungible token). But more of these groups are embracing blockchain-based experiences each month. This produces a virtual cycle that drives more users to register a wallet, mint an NFT, or integrate crypto assets, which also increases the value and utility of all other blockchain products, thereby bringing more to the technology.

“We should think of the Metaverse as raising the requirements for all aspects of networking — latency, reliability/resilience, and bandwidth — to that of AAA multiplayer games. It doesn’t matter how powerful your device is if it can’t receive all the information it needs in a timely fashion.”

“Blockchain-based revenues offer major publishers not just the potential for greater revenues, but a system that is equivalently fair and impossible for a competitor to manipulate. It is hard to imagine a better on ramp to openness.”

Why so many in the Metaverse community find blockchain technology so compelling is the fact that cryptocurrencies are essentially programmable payment rails. Bitcoin, for example, is programmed to automatically compensate those who operate its network using its own cryptocurrency.

“[Blockchain is] an indisputably neutral, distributed way of expressing individual ownership … the most plausible path towards an ultimate long-term open framework where everyone’s in control of their own presence, free of gatekeeping,” says Epic Games founder Tim Sweeney.

Content, Services, Assets

Ball says the companies that own the most beloved IP and brands will have a major role in the Metaverse. The largest activations in Fortnite to date have been Marvel, followed by Star Wars and the NFL, and the NBA continues to lead in NFTs. These sorts of franchises have endured for decades specifically because they work everywhere and dominate our endless imaginations and fandoms.

But in a sense current virtual movie theaters, virtual concerts, virtual XR theme parks are all fairly incremental.

“It will take time for the entertainment industry to figure out what a Metaverse radio show, TV series or porn site looks like. But the first companies, artists, and stories to do so will find themselves situated beside the likes of Captain America and Luke Skywalker.”

“As transformative as Travis Scott’s [Fortnite] concert was, it wasn’t live motion captured, it didn’t truly involve the audience, and it only reached fifty people at a time. What happens when this changes? What happens when, rather than make a titan-sized Travis Scott sitting on an outer space planet, he terraforms the planet as a god-like titan? Or if he’s a motion capture version of the planet!?”

It will take time for the entertainment industry to figure out what a Metaverse radio show, TV series or porn site looks like. But the first companies, artists, and stories to do so will find themselves situated beside the likes of Captain America and Luke Skywalker.

Evolving User & Business Behaviors

The most obvious behavioral change of the past year has been the increasing amount of time we spent online and in virtual worlds. Millions of skeptics have now participated in (and enjoyed) virtual worlds and activities such as Animal Crossing, or Roblox as they sought out things to do, attended events once planned for the real world, or tried to spend time with their kids indoors. Not only has this destigmatized virtual life, and ‘the Metaverse,’ but it might even mean an extra generation will participate in it.

Many businesses are also shifting their technology processes, pipelines, and investments to support the Metaverse. Hollywood is rapidly shifting productions over real-time rendering solutions such as Unreal and Unity, or developing their own engine, as Disney has with ILM’s Helios.

“Not only does this offer greater creative flexibility and more efficient shooting timelines, but it enables Hollywood to produce “virtual backlots.” Just as “The Disney Lot” is full of Spider-Man costumes, Brooklyn streets, infinity stones, and stored sets, Disney is amassing a virtual archive of digital assets — from the city of Nevarro and Ancient Mandalore, to Din Djarin’s Razor Crest, and Baby Yoda himself. Building operators, architectural firms, and automotive companies are also shifting to tools like Unreal or Unity, while previously vertical specific solutions like AutoCAD are moving into rich simulation.

“To participate in these experiences, brand owners must embrace something they almost never allow in an activation or marketing partnership: unlimited-term licenses, overlapping marketing windows, and no real control over editorial. Collectively, this means it’s now possible to dress as Neymar while wearing a Baby Yoda or Air Jordan backpack, holding Aquaman’s trident, and exploring Stark Industries. And the owners of these franchises want this to happen.”

The impact here is difficult to underestimate. The Metaverse needs to be populated with virtual content, but this content is costly to produce and will often lack an immediate short-term business case. Many of us might want to bike the planet Endor on Zwift, but producing this is expensive. If Disney has already made it, however, it just requires tailoring. In addition, Disney’s ability to leverage its virtual Endor should lead to a greater investment in its quality, too.

Ball also believes the growing transfer of assets, and currencies between virtual experiences will push the entertainment industry towards more openness to cross-pollinating its IP.

But to participate in these experiences, brand owners must embrace something they almost never allow in an activation or marketing partnership: unlimited-term licenses, overlapping marketing windows, and no real control over editorial. Collectively, this means it’s now possible to dress as Neymar while wearing a Baby Yoda or Air Jordan backpack, holding Aquaman’s trident, and exploring Stark Industries. And the owners of these franchises want this to happen.

Hollywood long believed that millennials would grow into the linear pay TV bundle or lose interest in YouTube. They never did. And while they now watch Netflix and Disney+, video’s share of leisure time is going down on a generational basis.

Today’s generation of children express themselves, often learn, and constantly socialize through virtual worlds they can touch, change, and collaborate in. That’s not going to stop. Rather, the capabilities of these virtual worlds will expand, their ease of use will improve, and their significance will grow.

“What’s more, the ‘iPad Native’ generation (or perhaps ‘virtual world native’ generation) will continue to mature. Most are still consumers, a few are creators, and almost none are business leaders. They will be. And their frames of reference will lead to transformative change.”

Spectacular Minecraft Creations Years in the Making

Ready to explore more of the Metaverse? Dive into this roundup, via Wired, of some of the most spectacular Minecraft creations to date. Creative and dedicated, the master builders of these creations often work in teams, crafting reconstructions of real-life buildings, fantasy or sci-fi settings, and even whole cities.

From Middle Earth and Games of Thrones to meticulously designed imaginary American cities and vast library collections of censored books, here are some of the most mind-boggling creations from Minecraft users:

Minecraft Middle-Earth: This map of Middle-Earth’s Minas Tirith, which has been around nearly as long as Minecraft itself, now covers an area equivalent to Dallas, Texas.

Mattupolis: This imaginary modern city combines elements of Vancouver and Seattle, with references to New York sprinkled in. Devised and constructed by Finnish architecture student Matias, “Mattupolis” has been redesigned and reorganized several times since its original release more than a decade ago.

Greenfield: Inspired by Los Angeles, this enormous fictional city is extraordinarily detailed. Its vast scale and realistic-looking buildings with fully modeled interiors include everything from beautiful beachfront property to industrial zones. Designed on a 1:1 to scale, it provides an authentic metropolitan feel ripe for exploration.

The Uncensored Library: A vast, and continually updating digital library, this Minecraft build features news from countries like Saudi Arabia, Russia, Vietnam, and Egypt.

WesterosCraft: Under construction for the better part of a decade, this map of places included in George R.  R. Martin’s fantasy series covers a whopping 485 square miles. While the series has ended, WesterosCraft still has an active community of fans coming together with the aim of building all 500 cities, castles and other landmarks that encompass Westeros.

Cyberpunk: inspired by the gorgeous, futuristic city of Cyberpunk 2077, this Minecraft build is from a team in France. It features an immense glowing megacity densely packed with incredibly tall skyscrapers awash in neon lights, billboards and holograms.

READ MORE: These Astonishing Minecraft Builds Were Years in the Making (Wired)