Hyperworld: Foundations for the Metaverse

Whatever you do, don't trust me to predict the future.  I didn't think people would shop on the Internet and that hypertext was a useful but boring technology.  So I don't know what the Metaverse will become or what will be "the thing" that makes it real but there are some foundational technologies that will be part of it. Of that, I'm sure.

It's rarely one thing that changes the world.  It's always conflucence of things, some of which seem mundane.  One idea may get all the credit but, behind its success, you can bet there are a host of supporting technologies, without which the "big idea" would not have flourished. You can be too early to the party and it doesn't matter how well you're dressed, you won't get to dance if there is no music and nobody to dance with. 

Hypertext 

To get where we are today, at least three ideas had to merge. First, hypertext allowed words in documents to *link* to other documents. On a standalone computer, this was useful but constrained. The second idea was that these links could be spread across computer using the internet using a universal resource locator (URL).  This gave documents a fixed "address" on the internet. These two alone were not enough though because we needed a formal language with which to create new documents with hypertext and that was HTML, the hypertext markup language. 

The Real and the Virtual

If the Metaverse just turns out to be a video game then it won't be transformative. The central tenent must be to pull down the boundaries between the physical and virtual worlds. We can approch this from both directions. Video game companies are working on making the virtual world more like the real one. But how do we make the real world become more virtual and what does that enable?

The World as a Document

Let's take the analogy of hypertext. I can add a link to the word hypertext that takes you to a document that defines the term and gives you its history. The analogy in the real world would be to look at a chair and be able to follow a "link" from it to find out who made the chair, where I can buy the chair, which of my friends has this chair, etc.  I don't really know what people will do with the ability to link things to the physical world any more than I knew how hypertext would turn into Amazon.com.  But the linking is the key.  Connecting one thing to another thing is a form of power. If you make that linking easy to do, people will find ways of making it useful and ways to make money with it.

Current computer vision technology is getting good enough to enable this and examples already exist.  Today I can take an image of a fashion influencer and automatically identify each piece of clothing and connect the image pixels to shopable links. Another example is X-Ray on Amazon Prime video. Pause a video and X-Ray gives you links to information about all the actors who are on screen. These are like the hyperlink idea without HTML. There is currently no universally accepted language for "parsing" the world and making everything linkable.  This will take off when someone makes it as easy as editing HTML.

The true power of this, however, is unlikely to be realized without augmented reality (AR) glasses.  These are analogous to the Internet -- the physical backbone that supports all the links. Talking about the world as a document today is a little like hypertext before the Internet was widespread. People know this and it is why so many companies are throwing money at building AR hardware. If one company does this, it will be like owning the hardware of the Internet.  

The document analogy only goes so far, however.  Unlike the real world, documents are mostly static. You write the document (web page) and define the links. If it is a shopping site, the document and links are constantly changing but only in very well defined ways. The world is not well defined and how it will change is not known in advance. The real world is a document with constantly changing content. When I interact with a document, I don't fundamentally change it. But the act of living in the world is one of causing change. Every action changes the future just a little. This is very unlike reading a document.

The World as a Video Game

Unlike a document, a video game is dynamic and the future changes based on the actions of a player.  But video games today are like early hypertext systems -- they are closed worlds defined by a single game maker. There is no universal resource locator for game characters or worlds.  There is no HTML that defines games in a platform-independent way.  But most importatnly, there interface between the world of the play and the world of the game is compress through a very low bandwith channel -- the inputs of the game controller. Apart from a few attempts like the Microsoft Kinect, the game knows nothing about the world of the player.  

To bring the game out into the world and turn the world into the game again requires the two technolgies required to turn the world into a document -- computer vision to understand the world and AR to render content into it.

Waadi -- train autonomous cars in simulation.  This is brilliant because they can easily move from cars to anything else.  Just repuprose the world.  

Simulated training data -- datagen, microsoft, apple, meta just bought??

If the world is a document and video game, what does this entail?  

Metaverse or Simulacrum?

From Artificial Intelligence to Artificial Everything

Hyperreality. 

Takeaways

Computer vision. AR.  Define the "markup language".  Make sure its open so people can build on it.