How to make real-time multiplayer WebXR experiences

Intro

This is the third blog post detailing how to make real-time multiplayer WebXR experiences.

Having covered the conceptual sides of how it works and the practical sides of how to implement user position data, this blog post will clarify how to implement interactions with 3D models so users can interact with their environment in real time.

If you haven’t already, read the other two posts referenced above and have fun reading the below :D

Code Examples

In the previous blog post, I talk about the XRScene Higher Order Component (HOC) which is declared in the index directory.

We’ll start from there and expand on:

how to declare a 3D models
how to make those 3D models interactive
how to integrate the Websockets with the 3D models

Lets go :D

How to declare 3D models

In order to emit and retrieve a 3D models position data with Websockets, you need to first instantiate the Websockets (as explained in part 2 of this tutorial series) and then declare a 3D model component that is set up to leverage those Websockets.

Lets take a look at how I’ve set them up, by first looking at the index.js file.

The index.js file

See lines 8 and 20–24, which show where the 3D model is loaded.

You can see that we are passing a name, position and rotation property into this component — let’s take a closer look at the Shiba.js component to understand how those are being used.

import Head from 'next/head'
import dynamic from 'next/dynamic';
import React, { useRef, useState, Suspense, lazy, useEffect } from 'react'

import Header from '../components/Header'

const XRScene = dynamic(() => import("../components/XRScene"), { ssr: false });
const Shiba = lazy(() => import("../components/3dAssets/Shiba.js"), {ssr: false});
const Slide = lazy(() => import("../components/3dAssets/Slide.js"), {ssr: false});
const Dome = lazy(() => import("../components/3dAssets/Dome.js"), {ssr: false});

export default function Home() {
  return (
    <>
      <Head>
        <title>Wrapper.js Web XR Example</title>
      </Head>
      <Header />
      <XRScene>
        <Shiba
          name={'shiba'}
          position={[1, -1.1, -3]}
          rotation={[0,1,0]}
        />
        <Dome
          name={'breakdown'}
          image={'space.jpg'}
          admin={true}
        />
        <Slide
          name={'smile'}
          image={'smile.jpeg'}
          position={[-2, 1, 0]}
          rotation={[0,-.5,0]}
          width={10}
          height={10}
        />
        <ambientLight intensity={10} />
        <spotLight position={[10, 10, 10]} angle={0.15} penumbra={1} />
        <pointLight position={[-10, -10, -10]} />
        <spotLight position={[10, 10, 10]} angle={15} penumbra={1} />
      </XRScene>
    </>
  )
}

The 3D model component (Shiba.js)

This component is responsible for rendering the GLTF 3D model, allowing it to be interactive with XR controllers and connecting it to Websockets.

To break this down further:

Rendering the GLTF model: between lines 16–32 is the logic that is auto-generated by an opensource repository that converts GLTF’s into React-Three-Fiber components
Enabling the use of XR interactivity for the model: on line 12 and 39 the Higher Order Component (HOC) withXrInteractivity.js is used to provide Shiba.js with XR interactivity.
Enabling the use of Websockets for the model: on lines 11 and 40 the HOC withCollaboration.js is used to provide Shiba.js with

/*
Auto-generated by: https://github.com/pmndrs/gltfjsx
author: zixisun02 (https://sketchfab.com/zixisun51)
license: CC-BY-4.0 (http://creativecommons.org/licenses/by/4.0/)
source: https://sketchfab.com/3d-models/shiba-faef9fe5ace445e7b2989d1c1ece361c
title: Shiba
*/

import React, { useRef, forwardRef, useEffect } from 'react'
import { useGLTF, useAnimations } from '@react-three/drei'
import withCollaboration from './withCollaboration';
import withXrInteractivity from './withXrInteractivity';

const Model = forwardRef((props, group) => {
  const {name } = props;
  const { nodes, materials } = useGLTF('/shiba/scene.gltf')
  return (
    <group ref={group} {...props} dispose={null} name={name}>
      <group rotation={[-Math.PI / 2, 0, 0]}>
        <group rotation={[Math.PI / 2, 0, 0]}>
          <group rotation={[-Math.PI / 2, 0, 0]}>
            <mesh geometry={nodes.Group18985_default_0.geometry} material={nodes.Group18985_default_0.material} />
          </group>
          <group rotation={[-Math.PI / 2, 0, 0]}>
            <mesh geometry={nodes.Box002_default_0.geometry} material={nodes.Box002_default_0.material} />
          </group>
          <group rotation={[-Math.PI / 2, 0, 0]}>
            <mesh geometry={nodes.Object001_default_0.geometry} material={nodes.Object001_default_0.material} />
          </group>
        </group>
      </group>
    </group>
  )
});

useGLTF.preload('/shiba/scene.gltf')


const InteractiveModel = withXrInteractivity(Model);
const Shiba = withCollaboration(InteractiveModel);

export default Shiba;

How to make those models interactive

In order to enable the use of your XR controllers (e.g Oculus Quest 2 controllers or HoloLens 2 hand tracked controllers etc), you need to implement a library called react-three/xr.

In this example, I’ve implemented this in a HOC called withXrInteractivity.js, lets take a deeper look at it!

Higher Order Component — withXrInteractivity.js

This HOC is responsible for:

Enabling the object to become ‘grabbable’ by the XR controller: lines 40–46 we are using the RayGrab HOC that is provided by react-three/xr, for devices that are capable of XR interactions
Tracking the position values of the model that has been moved: lines 18–26 we are using the useXREvent function from react-three/xr to track the position of where an object has moved to and from
Updating the global app state with the name and positions of the object that has been moved: finally, between lines 28–36 we are figuring out which of the objects with this HOC attached to it have been moved and then update the app’s global state with that object’s name and position values

import React, { useRef, useEffect, useState } from 'react'
import { useThree } from '@react-three/fiber';
import { RayGrab, useXREvent } from '@react-three/xr';
import deviceStore from '../../stores/device';

import selectedObjectStore from '../../stores/selectedObject';
import { Matrix4, Vector3, } from 'three';

const withXrInteractivity = (BaseComponent) => (props) => {
  const { device } = deviceStore();
  const { setSelectedObject } = selectedObjectStore();
  const group = useRef();
  const { scene } = useThree();
  const [oldPosition, setOldPosition] = useState();
  const [newPosition, setNewPosition] = useState()


  if(device != '' && device != 'web') {
    useXREvent('selectstart', (e) => {
      updatePosition(props.name, scene, setOldPosition);
    });

    useXREvent('selectend', (e) => {
      updatePosition(props.name, scene, setNewPosition);
    })
  }

  useEffect(()=> {
    if(oldPosition && newPosition) {
      // if the old positions are not equal to the new positions
      if(oldPosition.x != newPosition.x || oldPosition.y != newPosition.y || oldPosition.z != newPosition.z) {
        // then you know this object has just been updated, execute logic to update websockets and analytics
        selectedObject(props.name, scene.getObjectByName(props.name), setSelectedObject, group);
      }
    }
  },[oldPosition, newPosition])

  return (
    <>
        {device != '' && device != 'web' &&
          <RayGrab>
              <BaseComponent
                ref={group}
                {...props}
              />
          </RayGrab>
        }
        {device != '' && device == 'web' &&
          <BaseComponent
          ref={group}
          {...props}
          />
        }
    </>
  )
};

const updatePosition = (name, scene, setPosition) => {
  let pos = new Vector3();
  let tempMatrix = new Matrix4;
  tempMatrix = scene.getObjectByName(name).matrixWorld;

  // set the oldPosition based on the matrix world positions
  pos.setFromMatrixPosition(tempMatrix);
  setPosition(pos)
};

const selectedObject = (objectname, object, setSelectedObject, group) => {
  let { x, y, z } = object.position
  setSelectedObject({
    objectname: objectname,
    position: {
      x:x,
      y:y,
      z:z
    },
    group: group
  });
}

export default withXrInteractivity;

How to integrate the Websockets with the 3D models

Once we’ve made the model interactive with the XR controllers, we then need to integrate the use of Websockets to allow other users to see the new position of the model you’ve moved.

There are two sides of this to understand, the first is the withCollaboration.js HOC that is responsible for the actual Websocket integration on the Front End and the second is what the data actually looks like in DynamoDB.

Higher Order Component — withCollaboration.js

Starting with the Front End, withCollaboration.js does the following:

Submits the positions of the object you’ve just moved with your XR controllers to the Websocket: lines 19–25 check if you’re device is capable of XR interactivity, if so then it will update the Websocket with the name of the model being moved, as well as the position of that moved model and the name of the user which moved it
Listens to the Websocket for any updates and updates the positions of the moved models: lines 28–32 listen for updates on the Websocket and then check if the submission was made from a different user from the one that submitted it, if so then the new position of the model is applied to the scene.

Building on that last bullet point, the reason it needs to check if the user that submitted the movement is different from the user receiving the positional data on the web app, is to prevent bouncing of position values for the user that originally submitted the new position of the model.

For example, if you submit position values to the Websocket, the Websocket would immediately detect that you had moved the model and would would try to update the position of the model you’ve just moved — this can cause confusion at the Three.JS layer and cause the position of the model to move erratically and become buggy.

import React, { useRef, useEffect, useState } from 'react'
import { useThree } from '@react-three/fiber';
import { RayGrab, useXREvent } from '@react-three/xr';
import deviceStore from '../../stores/device';
import socketStore from '../../stores/socket';
import cognitoStore from '../../stores/cognito';
import selectedObjectStore from '../../stores/selectedObject';
import { Matrix4 } from 'three';

const withCollaboration = (BaseComponent) => (props) => {
  const { device } = deviceStore();
  const { selectedObject } = selectedObjectStore();
  const { sendJsonMessage, lastJsonMessage } = socketStore();
  const { cognito } = cognitoStore();
  const { scene } = useThree();
  const [socketMode, setSocketMode] = useState('initialLoad');


  useEffect(()=> {
    if(device != '' && device != 'web') {
      if(selectedObject.objectname) {
        submitPositionsToCloud(selectedObject.objectname, cognito.username, scene.getObjectByName(selectedObject.objectname), sendJsonMessage);
      }
    }
  },[selectedObject])


  useEffect(()=> {
    if(device != '') {
      updateModelFromWebSockets(lastJsonMessage, props.name, cognito, scene.getObjectByName(props.name), socketMode, setSocketMode);
    }
  }, [lastJsonMessage])

      return (
        <BaseComponent
          {...props}
        />
      )
};

const submitPositionsToCloud = (objectname, username, object, sendJsonMessage) => {
  let newData = {
    type: 'objects',
    uid: objectname,
    data: {
      submittedBy: username,
      matrixWorld: object.matrixWorld
    }
  };
  sendJsonMessage({
    action: 'positions',
    data: newData
  });
}

const updateModelFromWebSockets = (lastJsonMessage, name, cognito, group, socketMode, setSocketMode) => {
  let data;
  if(lastJsonMessage != null){
    for(let x=0; x<lastJsonMessage.length; x++) {
      if(lastJsonMessage[x].uid == name) {
        if(lastJsonMessage[x].data.matrixWorld.length != 0) {
          data = lastJsonMessage[x].data;
        }
      }
    }
  }
  if(data) {
    if(socketMode == 'stream' && data.submittedBy != cognito.username || socketMode == 'initialLoad') {
      let tempMatrix = new Matrix4();
      tempMatrix.copy(data.matrixWorld);
      tempMatrix.decompose(group.position, group.quaternion, group.scale)
      if(socketMode == 'initialLoad') {
        setSocketMode('stream');
      }
    }
  }
}

export default withCollaboration;

Database for Websocket positional data

Having submitted the new position values of the model to the Websocket, this is then passed onto DynamoDB through the same process outlined in part 2 of this tutorial series.

I’ve attached a screenshot below of the DynamoDB table, which shows two types of entries: objects and users.

Users refers to people who have logged in and have moved around, their position values are stored there (see part 2 of this tutorial series).

Objects refers to the models that users have moved using their XR controllers as described in this blog post, let’s take a deeper look at this.

Screenshot of the positional data in the DynamoDB Table

Looking at an example of the model entry that is stored, you can see that the models name is stored (in this case ‘shiba’) as well as the position of the model (called ‘matrixWorld’) and the name of the user that submitted the new position.

The reason that we stored the position as a ‘matrixWorld’ instead of x/y/z value, is down to the complexities of how a model’s position is changed when you move it using an XR controller.

When you ‘grab’ a model using an XR controller, that model becomes a child of the controller, until the moment that you release that model.

In order to provide other users with the accurate position of where you’ve moved the model to, we are storing that models ‘matrixWorld’ position, which is essentially its global position — not its position in relation to the XR controller (aka not its position as a child of the XR controller).

{
  "type": {
    "S": "objects"
  },
  "uid": {
    "S": "shiba"
  },
  "data": {
    "M": {
      "matrixWorld": {
        "M": {
          "elements": {
            "L": [
              {
                "N": "-0.8743998152939527"
              },
              {
                "N": "-0.23723998239253116"
              },
              {
                "N": "-0.4231857431325429"
              },
              {
                "N": "0"
              },
              {
                "N": "-0.23408684389239517"
              },
              {
                "N": "0.9703077655463146"
              },
              {
                "N": "-0.0602815775361555"
              },
              {
                "N": "0"
              },
              {
                "N": "0.4249385470147294"
              },
              {
                "N": "0.046353861928324164"
              },
              {
                "N": "-0.9040077143825459"
              },
              {
                "N": "0"
              },
              {
                "N": "0.6380046282561553"
              },
              {
                "N": "1.0890667315312834"
              },
              {
                "N": "-4.233745217744318"
              },
              {
                "N": "1"
              }
            ]
          }
        }
      },
      "submittedBy": {
        "S": "hi@jamesmiller.blog"
      }
    }
  }
}

Conclusion

Now THAT is a lot of information!!!

I really hope this has been helpful in enabling you to make real-time multiplayer WebXR experiences using React-Three-Fiber and Websockets :D

With any luck, you will be able to adapt this code for your app — or even better use Wrapper.js to build your WebXR experiences!

I hope this is helpful and in the meantime — have fun :D

How to make real-time multiplayer WebXR experiences — part 3

Using Websockets, React Three Fiber and DynamoDB to allow multiple users to interact with 3D models in WebXR.

Table of contents

Intro

Code Examples

How to declare 3D models

How to make those models interactive

How to integrate the Websockets with the 3D models

Conclusion

How to make real-time multiplayer WebXR experiences — part 3

Using Websockets, React Three Fiber and DynamoDB to allow multiple users to interact with 3D models in WebXR.

Table of contents

Intro

Code Examples

How to declare 3D models

How to make those models interactive

How to integrate the Websockets with the 3D models

Conclusion

Did you find this article valuable?