Interaction with Objects

Interactions in natuerlich work through properties, such as onClick. The interaction is based on @coconut-xr/xinteraction. For a more in-depth explanation of the event system and its features, visit the xinteraction documentation.

In the following, we define a box that reacts to the click event and increases the box size by 10%. The useState hook stores the scale of the box, and each call to setScale takes the current state and multiplies it with 1.1. When using the default Controllers and Hands with type="grab", the click event will be triggered by grabbing and releasing the box.

All interactions built with natuerlich work with hands, controllers, and normal mouse and touch controls.

import {
  XRCanvas,
  Hands,
  Controllers
} from "@coconut-xr/natuerlich/defaults";
import { getInputSourceId } from "@coconut-xr/natuerlich";
import {
  useEnterXR,
  NonImmersiveCamera,
  ImmersiveSessionOrigin,
  useInputSources
} from "@coconut-xr/natuerlich/react";
import { useState } from "react";

const sessionOptions: XRSessionInit = {
  requiredFeatures: ["local-floor", "hand-tracking"]
};

export default function Index() {
  const enterAR = useEnterXR("immersive-ar", sessionOptions);
  const inputSources = useInputSources();
  const [scale, setScale] = useState(0.2);
  return (
    <div
      style={{...}}
    >
      <button onClick={enterAR}>Enter AR</button>
      <XRCanvas>
        <mesh
          onClick={() => setScale((s) => s * 1.1)}
          scale={scale}
          position={[0, 1.5, 1]}
        >
          <boxGeometry />
          <meshBasicMaterial color="red" />
        </mesh>
        <NonImmersiveCamera position={[0, 1.5, 4]} />
        <ImmersiveSessionOrigin position={[0, 0, 4]}>
          <Hands type="grab" />
          <Controllers type="grab" />
        </ImmersiveSessionOrigin>
      </XRCanvas>
    </div>
  );
}

Implementing more complex interactions, such as dragging, can be achieved with setPointerCapture. Just as in the web, setPointerCapture allows to capture events until a button is released, allowing the implementation of a custom drag behavior. For more information, visit the event capture documentation of xinteraction. setPointerCapture can be used to implement all kinds of interactions ranging from a 2D simple slider to a 3D steering wheel.

In the following code, we use the onPointerDown listener to capture the state of the box when it is grabbed and then apply the position offset to the object inside the onPointerMove listener.

The example also shows that the dragging interactions work with hands, controllers, mouse, and touch controls.

import {
  XRCanvas,
  PointerHand,
  PointerController
} from "@coconut-xr/natuerlich/defaults";
import { getInputSourceId } from "@coconut-xr/natuerlich";
import { useRef } from "react";
import {
  useEnterXR,
  NonImmersiveCamera,
  ImmersiveSessionOrigin,
  useInputSources
} from "@coconut-xr/natuerlich/react";
import { isXIntersection } from "@coconut-xr/xinteraction";

const sessionOptions: XRSessionInit = {
  requiredFeatures: ["local-floor", "hand-tracking"]
};

export default function Index() {
  const enterAR = useEnterXR("immersive-ar", sessionOptions);
  const inputSources = useInputSources();
  const ref = useRef<Mesh>(null);
  const downState = useRef<{
    pointerId: number;
    pointToObjectOffset: Vector3;
  }>();
  return (
    <div
      style={{...}}
    >
      <button onClick={enterAR}>Enter AR</button>
      <XRCanvas>
        <mesh
          scale={0.1}
          onPointerDown={(e) => {
            if (
              ref.current != null &&
              downState.current == null &&
              isXIntersection(e)
            ) {
              e.stopPropagation();
              (e.target as HTMLElement).setPointerCapture(e.pointerId);
              downState.current = {
                pointerId: e.pointerId,
                pointToObjectOffset: ref.current.position.clone().sub(e.point)
              };
            }
          }}
          onPointerUp={(e) => {
            if (downState.current?.pointerId != e.pointerId) {
              return;
            }
            downState.current = undefined;
          }}
          onPointerMove={(e) => {
            if (
              ref.current == null ||
              downState.current == null ||
              e.pointerId != downState.current.pointerId ||
              !isXIntersection(e)
            ) {
              return;
            }
            ref.current.position
              .copy(downState.current.pointToObjectOffset)
              .add(e.point);
          }}
          ref={ref}
          position={[0, 1.5, 1]}
        >
          <boxGeometry />
          <meshBasicMaterial color="red" />
        </mesh>
        <NonImmersiveCamera position={[0, 1.5, 4]} />
        <ImmersiveSessionOrigin position={[0, 0, 4]}>
          <Hands type="grab" />
          <Controllers type="grab" />
        </ImmersiveSessionOrigin>
      </XRCanvas>
    </div>
  );
}

note

For more examples on more complex interactions including resizing when grabbed by multiple input sources take a look at the Grabbable Implemenation.

---

Question not answered?

If your questions were not yet answered, visit our Discord 😉