Profile PictureBalazs Farago Dev
Open menu

Gaussian Splatting, React Three Fiber

February 7, 2024

Displaying Gaussian Splatting 3D Elements with React Three Fiber

In this tutorial, we'll demonstrate how to display Gaussian splatting using React Three Fiber, a React renderer for three.js that brings 3D graphics into the react ecosystem.

Installing Necessary Packages

POWERSHELL
npm create vite@latest

Choose the react + typescript + SWC

POWERSHELL
npm install three @types/three @react-three/fiber

Due to the project, the following installations are still necessary:

POWERSHELL
npm install @react-three/drei

Main settings

In this section, we'll configure the environment and the containers in which we'll store the objects. We'll set up a basic scene with React Three Fiber to host our Gaussian splatting demonstration. This involves initializing a 3D canvas, setting up lighting, and preparing any additional 3D objects or effects we plan to include in our scene.

PLAIN TEXT
//src/index.css
* {
  box-sizing: border-box;
}


html,
body,
#root {
  width: 100%;
  height: 100%;
  margin: 0;
  padding: 0;
}


body {
  overscroll-behavior: none;
  background: #f0f0f0;
  font-family: "Inter";
}

App.tsx set

The Canvas component creates a 3D rendering space where our scene will appear. The settings within the camera prop define the camera's position, field of view, and rendering depth (near and far clipping planes). Simple glb files are added to the Canvas which will store the GLB and Gaussian splatting objects.

The Environment component uses an HDR (High Dynamic Range) image as the ambient light source, providing more realistic lighting and reflections. The Lightformer components add various shapes and intensities of light sources to the scene, enhancing the visual impact.

The ContactShadows component generates shadows under objects, adding depth and a sense of space to the scene. CameraControls allow the user to interactively move and zoom the camera within the 3D space, increasing the interactivity of the scene.

The Preload component preloads all necessary resources to ensure a smooth user experience and reduce loading times when the user first visits the page.

TYPESCRIPT
//src/App.tsx
import { Canvas } from "@react-three/fiber";
import {
  Preload,
  Lightformer,
  Environment,
  CameraControls,
  ContactShadows,
} from "@react-three/drei";


import Aquarium from "./components/Aquarium";


export default function App() {
  return (
    <Canvas
      dpr={[1.5, 2]}
      camera={{ position: [50, 5, -10], fov: 45, near: 1, far: 300 }}
    >
      <Aquarium position={[-0.1, 10.8, 3]} rotation={[Math.PI / 2, 0, 0]}>
      </Aquarium>
      <Aquarium position={[0, 0, 0]} rotation={[0, 0, 0]}>
      </Aquarium>
      <Environment
        files="https://dl.polyhaven.org/file/ph-assets/HDRIs/hdr/1k/dancing_hall_1k.hdr"
        resolution={1024}
      >
        {/** On top of the HDRI we add some rectangular and circular shapes for nicer reflections */}
        <group rotation={[-Math.PI / 3, 0, 0]}>
          <Lightformer
            intensity={4}
            rotation-x={Math.PI / 2}
            position={[0, 5, -9]}
            scale={[10, 10, 1]}
          />
          {[2, 0, 2, 0, 2, 0, 2, 0].map((x, i) => (
            <Lightformer
              key={i}
              form="circle"
              intensity={4}
              rotation={[Math.PI / 2, 0, 0]}
              position={[x, 4, i * 4]}
              scale={[4, 1, 1]}
            />
          ))}
          <Lightformer
            intensity={2}
            rotation-y={Math.PI / 2}
            position={[-5, 1, -1]}
            scale={[50, 2, 1]}
          />
          <Lightformer
            intensity={2}
            rotation-y={-Math.PI / 2}
            position={[10, 1, 0]}
            scale={[50, 2, 1]}
          />
        </group>
      </Environment>
      <ContactShadows
        smooth={false}
        scale={100}
        position={[0, -5.05, 0]}
        blur={0.5}
        opacity={0.75}
      />
      <CameraControls
        makeDefault
        dollyToCursor
        minPolarAngle={0}
        maxPolarAngle={Math.PI / 2}
      />
      <Preload all />
    </Canvas>
  );
}

The useRef hook creates a reference to the group (group) object, which is used for storing and manipulating all objects within the scene. The useMask hook, on the other hand, creates a stencil mask used for manipulating the materials of objects to achieve special rendering effects, such as displaying transmission materials.

The useLayoutEffect hook allows the component to set the material of each mesh with the mask as soon as the DOM is built, ensuring that the 3D objects appear with the correct material settings. This process dynamically modifies the material properties to match the behavior specified by the stencil mask.

The MeshTransmissionMaterial is a special material provided by @react-three/drei that allows light to pass through the 3D object, creating a glass-like, transparent effect, which is perfect for visualizing an aquarium. This material comes with numerous parameters, such as sampling (samples), thickness (thickness), chromatic aberration (chromaticAberration), and many others, allowing for fine-tuning of the visual effects.

TYPESCRIPT
//src/components/Aquarium.tsx
import { useLayoutEffect, useRef } from "react";
import { MeshTransmissionMaterial, useGLTF, useMask } from "@react-three/drei";


import { Group, Mesh } from "three";
import { GLTF } from "three/examples/jsm/loaders/GLTFLoader.js";


type AquariumProps = JSX.IntrinsicElements["group"] & {
  children: React.ReactNode;
};
interface GLTFResult extends GLTF {
  nodes: {
    [name: string]: THREE.Mesh;
  };
  animations: THREE.AnimationClip[];
}


const Aquarium: React.FC<AquariumProps> = ({ children, ...props }) => {
  const ref = useRef<Group>(null);
  const { nodes } = useGLTF("/shapes-transformed.glb") as unknown as GLTFResult;
  const stencil = useMask(1, false);


  useLayoutEffect(() => {
    if (ref.current) {
      ref.current.traverse((child: THREE.Object3D) => {
        if ((child as Mesh).isMesh) {
          // Now TypeScript knows 'child' is a Mesh, so 'child.material' is safe to access
          const mesh = child as Mesh;
          if (Array.isArray(mesh.material)) {
            mesh.material.forEach((material) =>
              Object.assign(material, stencil)
            );
          } else {
            Object.assign(mesh.material, stencil);
          }
        }
      });
    }
  }, [stencil]);


  return (
    <group {...props} dispose={null}>
      <mesh
        castShadow
        scale={[0.61 * 6, 0.8 * 6, 1 * 6]}
        geometry={nodes.Cube.geometry}
      >
        <MeshTransmissionMaterial
          backside
          samples={4}
          thickness={3}
          chromaticAberration={0.025}
          anisotropy={0.1}
          distortion={0.1}
          distortionScale={0.1}
          temporalDistortion={0.2}
          iridescence={1}
          iridescenceIOR={1}
          iridescenceThicknessRange={[0, 1400]}
        />
      </mesh>
      <group ref={ref}>{children}</group>
    </group>
  );
};


export default Aquarium;

The Aquarium component ultimately creates a group that includes the 3D model and any optionally added child components, allowing users to place additional elements in the aquarium, thus increasing the scene's complexity and interactivity.

Adding gltf and glb Files

The GlbShoe functional component utilizes the shoe model and its materials based on data loaded with the useGLTF hook. The model consists of several different geometric parts (such as laces, mesh, sole, etc.), which are added to the scene as mesh elements individually. Each mesh element receives its own geometry and material, along with a unique color (material-color), allowing for the highlighting of details and customization of the model's overall visual appearance.

The ref hook creates a reference object for the group (group), enabling all meshes within the component to be managed as a single unit. The group's position, scale, and rotation are also set to ensure it is displayed correctly within the scene.

TYPESCRIPT
//src/components/GlbShoe.tsx
import React, { useRef } from "react";
import { Float, useGLTF } from "@react-three/drei";
import { Group } from "three";
import { GLTF } from "three/examples/jsm/loaders/GLTFLoader.js";


const App: React.FC = () => {
  return (
    <>
      <ambientLight intensity={0.3} />
      <Float rotationIntensity={2} floatIntensity={10} speed={2}>
        <GlbShoe />
      </Float>
    </>
  );
};


type GLTFResult = GLTF & {
  nodes: {
    [name: string]: THREE.Mesh;
  };
  materials: {
    [name: string]: THREE.Material;
  };
};


const GlbShoe = () => {
  const ref = useRef<Group>(null);
  const { nodes, materials } = useGLTF(
    "/shoe-draco.glb"
  ) as unknown as GLTFResult;
  return (
    <group
      ref={ref}
      scale={3}
      position={[0, 0, 0]}
      rotation={[-Math.PI / 2, Math.PI / 2, 0]}
    >
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe.geometry}
        material={materials.laces}
        material-color="orange"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_1.geometry}
        material={materials.mesh}
        material-color="red"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_2.geometry}
        material={materials.caps}
        material-color="red"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_3.geometry}
        material={materials.inner}
        material-color="orange"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_4.geometry}
        material={materials.sole}
        material-color="white"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_5.geometry}
        material={materials.stripes}
        material-color="orange"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_6.geometry}
        material={materials.band}
        material-color="orange"
      />
      <mesh
        receiveShadow
        castShadow
        geometry={nodes.shoe_7.geometry}
        material={materials.patch}
        material-color="orange"
      />
    </group>
  );
};


export default App;

The Float component gives the effect of the object gently floating within the 3D space of the application. This behavior can be controlled with two parameters: rotationIntensity (rotation intensity) and floatIntensity (floating intensity), along with the speed parameter, which adjusts the speed of the floating.

The Splat component has scale, position, and rotation props, which allow for customization of its position, size, and rotation in the 3D space. The src prop expects the URL of the texture used for rendering. In this case, the texture source is "https://huggingface.co/cakewalk/splat-data/resolve/main/nike.splat," a URL pointing to a file in a special format containing "splat" data.

TYPESCRIPT
//src/components/GaussianShoe.tsx
import React from "react";
import { Float, Splat } from "@react-three/drei";


const App: React.FC = () => {
  return (
    <>
      <ambientLight intensity={0.3} />
      <Float rotationIntensity={2} floatIntensity={10} speed={2}>
        <Splat
          scale={2}
          position={[4, 3, 1]}
          rotation={[0, Math.PI / 2, 0]}
          src="https://huggingface.co/cakewalk/splat-data/resolve/main/nike.splat"
        />
      </Float>
    </>
  );
};


export default App;

Adding the two new files to our previously created aquarium:

TYPESCRIPT
//src/App.tsx


...


import GaussianShoe from "./components/GaussianShoe";
import GlbShoe from "./components/GlbShoe";


export default function App() {
  return (
    <Canvas
      dpr={[1.5, 2]}
      camera={{ position: [50, 5, -10], fov: 45, near: 1, far: 300 }}
    >
      <Aquarium position={[-0.1, 10.8, 3]} rotation={[Math.PI / 2, 0, 0]}>
        <GlbShoe />
      </Aquarium>
      <Aquarium position={[0, 0, 0]} rotation={[0, 0, 0]}>
        <GaussianShoe />
      </Aquarium>


...

A shoe model stored in GLB format and another produced using Gaussian Splatting technique. The shoe model derived from the GLB file offers detailed and precise rendering, reflecting the high level of precision in digital model creation. This type of model is an ideal choice for applications where detail and visual quality are paramount, such as in virtual reality or high-quality graphics demanding games.

On the other hand, the shoe model created with the Gaussian Splatting technique, while less detailed, provides a fast and efficient method for visualizing 3D objects. This approach can be particularly useful in situations where speed and efficiency are more important than detail, such as in rapid prototyping.

Creating a GLB model typically requires more time and resources, as it must undergo detailed modeling and texturing processes. In contrast, Gaussian Splatting can quickly generate 3D elements with a relatively simple algorithm, allowing for immediate visualization and iteration.

Code

The complete code can be found:

https://github.com/balazsfaragodev/GaussianSplatting-ReactThreeFiber

Share this article

The Newsletter for Next-Level Tech Learning

Start your day right with the daily newsletter that entertains and informs. Subscribe now for free!

Related Articles