3D Gaussian Splatting: How It Works, Its History, and Key Uses

3D Gaussian Splatting is a scene-reconstruction and rendering method capable of producing high-quality, navigable views of real environments from collections of overlapping photographs or video frames. It has attracted attention across computer graphics, gaming, architecture, robotics, simulation, virtual production, and digital twins because it can render captured scenes interactively without first converting every visible surface into a conventional polygon mesh.

The technique became widely recognized after the publication of the 2023 research paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. Their method combined an explicit collection of optimized 3D Gaussians with a visibility-aware rendering process to achieve high visual quality and real-time novel-view synthesis.

That does not mean Gaussian splatting replaces conventional 3D modeling, BIM, photogrammetry, laser scanning, or architectural rendering. It is a different representation with specific strengths and limitations. Understanding those distinctions is essential before using it for professional design, construction, or spatial-documentation work.

What Is 3D Gaussian Splatting?

3D Gaussian Splatting, commonly shortened to 3DGS, represents a captured scene using many three-dimensional Gaussian primitives.

Each Gaussian can be understood as a soft, oriented volume located within three-dimensional space. Rather than behaving like a hard geometric point, it has properties that influence how it appears from different camera views. These properties typically include:

• A three-dimensional position.

• Scale along multiple axes.

• Rotation or orientation.

• Opacity.

• Color information.

• View-dependent appearance.

The Gaussians are often anisotropic, meaning they can stretch and rotate into ellipsoidal forms instead of remaining uniform spheres. During optimization, they are adjusted to collectively reproduce the appearance recorded in the source images.

When the scene is rendered, those Gaussians are projected onto the screen and blended in a visibility-aware order. Millions of these overlapping contributions can form a visually continuous environment that appears detailed and photorealistic from supported viewpoints.

Why Is It Called Gaussian Splatting?

The name combines two technical ideas.

A Gaussian is a smooth mathematical distribution commonly associated with the bell-shaped curve. In 3D Gaussian Splatting, the distribution exists in three dimensions and can be stretched, rotated, colored, and assigned opacity.

Splatting describes the process of projecting a sample onto the image plane and spreading its contribution across nearby pixels rather than rendering it as a single infinitesimal point. The projected footprint is blended with the footprints of other samples to produce the final image.

The term is therefore not based on the sound of a physical object hitting a surface. It refers to a long-established rendering operation in which samples distribute—or “splat”—their influence over an image.

(Carl Friedrich Gauss)

Who Invented Gaussian Splatting?

There is no single answer covering every form of Gaussian splatting.

The broader splatting concept has roots in earlier volume-rendering research. Lee Westover’s 1990 paper Footprint Evaluation for Volume Rendering described a forward-mapping rendering approach in which volumetric samples distribute their contributions onto an image plane.

Modern 3D Gaussian Splatting for radiance-field rendering was introduced by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis in their SIGGRAPH 2023 paper. The work was developed through Inria, Université Côte d’Azur, and the Max Planck Institute for Informatics.

The researchers did not simply rename traditional point rendering. Their contribution combined an explicit 3D Gaussian scene representation, adaptive density control, anisotropic covariance optimization, and a fast visibility-aware rasterization method.

The authors also released an official 3D Gaussian Splatting implementation, allowing researchers and developers to reproduce and extend the work.

How Does 3D Gaussian Splatting Work?

A typical Gaussian-splatting workflow contains several connected stages.

1. Capture Overlapping Images

The process begins with photographs or video frames showing the same environment from multiple viewpoints.

Good coverage is important. The camera should move through the space systematically while maintaining overlap between adjacent images. Blurred images, rapid exposure changes, moving objects, reflective surfaces, and missing viewpoints can weaken the reconstruction.

Outdoor capture may also be affected by moving vegetation, vehicles, pedestrians, harsh shadows, and changing sunlight. Interior capture can be affected by mirrors, glass, repetitive walls, narrow rooms, and limited lighting.

2. Estimate Camera Positions

The system must determine where each photograph was taken and how the camera was oriented.

This step is commonly performed through Structure from Motion. The process identifies matching visual features across multiple images, estimates camera parameters, and reconstructs an initial sparse set of 3D points.

COLMAP, a widely used open-source Structure-from-Motion and Multi-View Stereo pipeline, is commonly used within research and production workflows for camera estimation and sparse reconstruction.

If the software cannot establish reliable image relationships, the remaining Gaussian-splatting process will not have a stable spatial foundation.

3. Initialize the Gaussian Representation

The sparse points recovered during camera calibration can provide initial positions for the Gaussians.

Each Gaussian begins with properties such as position, size, orientation, opacity, and color. The initial representation is incomplete and does not yet reproduce the captured scene accurately.

4. Optimize the Scene

The system renders views of the current Gaussian representation and compares them with the original photographs. It then adjusts the Gaussians to reduce the difference between the rendered and reference images.

During this process, Gaussians may be:

• Repositioned.

• Rescaled.

• Rotated.

• Made more or less opaque.

• Assigned refined color information.

• Added where more detail is needed.

• Removed where they do not contribute effectively.

This iterative optimization allows the representation to develop around the visible structure and appearance of the captured environment.

5. Render New Viewpoints

After training, the optimized Gaussians can be projected from camera positions that were not part of the original capture.

The renderer determines which splats are visible, projects them into screen space, and blends their color and opacity contributions. The original research demonstrated that this explicit representation could achieve real-time rendering at high visual quality on the evaluated datasets.

Gaussian Splatting vs. Point Clouds

Gaussian splats and point clouds may appear similar because both can contain large numbers of spatial samples. However, they do not represent or render those samples in the same way.

A conventional point cloud generally stores coordinates and may also contain RGB color, intensity, classification, or other attributes. Each point represents a measured or reconstructed location.

A Gaussian splat carries a larger visual footprint. Its scale, orientation, opacity, and appearance help neighboring splats blend into a continuous view.

This allows Gaussian splats to reproduce complex visual detail without requiring a fully reconstructed mesh. However, a visually convincing splat does not automatically provide the measurement reliability, classification, or geometric structure expected from a professionally controlled survey or laser-scanned point cloud.

Gaussian Splatting vs. Photogrammetry

Photogrammetry reconstructs three-dimensional information from overlapping photographs. Its outputs may include camera positions, point clouds, textured meshes, orthophotos, and measurable surface models.

Gaussian splatting may begin with a similar image set and camera-estimation process, but its primary output is an appearance-oriented scene representation optimized for view synthesis.

Photogrammetry attempts to recover explicit geometry. Gaussian splatting attempts to reproduce how the captured environment appears from different viewpoints.

The two approaches can complement one another. A project may use photogrammetric or laser-scanning data for geometry and measurement while using Gaussian splatting for fast, immersive visual access to the captured environment.

Gaussian Splatting vs. NeRF

Neural Radiance Fields, or NeRFs, became prominent after the publication of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis in 2020.

A NeRF commonly uses a neural network to represent how color and density vary throughout a scene. To render a view, the system evaluates many samples along camera rays and integrates their contributions.

The original 3D Gaussian Splatting method uses an explicit collection of optimized Gaussian primitives instead of relying on a neural network as the primary scene representation. This explicit structure supports efficient rasterization and interactive rendering.

Both approaches can produce compelling novel views, but they differ in representation, training behavior, rendering requirements, storage, editability, and artifact patterns. Neither method should be assumed to be universally superior. The correct choice depends on the scene, hardware, workflow, output platform, and required level of control.

Gaussian Splatting vs. Polygon Models and BIM

A polygon model defines surfaces using vertices, edges, and faces. A BIM model adds structured building information such as materials, systems, objects, classifications, and relationships.

Gaussian splatting does not automatically provide that structure.

A captured Gaussian scene may look highly realistic while lacking:

• Clean architectural surfaces.

• Editable walls, doors, windows, and equipment.

• Parametric objects.

• Verified dimensions.

• Material definitions.

• Construction assemblies.

• Object classifications.

• Facility-management data.

• Reliable clash-detection geometry.

For architecture and construction, Gaussian splatting should therefore be treated as a captured visual representation—not an automatic substitute for CAD or BIM.

RENDEREXPO’s digital construction and digital twin services can help project teams determine how captured visual information should relate to BIM-based communication, construction visualization, progress documentation, and longer-term digital-twin objectives.

Key Uses of 3D Gaussian Splatting

Architectural and Real Estate Visualization

Gaussian splatting can create an immersive record of an existing building, interior, landscape, or development site. Users may navigate the captured environment more freely than they can with a fixed photograph.

For renovation and repositioning projects, this can help remote teams understand existing atmosphere, materials, surroundings, and spatial relationships. Proposed work may then be communicated separately through controlled architectural visualization and CGI.

The captured scene and the proposed design should be clearly distinguished so stakeholders understand which elements exist and which are conceptual.

Construction Documentation

Repeated site captures can provide visual records of construction conditions at different dates. These records may support progress discussions, remote review, issue communication, and comparisons between project stages.

However, a Gaussian splat alone does not verify compliance, dimensions, installation quality, schedule completion, or as-built status. Those conclusions require appropriate professional review and reliable project data.

Digital Twins and Facility Communication

Gaussian splatting can contribute visual context to a digital-twin workflow by creating an immersive representation of a real facility.

A functional digital twin requires more than realistic appearance. It may also require object identity, sensor data, asset information, operational relationships, maintenance records, and integration with other systems.

The splat may serve as one visual layer within that broader environment rather than being the complete digital twin.

Gaming and Interactive Experiences

Gaussian splatting can help capture real locations for interactive environments, virtual production, simulation, and games.

Its strength is the rapid reproduction of complex visual detail. Its challenges include collision geometry, lighting changes, animation, memory requirements, integration, and the need to combine the splat with conventional game-engine assets.

NVIDIA’s GPU-accelerated Gaussian-splatting guidance discusses the technique as a method for rendering complex captured scenes in real time for gaming, virtual reality, and professional visualization.

Robotics and Simulation

Robotics systems need realistic environments for perception, navigation, testing, and simulation. Gaussian-based reconstruction can help reproduce real locations with detailed visual appearance.

NVIDIA’s Neural Reconstruction and 3D Gaussian Splatting tools use reconstructed scenes from camera or LiDAR data within simulation workflows. This illustrates how Gaussian-based representations are expanding beyond visual demonstrations into spatial-computing and simulation pipelines.

Cultural Heritage and Existing-Condition Capture

Historic interiors, artifacts, landscapes, and culturally significant environments can be documented as navigable visual scenes.

This can improve accessibility and support communication, education, and archival work. Long-term preservation still requires careful source management, metadata, stable formats, and complementary survey documentation.

Limitations of 3D Gaussian Splatting

Gaussian splatting can look impressive, but several limitations must be considered.

Capture Quality Controls the Result

Missing views, motion blur, inconsistent exposure, reflections, transparent surfaces, moving objects, and repetitive textures can produce incomplete or unstable results.

Geometry Is Not Automatically Reliable

Visual realism should not be confused with survey accuracy. A Gaussian representation may reproduce appearance while containing uncertain surfaces or artifacts.

Lighting Is Difficult to Change

Much of the captured lighting is embedded within the reconstructed appearance. Relighting a Gaussian scene convincingly remains more complicated than changing lights in a deliberately modeled and materially defined 3D environment.

Editing Can Be Difficult

Removing objects, changing materials, modifying architecture, or inserting, or inserting proposed work may require specialized tools and additional processing. Traditional models remain more controllable when the design itself must be revised repeatedly.

Dynamic Elements Cause Artifacts

People, vehicles, vegetation, screens, reflections, and other moving elements can appear blurred, duplicated, incomplete, or suspended as unwanted “floaters.”

File Size and GPU Requirements Matter

Large scenes may contain millions of Gaussians and require substantial storage, graphics memory, optimization, compression, or level-of-detail management.

Viewpoints Are Not Unlimited

The strongest results generally occur within or near the camera coverage used during capture. Moving far outside that coverage can reveal missing information, stretched details, holes, or unstable geometry.

When Should Architecture and Construction Teams Use It?

3D Gaussian Splatting is most useful when a team needs rapid, visually rich access to a captured real environment.

It may be appropriate for:

• Existing-condition visualization.

• Remote site understanding.

• Immersive property presentation.

• Progress documentation.

• Visual context for renovation planning.

• Simulation environments.

• Digital-twin visualization layers.

• Cultural or spatial documentation.

A conventional model, BIM workflow, laser scan, or professional rendering remains more appropriate when the work depends on:

• Verified measurements.

• Coordinated geometry.

• Design revisions.

• Construction documentation.

• Material control.

• Code analysis.

• Clash detection.

• Quantity extraction.

• Permit information.

• Repeatable proposed-design views.

In many professional projects, the strongest solution is hybrid. Captured reality provides context, BIM or CAD provides controlled geometry, and professional visualization communicates the intended future condition.

Frequently Asked Questions

What is 3D Gaussian Splatting?

3D Gaussian Splatting is a method for representing and rendering a photographed scene using many optimized, semi-transparent 3D Gaussian primitives. It can produce high-quality novel views at interactive frame rates.

Why is it called Gaussian splatting?

“Gaussian” refers to the smooth mathematical distributions used as scene primitives. “Splatting” refers to projecting each primitive onto the image plane and blending its footprint across pixels.

Who invented Gaussian splatting?

Lee Westover’s work helped establish splatting in volume rendering around 1990. Modern 3D Gaussian Splatting for real-time radiance-field rendering was introduced in 2023 by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.

Is Gaussian splatting a point cloud?

It is related to point-based representations, but each Gaussian contains additional properties such as scale, rotation, opacity, and view-dependent color. It is more visually expressive than a basic point cloud.

Is Gaussian splatting better than NeRF?

It can train and render more efficiently in many workflows, but “better” depends on the scene, tools, hardware, output requirements, and desired control. Both are active areas of development.

Can Gaussian splatting be used in games?

Yes, particularly for captured environments and visually rich scene elements. Production use may still require conventional geometry for collisions, interaction, animation, lighting, and performance control.

Can Gaussian splatting replace BIM or laser scanning?

No. It can provide valuable visual context, but it does not automatically contain the verified geometry, object information, documentation, or measurement reliability required for BIM and professional surveying.

(Neutral Radiance Fields (NeRFs))

Conclusion

3D Gaussian Splatting is an important development in real-time scene reconstruction and novel-view rendering. Its explicit Gaussian representation can reproduce complex captured environments with high visual quality while supporting interactive navigation.

Its value comes from using it for the right purpose. Gaussian splatting is particularly effective for visual capture, immersive review, simulation, site context, and appearance-based documentation. It is less appropriate when a project requires editable architectural geometry, verified dimensions, coordinated BIM objects, controlled material changes, or construction documentation.

For architects, developers, contractors, owners, and facility teams, the practical opportunity is not to replace every established workflow. It is to connect captured reality with professional modeling, visualization, digital construction, and spatial information according to the decisions the project needs to support.

RENDEREXPO helps project teams evaluate and communicate complex built environments through architectural visualization, BIM-based communication, construction visualization, spatial mapping, and digital-twin strategy. To discuss the right visual or digital workflow for a project, contact RENDEREXPO.