← Back to my projects
VMV Logo Drone VCE Logo

Towards Integrating Multi-Spectral Imaging with Gaussian Splatting

Josef Grün‡, Lukas Meyer‡ ORCID iD , Maximilian Weiherer ORCID iD , Bernhard Egger ORCID iD , Marc Stamminger ORCID iD , Linus Franke ORCID iD

Visual Computing Erlangen, Friedrich-Alexander-Universität Erlangen - Nürnberg , Germany

‡ Equal contribution

Paper (tbd) arXiv (tbd) Video Code (tbd)

Abstract

We present a study of how to integrate color (RGB) and multi-spectral imagery (red, green, red-edge, and near-infrared) into the 3D Gaussian Splatting (3DGS) framework — a leading explicit radiance-field method for fast and high-fidelity 3D reconstruction from multi-view images (Kerbl et al.). While 3DGS excels on RGB data, naive per-band optimization of additional spectra yields poor reconstructions due to inconsistently appearing geometry in the spectral domain. This problem is prominent, even though the actual geometry is the same, regardless of spectral modality.

To investigate this, we evaluate three strategies:

  1. Separate: per-band reconstruction with no shared structure;
  2. Split optimization: first optimize RGB geometry, copy it, and then fit each new band to the model by optimizing both geometry and band representation;
  3. Joint: jointly optimize all modalities, optionally with an initial RGB-only phase.

We showcase through quantitative metrics and qualitative novel-view renderings on multi-spectral datasets the effectiveness of our optimized Joint strategy, which improves overall spectral reconstruction and enhances RGB results through spectral cross-talk.

We therefore propose the integration of multi-spectral data into the spherical harmonics color components to compactly model each Gaussian’s multi-spectral reflectance. Our analysis reveals key trade-offs in when and how to introduce spectral bands during optimization, offering practical hints for robust multi-modal 3DGS reconstruction.

Acknowledgements

Lukas Meyer was funded by the 5G innovation program of the German Federal Ministry for Digital and Transport under the funding code 165GU103C. Maximilian Weiherer was funded by the German Federal Ministry of Education and Research (BMBF), FKZ: 01IS22082 (IRRW).

The authors are responsible for the content of this publication and thank the reviewers for their insightful comments.

The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project b162dc. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683.

BibTeX

@InProceedings{MSSplatting2025,
  author    = {Josef Grün and Lukas Meyer and Maximilian Weiherer and Bernhard Egger and Marc Stamminger},
  title     = {Towards Integrating Multi-Spectral Imaging with Gaussian Splatting},
  booktitle = {Proceedings of the Vision, Modeling, and Visualization Conference (VMV)},
  year      = {2025},
  month     = sep,
  publisher = {Eurographics Association},
  url       = {https://meyerls.github.io/towards_multi_spec_splat}
}