Dense Prediction

Abstract

In the realm of computer vision, material segmentation of natural scenes represents a challenge, driven by the complex and diverse appearances of materials. Traditional approaches often rely on RGB images, which can be deceptive given the variability in appearances due to different lighting conditions. Other methods that employ polarization or spectral imagery offer more reliable material differentiation, but their cost and accessibility restrict everyday usage. This work proposes a deep learning framework that uses paired RGB-D and spectral data during training to embed spectral information through a Spectral Feature Mapper (SFM) layer, enabling material segmentation from standard RGB-D images after training. The method also generates a 3D point cloud from the RGB-D pair to enrich scene understanding, and experiments on public datasets plus real captures from an iPad Pro show superior material segmentation performance.

Abstract

Accurate depth completion is crucial for safety-critical applications such as autonomous navigation. This paper introduces a stereo-LiDAR fusion network with a domain-aware attention mechanism that jointly reasons in depth and disparity space instead of relying on only one representation. The method uses sparse but precise LiDAR measurements, propagates them according to disparity density, and predicts dense disparity and depth-residual maps that correct stereo triangulation errors and disparity-to-depth conversion inaccuracies. On the KITTI validation set, the proposed approach achieves state-of-the-art Mean Absolute Error with rapid convergence.

Abstract

NeRF-based segmentation methods usually focus on semantics and rely only on RGB data, which limits material understanding. UnMix-NeRF integrates spectral unmixing into neural radiance fields to jointly perform hyperspectral novel view synthesis and unsupervised material segmentation. The model represents spectral reflectance with diffuse and specular components, learns a global dictionary of endmembers as pure material signatures, and estimates per-point abundances to capture their distribution across the scene. These learned spectral signatures also support unsupervised material clustering and material-aware scene editing, while experiments show stronger spectral reconstruction and material segmentation than prior methods.