Autonomy Tea Talk

"What's an Object? Creating Task-Driven Open-Set 3D Semantic Maps"

Open-set 3D semantic mapping requires (i) determining the correct granularity to represent the scene (e.g., how should objects be defined), and (ii) fusing semantic knowledge across multiple 2D observations into an overall 3D reconstruction —ideally with a high-fidelity yet low-memory footprint. While most related works bypass the first issue by grouping together primitives with similar semantics (according to some manually tuned threshold), we recognize that the object granularity is task-dependent, and develop a task-driven semantic mapping approach. To address the second issue, current practice is to average visual embedding vectors over multiple views. Instead, we show the benefits of using a probabilistic approach based on the properties of the underlying visual-language foundation model, and leveraging Bayesian updating to aggregate multiple observations of the scene. The result is a task-driven and probabilistic approach for open-set semantic mapping. To enable high-fidelity objects and a dense scene representation, we use 3D Gaussians which we cluster into task-relevant objects, allowing for both easy 3D object extraction and reduced memory usage.