How would I project & select a point from the mouse into the gaussian?

[Zk2u]

Hello, I'm planning to use this for a small sideproject on the web. I don't have the indepth graphics knowledge to be able to tell how I would go about this:

I explore the splat with a 3D camera, and I want to "mark" or select either objects or points in the scan. Kind of like how polycam allows you to annotate objects. Would this crate support something like this and if so, how do you recommend I go about it? Points are likely to be easier, but marking/highlighting objects with a kind of brush tool on the mouse would be a better suit for our usecase.

Thanks so much in advance

[mosure]

for mesh objects, the existing bevy picking mesh ray casting backend can be used:

• https://docs.rs/bevy/latest/bevy/picking/mesh_picking/index.html
• https://bevy.org/examples/picking/mesh-picking/
• https://github.com/bevyengine/bevy/blob/latest/examples/picking/mesh_picking.rs

I haven't yet implemented a gaussian picking backend, but it should be pretty similar to the mesh implementation. without acceleration structures, it may be functional but not as performant for larger numbers of gaussians. a good starting point for coding this up would be this module:

https://github.com/bevyengine/bevy/blob/main/crates/bevy_picking/src/mesh_picking/mod.rs#L69

e.g. emit pointer events on ray/gaussian intersections (sorted by depth)

the brush-size would probably require multi-ray processing or more fuzzy ray intersections

[Zk2u]

When you say mesh objects, do you mean normal 3D models rather than gaussians?

[mosure]

yes, the existing bevy picking backend operates on meshes. if your scenes are comprised of smaller gaussian objects, you could write the picking backend for gaussian object AABB (more performant than individual point selection, but not useful for large gaussian scenes):

gaussian AABBs are already computed/inserted as components here:

bevy_gaussian_splatting/src/gaussian/cloud.rs

Line 58 in 6db1932

pub fn calculate_bounds<R: PlanarSync>(

---

bevy picking ray_aabb_intersection_3d:
https://github.com/bevyengine/bevy/blob/27b64c3bf15882761ac914c72bfdad675bebe84a/crates/bevy_picking/src/mesh_picking/ray_cast/intersections.rs#L254

[Zk2u]

Our usecase will just be a single large splat. We'd be scanning a room or large interior area for example - for that the mesh backend wouldn't work right? Cos we'd be brush selecting or point selecting inside that one gaussian

[mosure]

correct, for a single large splat, both the existing mesh backend and gaussian AABB won't be sufficient.

you'd have to use a custom gaussian picking backend which can be derived from the existing mesh picking backend.

the custom picking backend could select based on brush size + contiguous spatial locality for a more 'object-like' selection.

for frozen object selection from a pre-processed large scene, I'd recommend an offline instance segmentation method on your input images, utilizing the visibility property on bevy_gaussian_splatting's gaussian (e.g. each object gets a unique visibility ID):

• https://docs.ultralytics.com/tasks/segment/

• bevy_gaussian_splatting/src/gaussian/f32.rs

  Line 76 in 6db1932

  pub visibility: f32,

this allows using the classification render mode (e.g. each integer value of visibility is rendered as a unique color). also, it gives logical grouping to the large gaussian scene (e.g. ability to render a subset of the scene, move a subset of gaussians, etc.)