3D selection through hand tracking in XR for point clouds using signed distance fields

Abstract

Three-dimensional (3D) visualizations are commonly used in many contexts, e.g., medicine, geography, astronomy, and they are gaining ever more popularity with the recent developments in extended reality (XR). While 3D visualizations can offer unique spatial insights and thus are useful in a variety of ways, interacting with them remains challenging despite being of essential importance in working with 3D visualizations. A specific problem with interacting with 3D visualizations is the lack of an effective and efficient way to select data. Traditional selection methods are primarily designed for, and limited by, 2D interfaces, therefore often do not provide the spatial precision, necessary degrees of freedom and sufficient flexibility necessary for effective 3D selection. Existing approaches for 3D are restricted to simple selection volumes, and do not consider optimization large datasets such as point clouds. Given the above, in this paper, we tackle the challenges associated with the 3D selection problem by designing, developing and user testing a GPU-based selection approach utilizing Signed Distance Fields (SDFs). Our approach allows for computationally efficient, realtime selection of near-arbitrary subsets of point clouds visualized in XR environments. Implemented in Unity, the solution combines novel and existing selection techniques i.e., two direct geometry based ones (shapes, convex hull), and two brushing based ones (brush sphere and brush hands).