Visual analytics of nonverbal behavior to evaluate collaborative group engagement

Abstract

Despite rapid advances in AI, computer vision, and the availability of off-the-shelf tools, analyzing and understanding the dynamics of nonverbal behavior (NVB) remains a significant challenge, especially in the analysis of collaborative group engagement. Research areas such as Social Signal Processing aim to leverage computational methods to automatically extract NVB from highvolume, multimodal video, audio, and language data, but with moderate success. These automated approaches rely heavily on large, high-quality training datasets and often face issues related to predicted constructs’ theoretical soundness and context-specific validity. A promising alternative is Visual Analytics (VA), which integrates human reasoning with computational methods for data interpretation. This poster explores a methodological approach using VA to extract and analyze NVB in collaborative learning. We employ state-of-the-art computer vision techniques to generate highresolution time series of facial, hand, and body landmarks from video recordings of small student groups collaboratively solving computer-based tasks. These landmarks are then processed into meaningful NVB signals and visualized to enable exploration and analysis. We also introduce visual-mapping strategies to address the challenges posed by high-dimensional data and the information loss introduced by aggregation. Finally, we demonstrate the potential and limitations of VA to support the analysis of both individual and dyadic NVB, highlighting temporal patterns in head movement and mutual orientation (facing direction) within small-group interactions.