Multi-Modal Learning
Built with FlexiTac

ViTaForce: Pairwise Visuo-Tactile-Force Representation Learning for Contact-Rich Imitation Learning

Yueyang Wang*, Chaofan Xu*†

Harbin Institute of Technology; Beijing Institute of Spacecraft System Engineering

Conference on Robot Learning (CoRL), 2026

First page of the ViTaForce: Pairwise Visuo-Tactile-Force Representation Learning for Contact-Rich Imitation Learning paper

Abstract

Precise robotic assembly requires learning relationships between visual observations and physical contact. Vision provides scene information, fingertip pressure characterizes local contact, and end-effector wrench captures interaction dynamics. Learning these relationships directly from action demonstrations is challenging because the modalities differ in spatial structure and temporal resolution. We present ViTaForce, a pairwise representation learning framework that learns cross-modal contact representations before policy training. Its three encoders organize contact into complementary where–when–what questions: wrist vision and touch model where pressure occurs, scene vision and wrench model when events occur within an observed window, and tactile–wrench context models what tactile activity follows. Their pretrained representations jointly condition a Diffusion Policy. On nut-on-screw insertion, in-hand peg reorientation and insertion, and keyed-peg rotational search, ViTaForce improves success over a vision-only policy by 50, 48, and 40 percentage points. The combined representation also exceeds raw multimodal inputs, unpretrained fusion, and each single-pair variant, and context perturbations show that the encoders use their paired inputs.

Links & Resources