Video Overview
Cluttered Dexterous Grasping
Retrieving a target from clutter requires a dexterous hand to reach through narrow gaps and establish stable contacts while interacting with nearby objects. We study how sensing requirements change with scene and object demands, combining vision, fingertip tactile sensing, and estimated wrenches on a compliant arm–hand system.
Vision-Only Failure and Recovery
Vision only repeatedly contacts the target region but lifts without the target. Starting from the resulting scene, the full policy uses interaction feedback to regrasp, stabilize the target, and complete the lift. This is a sequential policy-switch case study.
When Does Touch Matter?
We compare eight controlled policy variants across five conditions, with five trials per policy and condition. The full policy succeeds in 24/25 trials, compared with 14/25 for Vision only. Across the three confined conditions (S3–S5), the difference is 15/15 versus 6/15.
All controlled variants use the same 320 demonstrations, visual inputs, action interface, and compliant controller. Scenes are reset by hand toward each reference layout. Each trial allows repeated autonomous attempts; success requires lifting and retaining the target without human intervention.
Watch all 50 Vision-only and Ours trials, including successes and failures. Select a scene result in either row to view its five recordings.
| Policy | Input | S1 | S2 | S3 | S4 | S5 | Overall |
|---|---|---|---|---|---|---|---|
| Vision only | V | 14/25 | |||||
| Wrench | V + W | 5/5 | 3/5 | 3/5 | 3/5 | 1/5 | 15/25 |
| Taxel-Attn | V + T | 5/5 | 4/5 | 2/5 | 4/5 | 3/5 | 18/25 |
| GNN | V + T + W | 5/5 | 4/5 | 4/5 | 4/5 | 4/5 | 21/25 |
| FC-full | V + T + W | 5/5 | 3/5 | 5/5 | 3/5 | 1/5 | 17/25 |
| PolicyConsensus | V + T + W | 4/5 | 3/5 | 5/5 | 2/5 | 1/5 | 15/25 |
| Reactive DP | V + T + W | 4/5 | 4/5 | 4/5 | 4/5 | 5/5 | 21/25 |
| VWT-Attn (Ours) | V + T + W | 24/25 |
Trial recordings
Recording lengths include setup and ending footage. Table 2 uses the evaluation timing defined in the paper.
Wrench and taxel observations provide complementary gains. Among successful episodes, the full policy also uses fewer full lift attempts and has a lower mean duration in every condition.
Execution efficiency
| Policy | S1 | S2 | S3 | S4 | S5 |
|---|---|---|---|---|---|
| Vision only | 19.0 / 1.50 | 18.5 / 1.50 | 22.7 / 1.67 | 38.8 / 2.50 | 41.6 / 3.00 |
| VWT-Attn (Ours) | 14.1 / 1.00 | 15.7 / 1.00 | 22.5 / 1.20 | 24.7 / 2.00 | 39.4 / 2.40 |
A full lift attempt is an upward end-effector excursion of at least 10 cm. Duration and attempt counts are measured among successful episodes only; Vision only has two successes on S4 and one on S5.
Contact-Guided Grasp Refinement
The policy adjusts its contacts before committing to a lift. Synchronized tactile measurements show the contact distribution changing as the fingers establish a supported grasp. The hand carries four PaXini pads with 508 three-axis taxels.
Compliant Arm and Hand
The platform combines an xArm7 arm, a 16-DoF LEAP hand, four tactile pads, and two RGB-D cameras. Estimated external wrenches drive arm and fingertip admittance, allowing the robot to yield during contact. Every controlled policy variant uses this controller, including Vision only.
Method
The policy receives a target–context point cloud, proprioception, and the enabled interaction channels. A point encoder and tactile attention encoder produce visual and tactile features, which are concatenated with wrench and proprioceptive states. A diffusion policy predicts arm and hand references executed through the shared compliant controller.
The experiments vary the available modalities, interaction encoding, and fusion or action-generation architecture while keeping the demonstrations and control interface fixed.
Failure Modes
The policy can lose track of the target when it becomes fully occluded in both policy cameras, or stall without making progress. Thick fingertips also limit access to narrow gaps. These remain challenges for persistent state estimation, recovery, and hand design.
Implementation Notes
ClutterDexGrasp reproduction and hardware deployment
We independently reproduce the teacher–student approach of ClutterDexGrasp in our simulation environment. The vision-only DP3 student uses a 5D segmented point cloud and absolute hand joint targets. It achieves 50.2% simulation success over 1,280 trials.
The original paper’s Table 1 reports 80.9–92.6% success for its teacher and 73.5–90.8% for its student. Our simulator, embodiment, and evaluation scenes differ, so these numbers are not a matched comparison. The clips below show a successful lift and a failed attempt from our student checkpoint.
In a preliminary single-object hardware test with arm and hand compliance enabled, the reproduced student moved rapidly downward into the table, prompting an emergency stop. The tested checkpoint did not transfer safely to our hardware.
RDP: why the fast decoder uses sensor-local taxels
Our main policy represents taxel positions in the world frame. The sensor-local exception is the fast decoder in our Reactive Diffusion Policy (RDP) baseline; it does not change the representation used by our main policy.
RDP combines a slow planner with a fast decoder that reacts to interaction observations. In our initial implementation using world-frame taxels, we observed nearly stationary arm behavior. These coordinates also reveal hand pose, which we suspect allowed the decoder to reproduce the current pose instead of following the planned motion. Using sensor-local coordinates removed this behavior in our tests. The RDP results in Table 1 use this sensor-local configuration.