PragyaDex · Contact-Aware Dexterity Intelligence

Language → Contact-Aware Dexterity → Robot Hand Motion

Give PragyaDex a natural-language manipulation goal. The system constructs a physical context, proposes dexterity-program tokens, searches contact- and object-state-valid futures, retargets the winning program to the robot embodiment, and executes one verified transition at a time.

WHAT SHOULD PRAGYADEX DO?● READY
OPEN BOTTLE→POUR INTO CUP
UNDERSTAND
→
PROPOSE
→
SEARCH
→
RETARGET
→
VERIFY
→
EXECUTE
01 · Observe → Structure the Physical Context

The planner begins from physical evidence, not a symbolic cartoon.

The observation is grounded in the same robot, bottle, cup, and hand configuration used by the manipulation rollout. Contact and object state are then extracted as structured planning variables.

MANUSCRIPT · SCENE / CONTACT / HAND STATESEARCH STATE Xt · PROPOSED EXTENSION
OBSERVED PHYSICAL STATEŜt
FRAME-ALIGNED EVIDENCE · dex_1
Observed robot manipulation state
LEFT SUPPORTpalm/fingers maintain bottle contact
CAP CLOSEDopening prerequisite unresolved
RIGHT HAND FREEavailable for cap approach
CUP TARGETterminal pour region
OBSERVATION SOURCErobot manipulation frame
SUPPORT CONTACTactive
CAP STATEclosed
POUR LEGAL?no
CONTACT STATECt
frame → graph
BOTTLE
CAP
L PALM
1
0
R THUMB
0
0
R INDEX
0
0
L palmR hand bottlecap
ACTIVE EDGEL palm ↔ bottle
OBJECT STATEOt
prerequisite trace
CLOSEDcap attached
LOOSENINGrotation underway
OPENpour admissible
POURINGtarget transition
REMAINING TASK GRAPHGtrem
instruction → prerequisites
01OPEN BOTTLEunresolved prerequisite
enables
02POUR INTO CUPterminal goal
STATE FUSIONphysical evidence → planner context
instruction xscene Ŝtrobot qt,q̇tcontact Ctobject Otgoal Gtrem
→
Xt(x, Ŝt, qt, q̇t, Ct, Ot, Gtrem, Mt, Ut)
The action space is conditioned on what is physically true now: current contacts, current object state, current robot state, and unresolved task prerequisites.
02 · Decode → Rank Physical Action Hypotheses

A dexterity token is a structured physical program fragment.

The decoder does not emit a bare verb. Each candidate binds a primitive to hand roles, contact additions/removals, object-state effect, contact geometry, continuous parameters, and duration.

DEXTERITY TOKEN RECORDdt+1
example candidate · UNSCREW CAP
p · primitiveUNSCREW CAP
r · bimanual roleL stabilize · R rotate
Γ+ · add contactR thumb/index ↔ cap
Γ− · releasenone
gc · contact geometrycap rim / opposing digits
ΔO · object effectCLOSED → LOOSENING
θ · continuous paramswrist rotation · contact force
T · durationlocal action horizon
dt+1 = (p, r, Γ+, Γ−, gc, ΔO, θ, T)
PHYSICAL INTERPRETATIONcandidate preview
motion evidence, not decoration
Unscrew dexterity token physical preview
LEFTsupport bottle
RIGHTrotate cap
CONTACTcap grasp maintained
OBJECTclosed → loosening
CANDIDATE DISTRIBUTIONPθ(d | Xt)
SIMULATED WORKED TRACE
STABILIZE BOTTLE.31
GRASP CAP.27
REGRASP BOTTLE.18
TILT BOTTLE.13
APPROACH CAP / CUP.11
TOP-K5
ENTROPY1.44
TOP-1 MASS.31
TOP-5 MASS1.00
TOP-5 PHYSICAL HYPOTHESESclick to inspect
inactive candidates remain static
STABILIZE BOTTLEPhysical hypothesis corresponding to the currently inspected decoder candidate.
𝒟t+1 = DiverseTopK5Pθ(dt+1 | Xt)probability proposes · search decides
04 · Selected Program τ*

The search collapses into one physically coherent manipulation program.

Inspect the same winning program simultaneously as physical motion, primitive sequence, bimanual roles, contact evolution, object-state evolution, likelihood, and terminal quality.

SELECTED PROGRAM · SIMULATED WORKED TRACE

STABILIZE → GRASP CAP → UNSCREW → REGRASP → POUR

τ* = [d₁*, d₂*, d₃*, d₄*, d₅*]

S(τ*).845
RANK#1 / 3,125
Stabilize bottle01STABILIZE
→
Grasp cap02GRASP CAP
→
Unscrew cap03UNSCREW
→
Regrasp04REGRASP
→
Pour05POUR
PRIMITIVESTABILIZEGRASPUNSCREWREGRASPPOUR
LEFT HANDSUPPORTSUPPORTSUPPORTSUPPORTASSIST
RIGHT HANDAPPROACHGRASPROTATEREGRASPTILT
CONTACTBODY SUPPORTCAP GRASPROTATIONALPOUR GRASPPOUR SUPPORT
OBJECTCLOSEDCLOSEDLOOSENINGOPENPOURING
CUMULATIVE SEQUENCE LOG-LIKELIHOODΣ log P
012345
P(τ*) = e−5.82≈ 0.00296
TERMINAL QUALITY CONTRIBUTIONSS(τ*)=.845
Task+.2325
Contact+.1980
Object+.1365
Feasibility+.1040
Progress+.0890
Bimanual+.0800
Smoothness+.0350
Risk−.0300
LIKELIHOOD IS NOT THE FINAL OBJECTIVE.
Complete selected PragyaDex program
SELECTED PROGRAM · τ*

Full physical rollout

STABILIZE → GRASP CAP → UNSCREW → REGRASP → POUR

contact structurepreserved ✓object progressionclosed → pouringprogram rank#1
05 · Retarget → Validate the Selected Contact Program

Preserve manipulation function while embodiment geometry changes.

Retargeting is treated as an optimization problem: preserve task-relevant pose and contact structure while satisfying robot joint limits and collision constraints.

RETARGETING OBJECTIVE · MANUSCRIPT-BACKEDDIAGNOSTIC VALUES · ILLUSTRATIVE
CANONICAL HAND STATEH · 21 landmarks
task-space representation
OBJECT FRAME
contact landmarksthumb tip · index tipreference frameobject-centric
CONTACT INVARIANTC / G
what must survive transfer
thumb
CAP
index
palm
BOTTLE
preservecap grasp topology
preservebottle support relation
allowembodiment-specific joint geometry
ROBOT REALIZATIONR={qt}
selected token · UNSCREW
Robot realization of unscrew token
required cap contactspreservedrobot morphologydifferent geometry
OPTIMIZATION OBJECTIVEminq [ Lpose + λcLcontact + λlimLlimits + λcolLcollision ]
Contact is not a visual annotation here; it is an optimization term and feasibility condition.
RETARGET OPTIMIZATIONobjective vs. iteration
ILLUSTRATIVE TRACE
totalcontactlimitscollision
PHYSICAL FEASIBILITYper-token diagnostics
ILLUSTRATIVE
fingertip residual4.6 mm
orientation residual3.2°
minimum clearance6.1 mm
joint limitsPASS
self collisionPASS
object collisionPASS
required contactPASS
Geometry changes. Functional contact survives. Expensive retargeting and physical validation are invoked only for the narrowed search frontier.
06 · Execute One → Measure → Update → Replan

Foresight is multi-step; commitment is one verified transition.

The selected five-token program is not executed open-loop. PragyaDex commits one token, measures contact and object evidence, updates the planner state, and regenerates the frontier.

RECEDING-HORIZON COMMITMENTτ* · horizon 5
only d₁* is committed
01STABILIZEEXECUTE
02GRASPplanned
03UNSCREWplanned
04REGRASPplanned
05POURplanned
EXECUTING d₁*STABILIZE BOTTLE
physical evidence · dex_1
Executed stabilize token
0 ms720 ms
EXPECTED vs OBSERVED CONTACTΔC
post-token check
EXPECTED
L palm↔bottle
−
OBSERVED
L palm↔bottle
CONTACT MISMATCH0
BOTTLE POSE ERROR · mm2.1
CONTACT CONFIDENCE.94
NEW EVIDENCECobs, Oobs, qt+1contact · object · robot state
→
STATE UPDATEXt + observationsreplace predicted state with measured state
→
NEXT CONTEXTXt+1new planning root
REPLAN CONDITIONevent gate
successful worked path
TOKEN COMPLETE
CONTACT LOST
SLIP / DROP RISK
OBJECT-STATE MISMATCH
COLLISION PROXIMITY
NEW LOCAL FRONTIERconditioned on Xt+1
compact replan view
Xt+1
GRASP CAPhighest task progress
MICRO-ADJUSTcontact refinement
REPOSITIONalternate geometry
Plan 5. Execute 1. Verify Contact. Replan.Every committed transition converts predicted physical state into measured physical evidence before the next search.
READY