1School of Computing, National University of Singapore 2Smart Systems Institute, National University of Singapore 3Shanghai Innovation Institute, Shanghai, China
*Corresponding author
The interactive demo runs our system in simulation on RoboMesh. A RoboMesh account is required.
Our system runs as an interactive demo on RoboMesh, the robot demo platform of the Smart Systems Institute (SSI) at NUS. RoboMesh hosts demos from many groups; ours is the room “Visibility Awared Mobile Grasping”, from the AdaComp Lab. Say something as simple as “grasp this”, point at the object you mean, and watch the robot plan and grasp it on its own, in simulation. Nothing to install.
grasp this
+ point at the object
That is the whole instruction. The point says which object; the words say what to do.
A RoboMesh account is required. RoboMesh hosts many groups’ demos, so after signing in, open the room above to reach ours.
robomesh.ssilabs.org/webappWe consider mobile grasping in unknown and dynamic environments, where a robot must approach and grasp a target object using a camera with a limited field of view. This setting introduces two visibility constraints. Incomplete environmental observations can create collision risks along the whole-body trajectory, while occlusions can hinder object pose refinement during grasping.
Previous systems often assume known or static environments and decouple perception from motion, leaving safety unresolved around unobserved obstacles. We present a unified system that enforces both visibility constraints at runtime through three core components. A velocity-aware gaze policy directs perception toward fast-moving regions of the planned swept volume, a hierarchical subgoal policy samples grasp and intermediate configurations that maintain target visibility, and a real-time whole-body planner replans over a receding horizon as the map updates.
Across 400 randomized simulation scenarios and real-world trials on a Fetch mobile manipulator, our system achieves success rates of 70.4% in unknown static environments and 57.9% in dynamic environments, outperforming the decoupled navigation-and-manipulation strategy by 16.3 and 9.3 percentage points, respectively, while significantly reducing collisions.
Supplementary video (3:04): the two visibility constraints, the velocity-aware gaze policy and the subgoal hierarchy, simulation and real-robot rollouts — including a person stepping into the route and a blocked manipulation attempt that re-samples its pre-grasp — closing with the quantitative results.
Mobile grasping already couples collision avoidance, kinematics, and grasping — constraints existing systems handle under a known map. A limited field of view in an unknown environment adds two more, and with a single camera they compete for the same gaze.
The map must cover the whole-body swept volume before the robot enters it. Otherwise a trajectory that is collision-free in the map can still hit an obstacle that was never observed. This matters most during execution.
The grasp configuration must admit a head orientation that keeps the target in view for continued pose refinement, respecting camera limits, self-occlusion, and mapped occlusion. This matters most during planning.
A receding-horizon loop updates the belief summary bt = (qt, Mt, gt) through four coupled components. New observations can invalidate both trajectories and goals, triggering replanning or goal resampling.
Gaze maximizes visibility of an importance field that adapts to the robot's phase. With no trajectory running, it observes the target for grasp generation. Once a trajectory is executing, it shifts to the swept volume, weighted by velocity and time-discounted lookahead so collision-critical regions are refreshed first.
A behavior tree with three fallback levels — grasp, pre-grasp, and observe. It tries the most direct level first and escalates when one fails, which is also how the system recovers at runtime. Pre-grasp sampling factors over base pose, torso height, and end-effector pose, then rejects candidates that fail IK, collision, or self-occlusion.
Hierarchical sampling with a non-holonomic kernel (Hybrid A* guidance, Reeds-Shepp local connections) for the base and RRT-Connect for the arm, synchronized and validated with SIMD collision checking. Planning runs in 50–80 ms; a 20 Hz whole-body velocity tracker accepts trajectory updates mid-execution.
| Method | Unknown static | Unknown dynamic | ||
|---|---|---|---|---|
| Success ↑ | Collision ↓ | Success ↑ | Collision ↓ | |
| Learning-based policies | ||||
| SG-VLA (Easy) | 1.6±1.2 | 78.8±1.9 | 1.2±0.8 | 80.5±2.1 |
| RL (MS-HAB, Easy) | 26.9±2.7 | 42.7±3.1 | 24.1±3.0 | 44.9±2.4 |
| Conventional mobile manipulation designs | ||||
| Direct Grasping | 27.2±2.5 | 2.5±0.8 | 16.1±1.1 | 14.8±1.9 |
| CapMap Placement | 53.0±2.4 | 9.7±1.0 | 49.7±0.9 | 8.2±0.8 |
| Navigation-and-manipulation | 54.0±2.4 | 7.7±1.1 | 48.6±3.4 | 7.5±1.7 |
| Ablations | ||||
| Ours w/o velocity weighting | 62.2±1.8 | 12.3±2.3 | 51.5±1.6 | 18.1±2.4 |
| Ours w/o observation stage | 68.3±3.7 | 5.8±2.4 | 56.3±3.4 | 13.2±4.4 |
| Ours (full system) | 70.3±1.1 | 5.0±1.5 | 57.9±1.5 | 13.9±3.4 |
Percentages, mean ± 95% confidence-interval half-width over five runs. The paper reports the full breakdown, including grasp, IK, reachability, perception, and planning failures, and the Hard evaluation protocol for the learning baselines. Headline figures above are quoted from the paper’s abstract (70.4%); the table reproduces its Table I (70.3%).
Navigation-and-manipulation commits to a base placement before the robot can observe the target's surroundings. That frequently produced unreachable configurations, doubled planning failures, and collided more often once fixed-base arm planning had to work in constrained space.
Removing the subgoal policy entirely (Direct Grasping) is the single largest drop — 43.2 and 41.8 points. Without fallbacks to repositioning or observation gathering, planning failures grew by 3.3–5.3×.
Removing it more than doubled collisions in static environments and raised them by a third in dynamic ones, while other failure categories barely moved. Velocity-aware perception improves success almost entirely by avoiding collisions — for 26 ms per cycle.
Its effect on success (2.1 and 1.6 points) is not statistically resolvable over five runs. Path efficiency separates the designs more clearly: 62.2% and 48.0% SPL for the full system against 54.7% and 45.8% for the ablation.
Even with the oracle Easy initialization, both learning baselines stayed low and SG-VLA near zero, with failures dominated by collisions. Neither showed signs of looking at the space it was about to enter. Visibility has to be an explicit objective.
Among 46 labeled grasp failures, 43.5% were geometrically sound but kinematically infeasible, 30.4% misaligned, 23.9% slipped on lift. The detector ranks by geometry alone while reachability depends on base placement — a clear case for kinematically informed grasp scoring.
Failure flow on our benchmark. Execution failures (collision, grasp, IK), system limitations (reachability, perception), and planning failures are tracked separately at each stage.
We deployed only the full system on the real robot, since baselines without active perception and reactive replanning risk damaging the robot and the environment. Across five indoor locations — dining table, kitchen counter, workstation, coffee table, sofa — it reached success rates comparable to simulation: 65.0% static (vs. 70.4%) and 55.0% dynamic (vs. 57.9%).
The remaining gap to simulation came from noisy observations caused by synchronization between localization and depth sensing, ground-plane filtering that occasionally removed low-lying points such as chair legs, execution noise when the robot stuck on friction and then accelerated, and segmentation or detection errors during grasping.
Beyond per-module attribution, the residual failures point to three structural limitations in how perception and planning are coupled — and to what a next system should do differently.
Velocity-aware gaze is effective once an obstacle has entered the map. The residual collisions were cases where space verified as free was invalidated by obstacles arriving from adjacent unobserved regions before the gaze revisited it. Active perception should anticipate where invalidation can originate — for example, at occlusion boundaries near the planned route.
The map clears freed space only along observed rays, so removing a point requires re-observing that region. Every dynamic obstacle left residue: once the robot looked away, the obstacle departed but its points persisted as floating obstacles, blocking valid paths and surfacing as planning failures. Removing this class needs a predictive environment model that propagates dynamic evidence forward and decays unconfirmed points while preserving occluded static obstacles.
The gaze policy conditions on the planned trajectory, but the motion policy optimizes collision cost in the map without asking whether its swept volume can be observed. Observability is a property of the trajectory itself — set by head reachability, self-occlusion, and motion direction, with backward motion as the extreme case. Visibility should enter motion planning as a cost or constraint.
@article{hu2026visibility,
title = {Visibility-Aware Mobile Grasping in Dynamic Environments},
author = {Hu, Tianrun and Xiao, Anxing and Hsu, David and Zhang, Hanbo},
journal = {arXiv preprint arXiv:2605.02487},
year = {2026}
}