Visibility-Aware Mobile Grasping
in Dynamic Environments

Tianrun Hu, Anxing Xiao, David Hsu, Hanbo Zhang

Paper arXiv Code Video Interactive Demo

The interactive demo runs our system in simulation on RoboMesh. A RoboMesh account is required.

Teaser: A mobile manipulator navigating around dynamic obstacles to grasp a target object.
A Fetch mobile manipulator must jointly reason about where to look and how to move its whole body to safely grasp objects in dynamic, unknown environments with limited field-of-view.
Interactive Demo

Try it yourself, in your browser.

Our system runs as an interactive demo on RoboMesh, the robot demo platform of the Smart Systems Institute (SSI) at NUS. RoboMesh hosts demos from many groups; ours is the room “Visibility Awared Mobile Grasping”, from the AdaComp Lab. Say something as simple as “grasp this”, point at the object you mean, and watch the robot plan and grasp it on its own, in simulation. Nothing to install.

grasp this

+ point at the object

That is the whole instruction. The point says which object; the words say what to do.

  1. Sign in to RoboMesh and open our room, Visibility Awared Mobile Grasping.
  2. Say or type what you want, even just “grasp this”, and point at the object. Language and a point together leave no ambiguity about what you mean.
  3. The robot decides where to look and how to move its whole body, online and together. It works around whatever is in the way and grasps, recovering on its own when an attempt fails.
Our room on RoboMesh Visibility Awared Mobile Grasping AdaComp Lab · SSI, NUS
Open the interactive demo

A RoboMesh account is required. RoboMesh hosts many groups’ demos, so after signing in, open the room above to reach ours.

robomesh.ssilabs.org/webapp

Abstract

This paper addresses the problem of mobile grasping in dynamic, unknown environments where a robot must operate under a limited field-of-view. The fundamental challenge is the inherent trade-off between "seeing" around to reduce environmental uncertainty and "moving" the body to achieve task progress in a high-dimensional configuration space, subject to visibility constraints.

We propose a unified mobile grasping system comprising two core components: (1) an iterative low-level whole-body planner coupled with velocity-aware active perception to navigate dynamic environments safely; and (2) a hierarchical high-level planner based on behavior trees that adaptively generates subgoals to guide the robot through exploration and runtime failures.

We provide experimental results across 400 dynamic simulation scenarios and real-world deployment on a Fetch mobile manipulator. The results show that in both static unknown environments and dynamic environments with suddenly appearing obstacles, our system achieves success rates of 68.75% and 58.0%, respectively, significantly outperforming baselines in both robustness and safety.

Video

Method Overview

System architecture diagram
System architecture. Our receding-horizon framework tightly couples perception and planning, jointly optimizing observation and whole-body motion at runtime.

Velocity-Aware Active Gaze

A state-dependent gaze policy πv that switches between observing the target during planning and monitoring the swept volume during execution. Prioritizes collision-critical regions based on velocity and temporal proximity.

Hierarchical Subgoal Generation

An adaptive behavior-tree policy πg with three progressive strategies: direct grasping, pre-grasp repositioning, and observation gathering — enabling runtime recovery from failures.

Fast Whole-Body Planning

A novel vmRRT-C (Vectorized Mobile RRT-Connect) planner achieving 50–80 ms planning times for real-time replanning across the 11-DoF heterogeneous configuration space.

Execution Example

Full execution sequence showing third-person view, first-person view, and belief map.
Full execution sequence. Top: third-person view. Middle: onboard camera (first-person). Bottom: occupancy belief map. The robot actively explores, repositions, and grasps the target while avoiding dynamic obstacles.

Results

68.75% Success — Unknown Static
58.0% Success — Dynamic
400 Simulation Scenarios
50–80ms Planning Time

Simulation Evaluation

Success rate comparison across methods
Success rate comparison against baselines across 400 scenarios in 20 ReplicaCAD scenes.

Failure Analysis

Failure flow in unknown static environments
Failure decomposition in unknown static environments.

Real-World Deployment

Real robot in static environments
Static environment: 65% success across 5 indoor locations.
Real robot with dynamic obstacles
Dynamic environment with moving pedestrians: 55% success.

Spatial Failure Distribution

Spatial distribution of successes and failures across scenes
Spatial distribution of successes and failures. Failures cluster in deep table areas (reachability limits), wall-adjacent regions (constrained approach angles), and cluttered zones (precision requirements).

Evaluation Environments

Diverse evaluation environments in simulation and real world
Our system is evaluated across diverse environments in both simulation (ReplicaCAD scenes) and real-world indoor locations.

Qualitative Results

Initial configuration
Initial Configuration
Active observation
Active Observation
Pre-grasp repositioning
Pre-Grasp Repositioning
Grasping pose
Grasping

BibTeX

@article{hu2026visibility,
  title         = {Visibility-Aware Mobile Grasping in Dynamic Environments},
  author        = {Hu, Tianrun and Xiao, Anxing and Hsu, David and Zhang, Hanbo},
  journal       = {arXiv preprint arXiv:2605.02487},
  year          = {2026}
}