Context & Engineering Objective
This individual ME 8913 project investigated how a local language interface could sit above a conventional robotic manipulation stack without replacing the geometry and control layers that make motion reliable. The simulated platform combined an RB5 six-degree-of-freedom arm, a Unitree dexterous hand, a wrist-mounted RGB-D camera, and source and destination trays containing bolts, nuts, and gears.
The engineering objective was to translate a plain-language command into a deterministic sequence of perception, localization, collision-aware motion, grasping, transport, and release. The language model selected a supported object class and destination; the lower layers remained responsible for establishing where the object was and whether the robot could reach it safely.
Role & Project Scope
I developed the simulation integration, perception-to-world geometry, controller interfaces, collision model, manipulation state machine, grasp presets, and local language proof of concept. The work was completed entirely in Isaac Lab at a 100 Hz physics rate. It demonstrates system integration in simulation, not a deployment on a physical RB5 or an open-ended task planner.
- Convert high-level natural-language intent into deterministic robot actions
- Localize bolts, nuts, and gears from wrist-mounted RGB and depth observations
- Maintain a palm-down grasp orientation while avoiding tables, tray walls, and neighboring objects
- Coordinate a 6-DOF arm with a 20-DOF hand without relying on kinematic grasp constraints
- Keep simulation, perception, planning, and control synchronized at a 100 Hz physics rate
Methodology
The system was organized as a layered perception-planning-control architecture with explicit interfaces between the language model, camera pipeline, state machine, and robot controller. YOLOv8-seg detected the three supported part classes at 960-pixel resolution. The center of each detection was paired with the median valid depth in a 7-by-7 pixel window, then transformed from camera coordinates through the simulator’s USD frame into the world frame.
- Trained YOLOv8-seg to detect bolts, nuts, and gears, then back-projected bounding-box centers using median depth from a 7-by-7 pixel window
- Generated arm targets with RMPflow using position attraction, palm-down orientation enforcement, joint-limit avoidance, and obstacle repulsion
- Implemented a 13-state pick-and-place controller from approach and pre-shape through transport, release, and return home
- Used object-specific finger-curl presets and gradual grasp closure to reduce premature contact with the parts
- Connected Ollama-hosted Llama 3 commands to the manipulation controller as a proof of concept for conversational robot control
The state machine imposed a 0.025 m target tolerance and 15-second timeouts so stalled motions could fail explicitly instead of hanging the simulation. A calibrated fingertip-cage offset connected the arm target to the actual grasp region, while staged approach, descent, closure, lift, transport, and release motions made the control sequence observable and debuggable.
Engineering Decisions & Trade-Offs
Several of the most important improvements were environmental and architectural rather than model changes. The source tray’s cross-hatch texture repeatedly produced false positives, so I covered it with simple black and blue kinematic cuboids. Those surfaces both removed the visual pattern and became collision geometry, solving a perception problem and a planning problem with the same scene change.
Isaac Lab’s actuator configuration did not override drive gains already baked into the USD assets, so the joint drives had to be changed at runtime. I also used the direct RMPflow API to avoid state-synchronization conflicts and increased the orientation metric from 10 to 2,000 to maintain a palm-down grasp. Fourteen collision spheres approximated the hand so obstacle avoidance accounted for the end effector rather than only the arm.
The initial Dex5 hand offered more articulation but created a large tuning space. Moving to the simpler Dex3 reduced the number of coupled grasp variables and improved experimental consistency. This was a deliberate reduction in dexterity to gain controllability, not a claim that the smaller hand was universally better.
Results & Validation
Simulated result After the tray redesign, perception was sufficient to support the documented manipulation trials, and the arm reached commanded targets through the RMPflow and state-machine pipeline. The report did not provide a held-out detection metric, so the page does not turn the observed improvement into an unsupported accuracy percentage.
Raw RGB-D reprojection retained approximately 5 cm of error from camera geometry, depth noise, and object settling. I corrected the simulated target by matching the detection to a live OpenUSD object transform within 0.12 m. This was an effective simulation-debugging technique, but it uses simulator ground truth and therefore cannot be presented as a sensor-only localization solution for a physical robot.
Most trials progressed through full manipulation cycles, but completion was not consistent. The dominant failure was object slip during lift or transport; timeouts and occasional neighboring-object collisions also occurred. A same-class obstacle-exclusion bug made nearby objects difficult to distinguish during planning. These results locate the remaining problem in grasp verification and object identity rather than in simple end-effector reachability.
Impact, Limitations & Next Steps
The project’s impact is the integrated architecture and the failure analysis it enabled. Language commands, vision, world-frame targets, reactive planning, collision avoidance, and dexterous control were connected through inspectable interfaces. When a trial failed, the system exposed whether the cause was perception, reprojection, planner convergence, state timeout, collision identity, or grasp retention.
The current language layer maps supported phrases to fixed object labels and predefined destinations; it does not perform open-ended multi-step planning. The localization correction depends on simulator truth, and the grasp has no force or visual verification after closure. A physical transfer would require calibrated camera extrinsics, sensor-only pose estimation, per-instance tracking, grasp-success detection, and a recovery state that can reobserve and retry after a slip.
