Context & Engineering Objective
Traditional kitting robots work well when product type, sequence, and coordinates are fixed in advance. A flower-ordering workflow is less structured: a customer describes quantities and colors in natural language, the available inventory is visually mixed, and delicate stems must be approached without disturbing neighboring flowers.
This ME 4853 team project explored an order-to-packaging workflow that connects a Streamlit chatbot and Neo4j knowledge graph to a vision-guided UR5. The objective was to translate a customer request into a confirmed Bill of Materials (BOM), an ordered robot task list, and reliable physical pick-and-place actions using a deliberately small calibration dataset.
Role & Project Scope
The team developed the language, graph, vision, learned joint-angle, error-compensation, and physical workcell subsystems. The project tested the chatbot and vision-guided robot independently and completed an integrated script, but the final report states that the combined chatbot-to-robot path was not physically validated. The page therefore presents a subsystem-integration prototype rather than a fully automated deployed flower shop.
- Turn an open-ended customer request into reliable flower quantities, colors, pickup order, and drop-off locations
- Detect visually similar tulips in a crowded bin under changing background and lighting conditions
- Map a two-dimensional camera observation to six UR5 joint angles with a small physical dataset
- Grip delicate flowers without colliding with neighboring stems or the workcell
- Keep the order representation adaptable while acknowledging that final drop-off poses remained predefined
Methodology
Language & Knowledge Layer
A Streamlit interface passes each order to a LangChain ReAct agent. Semantic matching resolves requested colors, while Neo4j stores the relationships among tulip classes, pickup order, and drop-off locations. The interface presents a BOM for human confirmation before exporting the Bill of Order used by the robot workflow.
- Modeled five color classes: white, pink, red, purple, and orange
- Stored class, sequence, and destination relationships in Neo4j
- Added a dedicated extraction tool to prevent numerical hallucinations in order quantities
- Reworked the agent prompt and history flow to stop a repeated tool-call loop after BOM confirmation
Separating quantity extraction into its own tool was a non-trivial reliability decision. Early prompts allowed the language model to alter numbers while reasoning over the rest of the order. A dedicated extraction step narrowed the task and made quantities explicit before graph retrieval. The prompt and chat-history flow were also revised so the agent stopped after confirmation rather than repeating the same tool call.
Vision & Robot Control
The vision dataset began with 30 original photographs and grew to 180 training images through rotation, zoom, skew, flip, and blur augmentation. Six validation images and three test images were retained. A black tape background was ultimately selected because the original white surface produced glare and a red intermediate surface confused the red-flower class.
YOLOv11 locates and classifies five tulip colors, and the controller prioritizes the detected flower with the largest image Y-coordinate as the most accessible pick. In parallel with image capture, the team recorded 64 camera-to-robot examples. A PyTorch network mapped the visual input to six UR5 joint angles, after which a KNN compensator with k = 3 applied local TCP offsets learned from nearby calibration attempts.
- Trained YOLOv11 for up to 100 epochs with patience-based stopping
- Reduced the joint-prediction mean squared error from 0.3 to 0.2 by using the most informative Y-pixel coordinate
- Added a waypoint 5 cm above the flower followed by a linear descent to reduce collisions
- Adjusted gripper opening and tool-center-point orientation for safer approach and pickup
Engineering Decisions & Trade-Offs
Using only the Y-coordinate reduced validation MSE from 0.3 to 0.2 because the robot’s dominant motion followed that image direction, while bounding-box dimensions and X position contributed little. The lower statistical loss did not translate to successful grasping, however: physical success remained 1/9 and the error direction varied across the workcell.
A deeper network would have increased overfitting risk on 64 points, while collecting a substantially larger dataset was not practical within the project schedule. The team instead used local KNN correction. Each taught location stored the TCP delta required to turn the network prediction into a successful approach, allowing nearby examples to correct spatially varying error without retraining the primary model.
Mechanical changes supported the same objective. The gripper was opened only halfway to avoid adjacent stems and bin walls, TCP orientation was iterated for clearance, and the robot moved to a waypoint 5 cm above the detected flower before descending linearly. These choices treated grasp success as a coupled perception, calibration, trajectory, and end-effector problem rather than as a model-score problem alone.
Results & Validation
Measured physical result KNN compensation increased pickup success from 1 of 9 trials to 8 of 9 without collecting a new neural-network training set. The eightfold change is the project’s strongest result because it connects a specific engineering pivot to physical robot behavior.
The small validation set showed correct classifications across the five documented colors at the selected threshold, but six validation and three test images are not enough to claim broad visual robustness. The report also does not document the joint-angle MSE units, target normalization, or exact split construction, so 0.3 versus 0.2 is useful only as a within-project model comparison. Likewise, the final report records the chatbot and vision-guided arm working independently and an integrated script being completed. Live physical testing of the combined path remained future work.
Impact, Limitations & Next Steps
The project demonstrates a pragmatic hybrid architecture: graph relationships keep order logic explicit, computer vision identifies physical inventory, a compact model provides a global motion estimate, and local memory corrects the errors that matter at the workcell. The result is more informative than the MSE improvement alone because it shows why physical validation must remain the deciding metric for robot calibration.
The demonstrated domain is limited to artificial tulips, five colors, one camera arrangement, predefined destinations, a small calibration set, and one documented nine-attempt pickup comparison without repeated batches or uncertainty estimates. The next milestone is to physically test the confirmed BOM/BOO path through repeated picks, reacquire the scene after every removal, and update inventory state. Broader flower types would also require pose- and stem-aware perception, adaptive gripper opening, and a larger evaluation set that separates detection, localization, grasp, and placement success.
