Robot Task MCP Server

MCP server that exposes a simulated robot arm (via PyBullet) as tools so any MCP-compatible LLM agent can plan and execute pick-and-place tasks.

Difficulty: 1-week | Stack: Python, FastMCP, PyBullet, Anthropic SDK, NumPy

Who this is for

Developers exploring standardized LLM-to-robot interfaces — lets any MCP client control a simulated robot without custom integration code

Build steps

1. Spin up a PyBullet scene with a UR5 arm, a table, and 3-5 named objects; expose get_scene_state() returning object poses as JSON
2. Implement MCP tools: move_to_pose(x,y,z,gripper_open), get_camera_image() returning base64 RGB, reset_scene()
3. Build IK solver wrapper (use PyBullet’s calculateInverseKinematics) so tools accept Cartesian coords, not joint angles
4. Write a Claude agent loop that calls get_camera_image + get_scene_state to perceive, then chains move_to_pose calls to execute pick-and-place
5. Add a success detector (object in target zone?) that the agent can query to close the loop and retry on failure

Risks

• PyBullet IK is unreliable near singularities — arm gets stuck or oscillates; need joint-limit clamping and retry logic
• LLM spatial reasoning over raw xyz coords is poor — may need to add a semantic layer (e.g. ‘above red block’) mapped to coords server-side
• MCP stdio transport + PyBullet GUI thread conflict on some OSes; run headless (p.DIRECT) for server mode