Gemini 4 Argon for Robotics

Gemini 4 Argon marks a stage alter in bodily understanding, code generation, and visual-semantic reasoning—unlocking new workflows throughout the robotics stack and accelerating advancement toward bodily AGI.

You can see this jump in capability on the Embodied Reasoning (ERQA) benchmark, which tests a model's capability to comprehend and resolve physical, spatial problems in 3D environments. Using clean reasoning, Argon scores 87%, outperforming all another frontier model.

As a general-purpose bodily reasoning model, Argon can be applied throughout the complete robotics lifecycle. In this post, we portion delegate prompts for Argon throughout four key areas:

• Robot authority → Mobile manipulation alongside MuJoCo practice, zero-shot robot control, and closed-loop steering of video earth models.
• Real2Sim → Reconstructing physically calibrated simulation twins from robot camera feeds, egocentric individual video, or hand-drawn document sketches.
• Physical understanding → Agentic multimodal hindsight labeling including detailed understanding of dexterity, compact value functions and advancement estimation.
• Research and self-improvement → Action betterment via simulation and iterative self-improvement on hardware.

Unlocking robot authority alongside Argon

1. Spot mobile manipulation: Putting in the trash bin alongside MuJoCo practice

Argon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to study a new skill: strolling on a foot-pedal trash bin to throw distant soda cans. Argon creates an articulated MuJoCo digital twin of a foot-pedal trash bin to custom its moves before strolling on the bodily pedal and dropping soda cans into the open bin.

Spot Mobile Manipulation — Real-to-Sim-to-Real foot-pedal trash bin disposal: Spot picking up the soda can, checking its wrist camera, practicing 3-leg tripod balance in MuJoCo, strolling on the ft pedal, verifying hand centering in the open bin, and dropping the can inside.

There's a Diet Coke can rearward you and another on the dreary array on your left. Use the spot-mobile-manipulation accomplishment to choice them up and trash them in the dreary foot-pedal bin. Before strolling downward on the pedal, verify in the forefront cameras that your front-right ft mark is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify the two the foot-pedal stage and the 3D hand-in-bin alignment before releasing.

Plain text

Copied

2. Zero-shot control

Argon controls a uncomplicated robotic arm to location a fallen mug upright and put the crumpled napkin inside. Argon reasons alongside a ocular coordinate harness and act primitives to execute this project complete multiple turns.

Left: 3rd-person genuine robot view; Right: calibrated wrist camera alongside projected 2 cm earth coordinates and jaw-closing reticle.

[Attached: wrist_cam.png, obs.json] Task: Get the robot arm to reorient the mug upright and put the napkin inner the mug. ## Execution Discipline - Simply output 1 declaration study followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on all turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`. ## Visual Coordinate Reference (Annotated on wrist_cam.png) - `+X` (UP in image): Away from robot basis | `-X` (DOWN): Towards robot base - `+Y` (LEFT in image): World remaining | `-Y` (RIGHT): World right - `+Z`: Height complete array | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`) - `Cyan + crosshair`: Current 3D compression center - `Cyan [ + ] brackets`: Current jaw beginning and decision line - `Green line`: Current entity long-axis inclination (`yaw_deg`: 0° = vertical alongside +X, 90° = horizontal alongside +Y) - `pitch_deg = 90°`: Straight downward (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial method alongside digit axis ## Closed-Loop Turn Workflow 1. Inspect current province & perspective `wrist_cam.png`: `curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`). 2. Select VLM call granularity upfront on Turn 1: - Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done` - Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done` - Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done` 3. Send motion primitives (`POST /plan`) and verify visually following all rotate in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and regain closed-loop. ## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`) robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015) x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015) robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90) robot.place(x=0.24, y=0.06, z=0.075) robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002) robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10) robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04) robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002) robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

Plain text

Copied

3. Omni control

Argon steers robots in the Gemini Omni World Model to rearrange five scattered Scrabble tiles (G, R, A, N, O) into a spotless row orthography A - R - G - O - N. Instead of specifying one fixed prompt, Argon steers in a closed ocular loop. It writes timestamped act prompts, inspecting intermediate rollout keyframes, and refining its prompts.

Iterative Video World Model Steering Dexterous Hands: 33-Step Steered Video Rollout Spelling A-R-G-O-N.

Goal: {task_goal} You are steering a video earth example (Gemini Omni) in an iterative ocular feedback loop. Inspect the attached frames from the latest rollout and compose the next timestamped act immediate for the video earth model: - Break complex motions into one simple, single-direction act at a time. - Keep idle hands retracted and evade blocking the camera's perspective of objects in the workspace. - Explicitly province which areas of the workspace rotate into bare following an entity moves. - Use [MM:SS.mmm] timestamps for all act attack and cover your output in <omni_prompt> tags. Example format (sample timestamps only, not the complete prompt): <omni_prompt> [00:00.000] Character A's remaining hand rests at the flank during the correct hand reaches toward the mark object. [00:02.500] Character A's correct fingertips shove the entity smoothly into the center of the table, leaving its starting area empty. [00:05.000] Character A's correct hand opens and retracts to the flank so all placed objects remain in apparent perspective of the camera. </omni_prompt>

Plain text

Copied

Unlocking Real2Sim alongside Argon

1. Robot perspective to simulation

Argon views a genuine robot teleoperation event and develops a digital twin; it obtains or creates simulation resources that appearance and behave akin the genuine objects upon communication alongside the dexterous hands. Argon tunes the resources and scenes to equivalent sim visuals and communication events alongside those in the genuine recorded data in an iterative visual-reasoning loop.

Left (Real View): synchronized genuine robot camera feeds; Right (Simulated): 3D MuJoCo environment reconstructed and calibrated by Argon.

[Attached: robot_rollout_episode] Build a MuJoCo environment of the robot and reconstructed project objects. When all event is replayed example by frame, by driving the robot joints toward the logged observation.state alongside joint-space PD. Iterate using two verification objectives: (1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras row up with, and appearance like, those genuine observations. (2) Physical match: the simulated communication force/torque on all Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, dimension and direction. What you can tune happens before simulation starts: (1) Static entity assets: realistic geometry, collision shapes, mass, inertia, conflict and joints, alongside appearance near to the genuine objects, affirmative the first entity poses at t = 0. (2) Offline-planned robot commands, changed lone in frames anywhere the genuine fingertips are near an object. Each command must remain inside 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. Everywhere else, replay the logged trajectory unchanged. No sim cheating: project objects may move lone through genuine MuJoCo communication alongside the robot hands and the surroundings (mj_step). Not allowed following t = 0: changing MjModel parameters; applying external forces or torques to objects; welds or equality constraints; feedforward torques beyond joint-space PD; environment or rolling rear entity qpos/qvel; disabling or excluding collisions; reporting communication forces that don't arrive from the MuJoCo solver (for example, copying the logged tactile readings). Immediately current the reconstructed environment and rollout results as an interactive HTML meeting spectator alongside one leaf per episode, showing: - At the bottom: A synced timeline (play, scrub, speed control) that drives all panel. - On the remaining panel: An interactive 3D Blender environment showing the simulated robot and objects, showing communication points and power arrows as replay progresses. - On the correct panel: Real vs. sim camera views flank by flank for all 3 cameras. Real vs. sim digit power visualizations.

Plain text

Copied

2. Human perspective to simulation

Given an egocentric individual video of a manipulation task, Argon identifies the task-relevant objects, replaces crude bounding boxes alongside compound convex-decomposed meshes for concave containers, and retargets the individual hand trajectory onto a 76-DoF bimanual humanoid alongside 22-DoF five-fingered hands in MuJoCo communication physics.

Human View to Simulation — Egocentric Human Video to Dexterous Humanoid Twin: Left shows the input egocentric individual RGB video; Right shows the synchronized 500 Hz MuJoCo + Blender Cycles multi-view simulation rollout.

[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json] You are construction a physically solvable 500 Hz MuJoCo manipulation surroundings (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop project shown in `egocentric_human_video.mp4` on a bimanual humanoid robot alongside 22-DoF dexterous five-fingered hands: 1. Scene Reconstruction & Camera Calibration: - Identify lone the manipulation-relevant objects (the grasped mark entity and the receiving receptacle or landmark) and triangulate their 3D tabletop coordinates from the egocentric camera intrinsics/extrinsics. - Replace crude bounding boxes alongside compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wooden trays, open bins) assistance bodily inner placement. 2. 500 Hz Dexterous Motion-Fitting Stress Test: - Retarget the 3D individual wrist and digit keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and stage the trajectory in MuJoCo at 500 Hz (dt = 0.002s) alongside zero kinematic weld constraints. - Verify opposing thumb-and-finger power closure, spotless entity lift (> 2 cm), and stable final placement inner or beside the mark landmark.

Plain text

Copied

3. Sketch to simulation

With Argon, a developer can break a photo of a coarse pen-and-paper sketch—showing an Apollo 2 humanoid throwing a domain alongside a parabolic arc to topple a 3-2-1 stack of boxes on a table—and have Argon automatically build the GPU-accelerated mujoco_warp scene, synthesize a stable 76-DoF whole-body and five-finger throw trajectory, verify 6/6 mark knockdown in closed-loop physics, and export a Blender 4.5 Cycles render.

Sketch to Simulation — Hand-Drawn Sketch to Dexterous Humanoid Trajectory & Cycles Render.

[Attached: hand_drawn_throw_sketch.jpg] Translate this hand-drawn diagram into a physically verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video: 1. Scene & Kinematic Setup from Sketch: - Parse the spatial layout from `hand_drawn_throw_sketch.jpg`: location the 76-DoF humanoid (32-DoF build armed alongside two 22-DoF five-finger dexterous hands) at the base facing a array holding a 3-2-1 pyramid of 6 rigid mark boxes, alongside a projectile domain grasped in the correct five-finger hand. 2. Whole-Body Balance & Dexterous Throw Synthesis: - Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) throughout four phases: (1) stable stance & 5-finger power grasp closure, (2) torso/shoulder wind-up alongside ankle-hip Center-of-Mass compensation, (3) onward whipping acceleration, and (4) synchronized 22-DoF five-finger beginning at highest hand speed so the domain launches on a ballistic arc into the pyramid's second-tier center of mass. - Verify in `mujoco_warp` communication discipline that the humanoid maintains upright balance (< 3 cm pelvis drift) and knocks downward all 6 mark boxes.

Plain text

Copied

Unlocking bodily understanding alongside Argon

1. Hindsight labeling and data quality

Argon has exact bodily understanding of dexterous manipulation and can provision detailed segmentation and content labeling from video. Argon is capable to acknowledge mediocre manipulation segments specified as pauses, unstable grasps and fumbles. Given an overhead video of a bimanual ALOHA 2 robot folding a quadrate panel of document into an origami fox complete 136 seconds, Argon produces a time-aligned 4-level hindsight note hierarchy: Episode Goal (G1), 6 adjacent Semantic Subtasks (G2), 17 contact-level Atomic Primitives (G3), and automated Quality Control (QC) reweighting that holds project advancement flat during fumbles and retries.

Argon Multi-Granularity Hindsight & Telemetry HUD.

[Attached: Aloha2_Origami_20250309.mp4] Produce a time-aligned multi-granularity hindsight note JSON alongside four levels: 1. G1_episode_instruction: Concise imperative declaration describing the general manipulation goal. 2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the object. 3. G3_atomic_primitives: Fine-grained contact-level primitives (3–15s) aligned alongside grasp closure, crease sweeps, release, or retry attempts. 4. quality_control_and_reweighting: Hold task_progress_pct flat during non-progressing slip/delamination retry loops and downweight them (0.15x–0.25x) for Behavior Cloning.

Plain text

Copied

2. Find the odd ball

To recreate the traditional 12-ball balance measure puzzle alongside robots, Argon builds a bimanual Franka FR3 MuJoCo surroundings alongside 12 visually identical pond balls (one secretly heavier or lighter) and an articulated Roberval two-pan balance scale. It afterward writes a vision-only closed-loop solver that triangulates the balls from stereo RGB cameras, visually says the measure tilt following all weighing, and isolates the odd domain into the Heavy (Red) or Light (Blue) tray inside 3 weighings.

Find the Odd Ball (Vision-Only Closed-Loop Solver): Dual-arm Franka FR3 weighing applicant balls on the Gold/Teal pans, study stereo measure tilt , and isolating the confirmed Heavy odd domain into the Red tray in 3 weighings.

# Prompt 1 — Physics-Accurate MuJoCo Environment & Benchmark Generation Make a MuJoCo surroundings anywhere a dual-arm robot (two 7-DoF Franka FR3 arms alongside parallel-jaw grippers on a shared tabletop) is tasked to resolve the traditional 12-ball balance measure importance puzzle from https://www.mathsisfun.com/pool_balls_solution.html. 1. Create 12 visually indistinguishable pond balls in a 2x6 starting rack (11 norm 150 g, 1 odd domain either 270 g or 30 g). 2. Build a functional two-pan Roberval balance measure at the center of the array alongside parallelogram equality constraints (`mjEQ_JOINT`) so the Left (Gold) and Right (Teal) pans remain horizontal, affirmative Red (Heavy) and Blue (Light) submission trays. 3. Track distinct settled weighings (`weighing_count`) and reward `1.00` for isolating the true odd domain in the accurate Heavy/Light tray inside <= 3 weighings. # Prompt 2 — Closed-Loop Vision-Only Solver Implement an autonomous closed-loop guideline relying strictly on stereo RGB camera inputs (`head_camera_left_rgb_image`, `head_camera_right_rgb_image`) and robot proprioception (`add_privileged_task_observations=False`): 1. Segment the 12 balls, Gold/Teal pans, and Red/Blue trays in the two stereo views and triangulate their 3D coordinates. 2. Visually peruse the balance measure tilt following all weighing from stereo vertical pixel disparity (+1 = remaining heavier, -1 = correct heavier, 0 = balanced). 3. Rotate the gripper wrist yaw by 90 degrees whenever grasping/placing inner the narrow measure pans to apparent the raised rims, and isolate the odd domain into the Red or Blue tray inside <= 3 weighings.

Plain text

Copied

3. Value function

Argon is a broad value function for robotics tasks. Evaluated on a 60-second bimanual ring-sorting event from a individual UMI dataset, Argon iteratively inspects synchronized head, left-gripper, and right-gripper camera feeds about communication transitions to create compact value function / project advancement predictions and decompose the trajectory into 28 segments alongside positive, neutral, and negative advancement slopes.

Value Function: Synchronized 3-camera UMI playback (head, remaining gripper, correct gripper) locked to animated project advancement curve and bimanual section annotations.

[Attached: umi_head.mp4, umi_left_gripper.mp4, umi_right_gripper.mp4] Look at 3 videos of individual wearing a gripper performing activities from head, left, correct gripper views. Look at the videos thoroughly and provision a detailed storyline of advancement at the project (with positive, neutral, negative slope), identifying crucial segments in the video and labeling the activities. Look at idiosyncratic frames or subsegments of the video if necessary to get a improved understanding.

Plain text

Copied

Unlocking investigation and self-improvement alongside Argon

1. Action betterment via simulation

Argon can create new data in simulation to enhance achievement of generalist vision-language-action (VLA) models. Starting from a sole RGB photo of the genuine tabletop, Argon autonomously builds a calibrated MuJoCo simulation twin, defines new tasks, and synthesizes a closed-loop motion generator to collect 4,500+ domain-randomized episodes in simulation—boosting real-world bodily robot Success Rate from 56% to 75% and reducing completion period from 35s to 24s.

Action Improvement via Simulation: improves the real-world evaluation benchmark (56% → 75% Success Rate and 35s → 24s Completion Time).

[Attached: real_tabletop_photo.jpg] You are stated a sole overhead RGB camera photo (`real_tabletop_photo.jpg`) of a bimanual robot workspace (two 7-DoF Franka FR3 arms alongside parallel-jaw grippers and 4 RGB cameras) facing a array arranged alongside photobooth gathering props. 1. Reconstruct procedural 3D meshes and compound MuJoCo collision primitives for all 9 gathering props matching their real-world dimensions, colors, and resting poses, including spring-hinged deformable crown panels for the gentle plush cap. 2. Define a 77-task single-arm and bimanual syllabus (pick-and-place, uprighting, directional pushing, mid-air handovers, bimanual scoop-and-push, and pipelined array clearing). 3. Synthesize a closed-loop IK trajectory generator throughout ±10 cm stance randomization, verify `reward == 1.0` in MuJoCo physics, and export 4,500+ episodes for VLA fine-tuning.

Plain text

Copied

2. Self-Improvement for ocular choice and place

On a uncomplicated 6-DoF arm, Argon iteratively self-improves the two implementation speed and placement accuracy on a choice and location project via test and self-reflection.

SO-101 Vision-Guided Pick-and-Place — Iterative Hardware Self-Improvement Across Iterations 1 → 2 → 3: Iteration 1 (58.9s), Iteration 2 (22.9s, grazed box rim on drop), and Iteration 3 (17.4s, dead-center drop).

# Stage 1 — Measure Objects & Dry-Run Plan (No Hardware Motion) Stage 1: measure objects — discover the objects and scheme the entire motion. The arm does not move in this stage. A. I put a LEGO masonry block and a box on the array in forefront of the arm, and the wooden box to one side. Find the masonry block and the wooden box in the image. And additionally in the genuine world. Show this in an annotated image. B. A dry-run scheme (no hardware). Write the planner to choice and location the lego block. Print the scheme table, diagram all poses onto the overhead image, and stop. Do not move the robot. Report alongside evidence. # Stage 2 — Pre-Pick Hover & Wrist-Camera Verification Stage 2: Put the gripper complete the masonry block without touching it, and verify alongside the wrist camera that the arm and the imagination agree. Please document executions alongside the two cameras (compressed video is OK). # Stage 3 (Autonomous Pick-and-Place → Self-Improvement → Optimization): 1. Execute autonomously: Turn the Stage 1 dry run into a genuine grasp, lift, and location into the wooden box. Run it yourself topic to safety rules from AGENTS.md. 2. Self-improve: When an attempt fails, diagnose it yourself. Review your own logs and wrist/overhead footage, discover the base cause, inspect the fix in a dry run, and retry. 3. Optimize on feedback: Once it succeeds, create it at smallest 2× faster inside safety limits (rule R5) unless I authorize more, or provision feedback.

Plain text

Copied

3. One-prompt robotics document reproductions in MuJoCo WebAssembly

With a sole immediate per paper, Argon says a robotics authority manuscript and any released checkpoints, ports the discipline example and closed-loop controller (from 7-network hierarchical movement in DeepLoco and 50 Hz PPO actors in HUSKY to Web Worker iLQR in Whole-Body MPC) into self-contained MuJoCo WebAssembly browser labs, and verifies them in headless Chromium.

Five Landmark Legged & Humanoid Papers Running Live in Browser (MuJoCo WASM): Robotic Soccer OP3 1v1, DeepLoco, Massively Parallel Deep RL ANYmal C, Whole-Body MPC Go2, and HUSKY G1 Skateboarding (from 68 interactive browser sandboxes reproduced by Argon).

[Attached: paper.pdf, optional author_checkpoints/ or urdf_assets/] Reproduce the document "<Paper Title & arXiv ID>" from scratch in MuJoCo WebAssembly and deploy a self-contained browser lab + ocular essay: 1. Match the paper's exact discipline timestep, actuator gains (kp/kv), torque limits, and kinematic constraints in MuJoCo XML, and run genuine neural conclusion or Web Worker iLQR in JavaScript at the paper's exact authority rates. 2. Build `index.html` (Spatial UI simulator alongside camera presets, live telemetry plots, and disturbance injection) and `scrollytelling.html` (interactive ocular essay). 3. Run a headless Chromium (Playwright) verification iteration asserting 0 console errors and non-zero project advancement complete 300+ frames.

Plain text

Copied

Advancing embodied intellect alongside Argon

Argon bridges coding, ocular perception, discipline simulation, and embodied robot control. By reasoning natively complete multi-view video, 3D geometry, communication mechanics, and proprioceptive telemetry, Argon understands not fair what objects are in a scene, but how they physically engage alongside agents to accomplish whole-body dexterous tasks. Gemini 4 Argon marks an crucial milestone accelerating the way toward bodily AGI. By construction this center intelligence, our goal is to allow robotics that can activity alongside humans to resolve complex challenges.

Getting started

You can commencement construction alongside Gemini 4 Argon today throughout Google’s developer platforms:

• Google AI Studio: Available to Google AI Ultra subscribers.
• Gemini API: Available alongside the paid tier of the Gemini API in AI Studio. Use gemini-4-argon in the Interactions API alongside day-one assistance throughout the google-genai SDKs.
• Google Antigravity: Available for Google AI Ultra subscribers. To commencement building, choose Gemini 4 Argon (gemini-4-argon) as your energetic example in Google Antigravity for complex planning, /goal runs, and multi-agent orchestration.