InteractiveWorld
People have long dreamed of an assistant that is always there: one that sees what they see, remembers what they forget, and guides them through what they have never done. Wearable AI is bringing that dream within reach.
Can today’s agents truly help people in the physical world?
The first environment that realistically simulates a human and a wearable AI assistant interacting in the physical world: the assistant sees through the human’s eyes and talks, the human decides and acts, and the world responds.
Pick a request, then watch the glasses assistant guide a simulated person through a photorealistic home.
Actions will appear here once you ask for help.
Done
The human agent turns guidance into short commands that name only an action and its
targets, such as [GRAB] <cup> or [PUTINSIDE] <mug> <dishwasher>.
The simulator works out the motion and renders what the person sees.
Grab
[GRAB] <olive oil>
Put
[PUT] <pan> <burner>
Put Inside
[PUTINSIDE] <mug> <dishwasher>
Open
[OPEN] <fridge>
Pour
[POUR] <tray> <olive oil>
Turn On
[TURNON] <microwave>
Cut
[CUT] <tomato> <knife>
Stir
[STIR] <mug> <chopstick>
Wipe
[WIPE] <sponge> <counter>
Use Computer
[USECOMPUTER] <keyboard>
The human in InteractiveWorld is an autonomous agent. It hears free-form advice, decides what to do from what it sees, and its body works out the rest: where to stand, how to get there around furniture and through doors, how to lean or squat, and how to grip each object.
[WALK] <washer>
Finds a reachable place to stand by the target, plans a collision-aware A* route across rooms and walks it with continuous turns. A closed door on the way is opened, then the walk goes on.
[GRAB] <apple> → [PUT] <apple> <counter>
Samples standing spots around the target and checks reach, clearance and collisions. The closest valid one wins, so the human walks round the island to the apple, then over to the counter.
[GRAB] <apple> · <bottle> · <box> · <pan> · <cup> · <bag>
The grasp follows each object's shape and parts: a pinch for small or flat things, a cylindrical wrap for tall ones, a round grip otherwise, and the palm on the bar for handles.
The first closed-loop benchmark for egocentric AI assistants, with 120 multi-turn tasks in six categories. In every task, each instruction shapes what the human does next, and success is read from the world’s final state.
The model under test plays the assistant and sees only the human’s first-person view. A fixed MLLM plays the human and only follows its guidance. InteractiveWorld carries out every action, so the assistant’s words decide the outcome.
Sees only the first-person view and the dialogue
Executable actions appear here
State stState st+1
Final State Check
✓Salad on the table
✓Cola not on the table
Task success
120 human-reviewed tasks, 20 per category, each ending in object states the simulator can check. Every clip below is a real InteractiveBench episode played by the oracle assistant, sped up, with its dialogue and actions overlaid.
Guides the user step by step, based on their progress.
ExampleGriddled Tomato. A first-time cook follows a recipe card. The assistant hands out the next step only once the last one is done.
I haven't made this dish before. Guide me one step at a time.
Plans steps within time and resource limits, and replans on the fly.
ExampleCooking for Four Guests. Four allergies, nine menu cards, one oven and one stove, 30 minutes: the assistant picks the safe maximum and schedules it.
No dairy, shellfish, gluten or sesame. Most safe dishes in 30 min?
Describes the scene so blind or low-vision users can act on their own.
ExampleKitchen Cleanup. The human agent gets no images. The assistant reads the cleanup guide and talks the user through glass, food and dishes.
What's on these surfaces? Guide me through the cleanup, one item at a time.
Makes everyday decisions from what it has seen plus world knowledge.
ExampleSupplier Comparison. An order for 40 filters is read in one room, two quotes with different pack sizes in another. The cheaper offer for the full order wins.
Watching the requisition being read.
Warns before a mistake happens, while avoiding needless interruptions.
ExampleLeaving Home. The fridge and the window were opened earlier. The assistant stays quiet, then stops the user at the exit.
Watching quietly. Nothing needs saying yet.
Finds missing objects by remembering where they went.
ExampleMisplaced Cola. The user leaves a red cola on a dining-room table, puts water in the fridge and takes a call, then asks where the cola went.
Watching where the red cola goes.
Twelve frontier MLLMs take turns in the assistant’s seat, with Gemini 3.1 Pro as the same human agent for all of them. A task passes only if every goal state holds at the end; MCR also gives credit for partial progress.
Pass rate (%) over all 120 tasks
Click a category or metric to re-rank the models
Share of passed capability checks (%). Color shows distance from the 12-model mean of each capability.
Even the strongest model helps the human finish fewer than half of the tasks (41.7%), and every other model stays at or below 33.3%. With an oracle assistant, the same human passes all of them.
The best models reach 85% on Decision, which often hinges on one scene judgment, but only 20% on Guidance and 25% on Planning, which demand continuous tracking, error spotting and replanning. Success also drops quickly as tasks get longer.
Misreading the scene or losing track of changes causes 70.4% to 85.7% of first errors. In a closed loop, a single misperception or forgotten state cascades into every instruction that follows.
GLM 5.3 Flash and Claude Fable 5.1 have almost the same average capability score, yet GLM scores far lower on communication and completes 11.7 points fewer tasks. Success is bounded by the weakest link in the chain, not by the average.
Even on tasks they pass, models lead the human through many unnecessary actions: 2.35× to 3.66× the oracle’s steps on Guidance, and up to 6.75× on Accessibility. Efficiency depends on the task more than on the model.
InteractiveWorld turns “does this assistant actually help?” into a question with a verifiable answer. Its state-based rewards also open the door to training assistants that truly help people.
@misc{wang2026interactiveworld,
title = {InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants},
author = {Wang, Qinsi and Wen, Wei and Zhou, Chong and Suri, Saksham and Liu, Zechun and Wu, Lemeng and
Li, Hai Helen and Chen, Yiran and Krishnamoorthi, Raghuraman and Qian, Qi},
year = {2026},
eprint = {2610.33966},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33966}
}