Simulating Household Activities via Programs

In order to learn to perform complex activities, autonomous agents need to know the sequences of actions needed to reach a given task. Towards this goal, we present VirtualHome, a 3D environment allowing to simulate and generate videos of activities as sequences of actions and interactions.

VirtualHome is built upon 3 main blocks: A Knowledge Base of household tasks, containing instructions of how certain common tasks should be performed. VirtualHome Environment, a 3D simulator that simulate and generate videos of such tasks and the script generation models, which allow to generate programs from descriptions or video demonstrations.

Task Knowledge Base
We collect a Knowledge Base of Daily Indoor Activities. Every activity is given with a description of how to perform it and a script to execute it.
VirtualHome Environment
We present a virtual environment that allows to execute the activity programs, generating long videos of people performing the activities described in them.
Script Generation
We propose models to generate activity programs given descriptions or videos portraying the activity. This allows to learn to perform activities from examples.
Task Knowledge Base

We have collected a knowledge base of activities people do at home. For every activity, we have descriptions of different ways to perform it, and a program that describes how to execute it.

The Knowledge Base Contains
  • +500 activities
  • +2800 programs
  • +300 objects and +2700 interactions

Prepare coffee
Open coffee maker. Put filter in main basket, put ground coffee in filter, fill reservoir with water, turn on coffee maker.

VirtualHome Environment

We present a virtual environment that allows to execute programs of activties, generating videos of people performing activites inside apartments.

The environment has
  • 6 apartments with 4 characters available
  • +350 objects per scene
  • Instance and Semantic Label Annotation, Depth, Pose and Optical Flow
Set up table
VirtualHome Videos and Programs

  • Coming Soon
Code and pretrained models

  • Coming Soon

Xavier Puig


Kevin Ra

McGill University

Marko Boben

University of Ljulbjana

Jiaman Li

University of Toronto

Tingwu Wang

University of Toronto

Sanja Fidler

University of Toronto

Antonio Torralba


  title = {VirtualHome: Simulating Household Activities via Programs},
  author = {Puig, Xavier and Ra, Kevin and Boben, Marko and Li, Jiaman and Wang, Tingwu and Fidler, Sanja and Torralba, Antonio},
  booktitle = {Computer Vision and Pattern Recognition (CVPR)},
  year = {2018}
MIT University of Toronto