RobIn Robot Interactive Intelligence Lab


2317 Speedway

Austin, Texas 78712

RobIn Lab is directed by Roberto Martín-Martín, and is part of the University of Texas at Austin, Department of Computer Science, UTAI Lab and Texas Robotics.

The research in RobIn explores at the intersection of robotics, computer vision and machine learning, with an emphasis on creating robotic solutions that accomplish interaction-rich tasks in the real world. Our research revolves around the question: what are the mechanisms that enable physically intelligent behavior in embodied agents? Our hypothesis is that intelligence is not a purely computational process, but an interplay between perceiving, reasoning, learning and interacting physically with the world.

Keywords in our research include: learning (reinforcement learning, imitation learning, representation learning), planning (motion planning, task planning, TAMP), control (optimal control, force control), perception (active, interactive, multimodal perception), 2D and 3D reasoning, and Bayesian state estimation, sim2real. Application domains we find exciting involve: mobile manipulation, stationary manipulation, contact-rich manipulation, social and visual navigation.

We are always looking for outstanding new members and collaborators for RobIn. Please, fill in this form and state clearly if you are an UG/MSc/PhD from UT, or a prospective visitor.


Jun 20, 2024 Two works accepted at RSS 2024: ScrewMimic learns bimanual manipulation behaviors from a single human video and Lang4Sim2Real helps bridge the sim2real gap with language.
Jun 10, 2024 Beginning of the UT Summer Academy offered by the RobIn lab. 25 students will fabricate their own copies of BaRiFlex (design by Gu-Cheol), collect data and train imitation learning policies for manipulation. It is going to be fun!
May 17, 2024 Roberto gives a talk at the ICRA24 workshop Vision-Language Models for Navigation and Manipulation (VLMNM) on “Language as Bridge between Simulation and the Real World” (Albert’s RSS paper). During his talk, he connected live with Austin to teleoperate remotely our Tiago robot using TeleMoMa (Shivin’s paper and code), our teleoperation framework for mobile manipulators. Japan-US connection!
Mar 18, 2024 Roberto was selected as a part of the New Generation Star Project at ICRA sponsored by Nokov!
Oct 28, 2023 Our work miniBEHAVIOR (paper) has been accepted for presentation at two NeurIPS workshops: Agent Learning in Open-Endedness (ALOE) and Generalization in Planning (GenPlan). MiniBEHAVIOR is a novel benchmark that extends the 2D gridworld miniGrid to include semantically meaningful tasks from BEHAVIOR-1K, enabling fast development and prototyping for reasoning in household activities. Congratulations Jiaheng and the rest of the team!
Oct 20, 2023 Roberto gives a keynote speaker talk at the Texas Robotics Symposium on the topic “Technologies for Household Robotics”.
Oct 18, 2023 Roberto participates in an expert panel discussion at the Texas Robotics Symposium on the topic “Household Robotics” together with Dr. Biswas, Dr. Stone and Dr. Fineberg.
Oct 1, 2023 Invited keynote talk by Roberto at the IROS workshop on Integrated Perception, Planning, and Control for Physically and Contextually-Aware Robot Autonomy. His talk was titled “Perceiving to Interact, Interacting to Perceive”.
Oct 1, 2023 First event of Queer in Robotics, a new initiative launched by Roberto and Ruchira with partners from others universities. Queer in Robotics will be organizing events at the upcoming major robotics conferences with the goal of creating a welcoming space to the LGBTQ+ community within the robotics space.
Sep 6, 2023 Invited Lecture by Roberto in the new UT Austin MOOC course Essentials of AI for Life and Society. Roberto introduced the topic “Probabilistic Modeling” to a live online audience of hundreds of participants.
Sep 1, 2023 Dr. Ben Abbatematteo joins the RobIn Lab as a postdoctoral fellow and Assistant Professor of Practice leading the Robot Learning FRI stream.
Jul 10, 2023 Roberto presents RobIn’s work at the RSS’23 workshop Robotics and AI: The Future of Industrial Assembly Tasks. His presentation touches on our research on “Learning Contact-rich Interaction Control”.
Jun 6, 2023 RobIn was awarded with one of the Good Systems grants to explore the effects of AI and robotics in society. Our project will explore the consequences of household and domestic robots in people’s lives, including the life of household workers.
Jun 1, 2023 Two papers presented at ICRA: M-EMBER with Bohan Wu and Fei-Fei Li, and HRON with Michael Lingelbach et al. M-EMBER was nominated to Best Student Paper and Best Manipulation Paper!
May 31, 2023 Andrey Kurenkov presents Scene Graph Memory at AAMAS23 in London, our work for object search in unknown, dynamically changing environments represented as scene graphs. An extension of this work will be also presented at ICML23.
Apr 25, 2023 Roberto presents at Prof. Kristen Grauman’s group on “What can robots learn from videos about long-horizon activities?” comprising the work on Paprika (CVPR23) and MaskViT (ICLR23).
Apr 23, 2023 One paper accepted at RSS23: “Causal Policy Gradient for Whole-Body Mobile Manipulation”, a collaboration with Jiaheng Hu and Peter Stone. Congratulations, Jiaheng!
Apr 14, 2023 Roberto gives a Spotlight talk at TEROS’23 about “Mobile Manipulation: Integration vs. Factorization” including some of the results of our recent RSS paper.
Mar 1, 2023 Final version of our ICLR’23 paper submitted! MaskViT: Masked Visual Pre-Training for Video Prediction, a collaboration with Agrim Gupta et al.
Feb 27, 2023 Two papers accepted at CVPR’23! Procedure-Aware Foundation Model for Instructional Video Understanding with Honglu Zhou and Juan Carlos Niebles and ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding with Le Xue et al.
Feb 24, 2023 Roberto presents RobIn at GradFest’23. This presentation summarizes some of the core ideas of our lab and broad future research directions.
Feb 8, 2023 Talk by Roberto at the Texas Innovation Center’s Engineering Education and Research Center. More info
Jan 25, 2023 Talk by Roberto on Datasets in Robotics at the Living and Working with Robots initiative, part of the Good Systems Challenge.
Jan 21, 2023 Two papers accepted at ICRA23: “Task-Driven Graph Attention for Hierarchical Relational Object Navigation”, a collaboration with Michael Lingelbach et al., and “M-EMBER: Tackling Long-Horizon Mobile Manipulation Via Factorized Domain Transfer”, together with Bohan Wu and Dr. Fei-Fei Li.
Jan 20, 2023 MaskVit, Masked Visual Pre-Training for Video Prediction, has been accepted at ICLR23!
Jan 5, 2023 Roberto delivers the William Pierson Field Lecture at Princeton on What do we want robots to do for us (and how do we get there)?
Dec 9, 2022 Talk by Roberto at Nuro on Robots and Humans.