Lirui (Leroy) Wang

I am an (incoming) researcher at OpenAI. I am interested in foundation models and embodied AI (robotics).

Previously, I received my Ph.D. in Computer Science from MIT, where I worked with Prof. Russ Tedrake and Prof. Kaiming He. Before coming to MIT, I received my B.S. and M.S. degrees at University of Washington and worked at NVIDIA with Prof. Dieter Fox.

[Email] / [Full Publications] / [Github] / [Twitter] / [LinkedIn]

Selected Publications

Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He
Neural Information Processing Systems (Spotlight), Neurips 2024
arxiv, Code, Website, Models, HF Code, News, Video, Bibtex

                            @inproceedings{wang2024hpt,
                              author    = {Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He},
                              title     = {Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformer},
                              booktitle = {Neurips},
                              year      = {2024}
                           }

PoCo: Policy Composition From and For Heterogeneous Robot Learning
Lirui Wang, Jialiang Zhao, Yilun Du, Edward Adelson, Russ Tedrake
Robotics: Science and Systems (R:SS), 2024
arxiv, Website, Video, News, Bibtex,

                            @inproceedings{wang2024poco,
                              author    = {Lirui Wang, Jialiang Zhao, Yilun Du, Edward H. Adelson, Russ Tedrake},
                              title     = {Policy Composition From and For Heterogeneous Robot Learning},
                              booktitle = {Robotics: Science and Systems (R:SS), 2024},
                              year      = {2024}
                           }

GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
Pu Hua*, Minghuan Liu*, Annabella Macaluso*, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang
Conference on Robot Learning (CoRL), 2024
arxiv, Code, Video, Demo, Website, News, Bibtex

                            @inproceedings{hua2024gensim2,
                              author    = {Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang},
                              title     = {GenSim2: Scalable Robotic Data Generation with Multi-modal LLMs},
                              booktitle = {The Conference on Robot Learning},
                              year      = {2024}
                           }

GenSim: Generating Robotic Simulation Tasks via Large Language Models
Lirui Wang, Yiyang Ling*, Zhecheng Yuan*, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang
Workshop on Language Grounding and Robot Learning (Best Paper), CoRL 2023
International Conference on Learning Representations (Spotlight), ICLR 2024
arXiv, Code, Website, Demo, Dataset, Video, News, Bibtex

                            @inproceedings{wang2023gensim,
                              author    = {Lirui Wang, Yiyang Ling*, Zhecheng Yuan*, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang},
                              title     = {GenSim: Generating Robotic Simulation Tasks via Large Language Models},
                              booktitle = {The Twelfth International Conference on Learning Representations},
                              year      = {2023}
                           }

Robot Fleet Learning via Policy Merging
Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, Russ Tedrake
International Conference on Learning Representations (ICLR), 2024
arxiv, Code, Video, Bibtex

                            @article{wang2023fleet,
                             title={Robot Fleet Learning via Policy Merging},
                             author={Wang, Lirui and Zhang, Kaiqing and Zhou, Allan and Simchowitz, Max and Tedrake, Russ},
                             journal={The Twelfth International Conference on Learning Representations},
                             year={2023}
                           }

Does Decentralized Learning with non-IID Unlabeled Data Benefit from Self Supervision?
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake
International Conference on Learning Representations (ICLR), 2023
arXiv, Code, Video, Bibtex

                            @inproceedings{wang2022does,
                          title={Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?},
                          author={Wang, Lirui and Zhang, Kaiqing and Li, Yunzhu and Tian, Yonglong and Tedrake, Russ},
                          booktitle={The Eleventh International Conference on Learning Representations},
                          year={2022}
                        }

Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox
The Conference on Robot Learning (CoRL), 2021
arXiv, Code, Video, Poster, Website, Bibtex

                                 @inproceedings{wang2022goal,
                                   title={Goal-auxiliary actor-critic for 6d robotic grasping with point clouds},
                                   author={Wang, Lirui and Xiang, Yu and Yang, Wei and Mousavian, Arsalan and Fox, Dieter},
                                   booktitle={Conference on Robot Learning},
                                   pages={70--80},
                                   year={2022},
                                   organization={PMLR}
                                 }

Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection
Lirui Wang, Yu Xiang, Dieter Fox
Robotics: Science and Systems (R:SS), 2020
arXiv, Code, Video, Website, Bibtex

                                 @article{wang2019manipulation,
                                title={Manipulation trajectory optimization with online grasp synthesis and selection},
                                author={Wang, Lirui and Xiang, Yu and Fox, Dieter},
                                journal={Robotics: Science and Systems (R:SS), 2020},
                                year={2019}
                              }