Leroy (Lirui) Wang

Lirui (Leroy) Wang

I am a Ph.D. student advised by Prof. Russ Tedrake at the MIT Computer Science and Artificial Intelligence Laboratory (MIT CSAIL). I also work closely with Prof. Kaiming He. Before coming to MIT, I received my B.S. and M.S. degrees at the University of Washington where I was very fortunate to work with Prof. Dieter Fox and collaborate with NVIDIA.

My research interest lies in Robotics and Machine Learning. In particular, I am interested in developing algorithms and systems for robots that can generalize in the complex and unstructured real-world environments. To achieve this, I currently work on fleet learning that can scale with heterogeneous robot data and scalable robot simulation systems.

Email: liruiw [at] mit (dot) edu [Google Scholar] / [Github] / [Twitter] / [LinkedIn]

Recent News

Checkout the website and paper for the new work policy composition from and for heterogeneous robot learning.

Our paper on code generation for robotic simulation won the outstanding paper award at the Language Robotics (LangRob) workshop CoRL 2023 and is accepted by ICLR 2024 as Spotlight. Checkout the website, demo, GPTs, and code.

Our paper on fleet learning via policy merging is accepted by ICLR 2024. Checkout the code.

Publications

PoCo: Policy Composition From and For Heterogeneous Robot Learning
Lirui Wang, Jialiang Zhao, Yilun Du, Edward Adelson, Russ Tedrake
GenAI4DM Workshop (ICLR), 2024
arxiv, Bibtex, Website, Video

                            @inproceedings{wang2024poco,
                              author    = {Lirui Wang, Jialiang Zhao, Yilun Du, Edward H. Adelson, Russ Tedrake},
                              title     = {Policy Composition From and For Heterogeneous Robot Learning},
                              booktitle = {arXiv:2410.01362},
                              year      = {2024}
                           }

Robot Fleet Learning via Policy Merging
Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, Russ Tedrake
International Conference on Learning Representations (ICLR), 2024
arxiv, Code, Video, Bibtex

                            @article{wang2023fleet,
                             title={Robot Fleet Learning via Policy Merging},
                             author={Wang, Lirui and Zhang, Kaiqing and Zhou, Allan and Simchowitz, Max and Tedrake, Russ},
                             journal={arXiv preprint arXiv:2310.01362},
                             year={2023}
                           }

GenSim: Generating Robotic Simulation Tasks via Large Language Models
Lirui Wang, Yiyang Ling*, Zhecheng Yuan*, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang
Workshop on Language Grounding and Robot Learning (Workshop Best Paper), CoRL 2023
International Conference on Learning Representations (Spotlight) (ICLR), 2024
arXiv, Website, Demo, Code, Dataset, Video, GPTs, Bibtex

                            @inproceedings{wang2023gensim,
                              author    = {Lirui Wang, Yiyang Ling*, Zhecheng Yuan*, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang},
                              title     = {GenSim: Generating Robotic Simulation Tasks via Large Language Models},
                              booktitle = {arXiv:2310.01361},
                              year      = {2023}
                           }

Does Decentralized Learning with non-IID Unlabeled Data Benefit from Self Supervision?
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake
International Conference on Learning Representations (ICLR), 2023
arXiv, Code, Bibtex, Video

                            @inproceedings{wang2022does,
                          title={Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?},
                          author={Wang, Lirui and Zhang, Kaiqing and Li, Yunzhu and Tian, Yonglong and Tedrake, Russ},
                          booktitle={The Eleventh International Conference on Learning Representations},
                          year={2022}
                        }

3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation
Guangyao Zhou*, Nishad Gothoskar*, Lirui Wang, Joshua B. Tenenbaum, Dan Gutfreund, Miguel Lázaro-Gredilla, Dileep George, Vikash K. Mansinghka
International Conference on Computer Vision (ICCV), 2023
arXiv, Website, Code, Bibtex

                            @inproceedings{zhou20233d,
                             title={3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation},
                             author={Zhou, Guangyao and Gothoskar, Nishad and Wang, Lirui and Tenenbaum, Joshua B and Gutfreund, Dan and L{\'a}zaro-Gredilla, Miguel and George, Dileep and Mansinghka, Vikash K},
                             booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
                             pages={21625--21636},
                             year={2023}
                           }

NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
Allan Zhou*, Moo Jin Kim*, Lirui Wang, Pete Florence, Chelsea Finn
The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2023
arxiv, Website, Bibtex

                            @inproceedings{zhou2023nerf,
                                title={Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis},
                                author={Zhou, Allan and Kim, Moo Jin and Wang, Lirui and Florence, Pete and Finn, Chelsea},
                                booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
                                pages={17907--17917},
                                year={2023}
                              }

PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, Nikolai Smolyanskiy
International Conference on Robotics and Automation (ICRA), 2022
arXiv, Bibtex, Video, Blog, Poster

                           @inproceedings{kamenev2022predictionnet,
  title={Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation},
  author={Kamenev, Alexey and Wang, Lirui and Bohan, Ollin Boer and Kulkarni, Ishwar and Kartal, Bilal and Molchanov, Artem and Birchfield, Stan and Nist{\'e}r, David and Smolyanskiy, Nikolai},
  booktitle={2022 International Conference on Robotics and Automation (ICRA)},
  pages={8936--8942},
  year={2022},
  organization={IEEE}
}

Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
Lirui Wang, Xiangyun Meng, Yu Xiang, Dieter Fox
IEEE Robotics and Automation Letters (RAL), 2022
arXiv, Bibtex, Video, Poster, Website, Code

                              @article{wang2022hierarchical,
  title={Hierarchical policies for cluttered-scene grasping with latent plans},
  author={Wang, Lirui and Meng, Xiangyun and Xiang, Yu and Fox, Dieter},
  journal={IEEE Robotics and Automation Letters},
  volume={7},
  number={2},
  pages={2883--2890},
  year={2022},
  publisher={IEEE}
}

Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox
The Conference on Robot Learning (CoRL), 2021
arXiv, Bibtex, Video, Poster, Website, Code

                                 @inproceedings{wang2022goal,
                                   title={Goal-auxiliary actor-critic for 6d robotic grasping with point clouds},
                                   author={Wang, Lirui and Xiang, Yu and Yang, Wei and Mousavian, Arsalan and Fox, Dieter},
                                   booktitle={Conference on Robot Learning},
                                   pages={70--80},
                                   year={2022},
                                   organization={PMLR}
                                 }

Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection
Lirui Wang, Yu Xiang, Dieter Fox
Robotics: Science and Systems (R:SS), 2020
arXiv, Bibtex, Video, Website, Code

                                 @article{wang2019manipulation,
                                title={Manipulation trajectory optimization with online grasp synthesis and selection},
                                author={Wang, Lirui and Xiang, Yu and Fox, Dieter},
                                journal={arXiv preprint arXiv:1911.10280},
                                year={2019}
                              }

Education

2021.9 - present, Massachusetts Institute of Technology
Ph.D. student in AI and Decision Making
2020.6 - 2021.3, University of Washington
Master of Science in Computer Science
2016.9 - 2020.6, University of Washington (Summa Cum Laude)
Bachelor of Science in Computer Science
Bachelor of Science in Electrical Engineering
Minor in Mathematics

Selected Honors

MIT EECS Fellowship (Xianhong Wu Fellow), MIT, 2021
Bob Herbold Data Science Fellowship, UW, 2021
Patricia Lynch and Theodora & Eugene Russell Memorial Scholarship, UW, 2020
Lawrence & Lucille Frey Endowed Scholarship, UW, 2019
Tau Beta Pi Scholarship, UW, 2018

Teaching

TA, 6.4210 Robotic Manipulation (MIT, Fall 2022)
TA, CSE 455 Computer Vision (UW, Winter 2021)
TA, CSE 543 Deep Learning (UW, Fall 2020)
TA, CSE 473 Artificial Intelligence (UW, Winter 2020)
TA, CSE 455 Computer Vision (UW, Fall 2019)

Patent

Model-based reinforcement learning for behavior prediction
Nikolai Smolyanskiy, Alexey Kamenev, Lirui Wang, David Nister, Ollin Boer Bohan, Ishwar Kulkarni, Fangkai Yang, Julia Ng, Alperen Degirmenci, Ruchi Bhargava, Rotem Aviv
US Patent App. 17/453,055, 2022

Misc Projects (Pre-PhD)

Pose Tracking
UW

Server Mover
Microsoft

Auto Cooperation
DJI