Lirui (Leroy) Wang

I am a Ph.D. student advised by Prof. Russ Tedrake at MIT Computer Science and Artificial Intelligence Laboratory (MIT CSAIL). I also work closely with Prof. Kaiming He. Before coming to MIT, I received my B.S. and M.S. degrees at University of Washington where I was very fortunate to work with Prof. Dieter Fox and collaborate with NVIDIA.

My research interest lies in Machine Learning and Robotics. In particular, I am interested in developing algorithms and systems that can generalize in the complex and unstructured real-world environments. To achieve this, I work on fleet learning that can scale with the heterogeneous data.

Email: liruiw [at] mit (dot) edu [Google Scholar] / [Github] / [Twitter] / [LinkedIn]



    Publications

    Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
    Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He
    Neural Information Processing Systems (Spotlight), Neurips 2024
    arxiv, Code, Website, Models, HF Code, Video, Bibtex
    GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
    Pu Hua*, Minghuan Liu*, Annabella Macaluso*, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang
    Conference on Robot Learning (CoRL), 2024
    arxiv, Code, Video, Demo, Website, Bibtex
    PoCo: Policy Composition From and For Heterogeneous Robot Learning
    Lirui Wang, Jialiang Zhao, Yilun Du, Edward Adelson, Russ Tedrake
    Robotics: Science and Systems (R:SS), 2024
    arxiv, Bibtex, Website, Video, News,
    GenSim: Generating Robotic Simulation Tasks via Large Language Models
    Lirui Wang, Yiyang Ling*, Zhecheng Yuan*, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang
    Workshop on Language Grounding and Robot Learning (Best Paper), CoRL 2023
    International Conference on Learning Representations (Spotlight), ICLR 2024
    arXiv, Bibtex , Website, Demo, Code, Dataset, Video, News
    Robot Fleet Learning via Policy Merging
    Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, Russ Tedrake
    International Conference on Learning Representations (ICLR), 2024
    arxiv, Bibtex, Code, Video
    Does Decentralized Learning with non-IID Unlabeled Data Benefit from Self Supervision?
    Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake
    International Conference on Learning Representations (ICLR), 2023
    arXiv, Bibtex, Code, Video
    Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks
    Jialiang Zhao, Yuxiang Ma, Lirui Wang, Edward H. Adelson
    Conference on Robot Learning (CoRL), 2024
    arxiv, Bibtex, Code, Website
    3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation
    Guangyao Zhou*, Nishad Gothoskar*, Lirui Wang, Joshua B. Tenenbaum, Dan Gutfreund, Miguel Lázaro-Gredilla, Dileep George, Vikash K. Mansinghka
    International Conference on Computer Vision (ICCV), 2023
    arXiv, Bibtex, Website, Code
    NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
    Allan Zhou*, Moo Jin Kim*, Lirui Wang, Pete Florence, Chelsea Finn
    The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2023
    arxiv, Website, Bibtex
    PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation
    Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, Nikolai Smolyanskiy
    International Conference on Robotics and Automation (ICRA), 2022
    arXiv, Bibtex, Video, Blog, Poster
    Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
    Lirui Wang, Xiangyun Meng, Yu Xiang, Dieter Fox
    IEEE Robotics and Automation Letters (RAL), 2022
    arXiv, Bibtex, Video, Poster, Website, Code
    Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
    Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox
    The Conference on Robot Learning (CoRL), 2021
    arXiv, Bibtex, Video, Poster, Website, Code
    Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection
    Lirui Wang, Yu Xiang, Dieter Fox
    Robotics: Science and Systems (R:SS), 2020
    arXiv, Bibtex, Video, Website, Code

    Education

  • 2021 - 2024, Massachusetts Institute of Technology
    Ph.D. student in AI and Decision Making
  • 2020 - 2021, University of Washington
    Master of Science in Computer Science
  • 2016 - 2020, University of Washington (Summa Cum Laude)
    Bachelor of Science in Computer Science
    Bachelor of Science in Electrical Engineering
    Minor in Mathematics