Siheng Zhao

I am a first-year CS Ph.D. Student at University of Southern California, advised by Prof. Yue Wang.

Previously, I was honoured to work with Prof. Tao Yu, Prof. Lin Shao, and Dr. Jiangmiao Pang at The University of Hong Kong, National University of Singapore, and Shanghai AI Lab.

Email  /  Google Scholar  /  Semantic Scholar  /  Github  /  Twitter

profile photo
  • [Jan, 2024] One paper was accepted by EACL 2024.
  • [Jan, 2024] Two papers were accepted by ICLR 2024 as spotlight.
  • [Dec, 2023] Awarded SenseTime Scholarship (30 undergraduates in the field of AI in China).
  • [Dec, 2023] Awarded Nanjing University Top-Grade Scholarship (highest honor in Nanjing University).
  • [July, 2023] One paper was accepted by ICCV 2023.
  • [June, 2023] One paper was accepted by IROS 2023.

My research interests now lie at the intersection of Embodied AI and Natural Language Processing. My long-term goal is to build the next generation of autonomous and intelligent agents that can proactively sense, plan, and interact with both the physical and digital world. To achieve this goal, I primarily focus on:

  • Design effective reasoning and learning algorithms for both physical (embodied) and digital agents: Text2Reward (ICLR'24 Spotlight)
  • Develop backbone models (LLMs, VLMs, World Models) for versatile agents: Lemur (ICLR'24 Spotlight)
  • Build robust agent benchmarks with executable environments and human-centric metrics: OSWorld, GRUtopia

Specifically, I'm interested in the following areas: Embodied AI, Language Agent, Vision-language Models. Previously, I was also interested in Robotics, especially the simulation, perception and manipulation of deformable objects: DiffClothAI (IROS'23), ClothesNet (ICCV'23), TieBot.

Selected Publications

* denotes equal contribution. For the full publication list, please refer to my Google Scholar .

profile photo OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu
arXiv, 2024
[arxiv] [project]

profile photo Text2Reward: Dense Reward Generation with Language Models for Reinforcement Learning
Siheng Zhao*, Tianbao Xie*, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, Tao Yu
International Conference on Learning Representations (ICLR) 2024 Spotlight
[arxiv] [project]

profile photo DiffClothAI: Differentiable Cloth Simulation with Intersection-free Frictional Contact and Differentiable Two-Way Coupling with Articulated Rigid Bodies
Siheng Zhao*, Xinyuan Yu*, Siyuan Luo, Gang Yang, Lin Shao
International Conference on Intelligent Robots and Systems (IROS) 2023
[paper] [project] [video]

USC logo University of Southern California
Ph.D. in Computer Science (2024 - )
Advisor: Prof. Yue Wang
NJU logo Nanjing University
B.Eng. in Artificial Intelligence (2020 - 2024)
Overall GPA: 94.4/100, Ranking: 1/97
NUS logo National University of Singapore
Exchange in Computer Science (2023)
Overall GPA: 4.0/4.0
Work Experience
AILAB logo Shanghai AI Lab, OpenRobot Group
Research Intern (2024)
Advisor: Dr. Jiangmiao Pang
HKU logo The University of Hong Kong, NLP Group
Research Assistant (2023)
Advisor: Prof. Tao Yu
  • Conference Reviewer:
    • Annual Meeting of the Association for Computational Linguistics (ACL) 2024
    • Neural Information Processing Systems (NeurIPS) 2024
    • European Conference on Computer Vision (ECCV) 2024
    • IEEE International Conference on Robotics and Automation (ICRA) 2024
  • Journal Reviewer:
  • OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, Shanghai AI Lab, 2024
  • Text2Reward: Reward Shaping with Language Models for Reinforcement Learning, SenseTime, 2024
  • Text2Reward: Reward Shaping with Language Models for Reinforcement Learning, Shanghai AI Lab, 2023, 2024
  • Large Language Model for Robotics: a High-level Planner, Nanjing University NLP Group, 2023
Honors & Awards
  • Outstanding Graduate of Nanjing University, 2024
  • Travel Award of the International Conference on Learning Representations, 2024
  • Jiangsu Province Study Abroad Scholarship, 2024
  • SenseTime Scholarship, 2023, awarded to 30 undergraduates in the field of AI in China
  • Nanjing University Top-Grade Scholarship, 2023, the highest honor in Nanjing University
  • Bao Gang Scholarship & Special Prize Nomination, 2023
  • Heng Fang Scholarship, 2022
  • People's Scholarship at Nanjing University, 2022, 2023
  • National Scholarship, 2021, the highest honor in China's University
  • China Telecom Scholarship, 2021
  • Outstanding Student Leader at Nanjing University, 2021, 2022
  • Outstanding Student in Social Practice at Nanjing University, 2021

This homepage is designed based on Jon Barron's website and deployed on Github Pages.

Copyright 2024 © Siheng Zhao