Building and evaluating generalist language agents

How can we build generalist language agents that assist us in the digital and physical world? First, we will discuss OSWorld, a new interactive, executable testbed for generalist agents that follow natural language instructions to perform long-horizon real-world tasks in virtual machines in real-time. Second, we will examine recent and ongoing efforts to train generalist agents, including learning from both human and automatic language feedback. Third, we will introduce AgentArena, a newly launched dynamic evaluation platform for OS agents that challenges state-of-the-art foundation model agents. Finally, we will explore ongoing and future directions in evaluating and building generalist agents that leverage the AgentArena platform.

About the Vector Distinguished Lecture Series

The Vector Distinguished Lecture Series is a formal gathering of academic and industrial data scientists across the Greater Toronto Area (GTA) to discuss advanced topics in machine learning and its goal is to build a stronger machine learning community in Toronto. The talks will be given by international and local faculty and industry professionals.

The seminar series is intended for university faculty and graduate students in machine learning across computer science, ECE, statistics, mathematics, linguistics, and medicine, as well as PhD-level data scientists doing interesting applied research in the GTA. The Toronto machine learning community will be stronger when we know each other and know what problems people are working on.

Vector Distinguished Lecture Series is currently open to the public remotely. Researchers in the Vector community will have opportunities to meet speakers in person. All talks will be streamed online and be posted on the Vector YouTube Channel.