Agent Evaluation Science NYC Takes on AI Agent Reliability

Agent Evaluation Science NYC will bring researchers and practitioners to downtown Manhattan on November 20, 2026, for a one-day symposium presented by EvalScience. The gathering centers on a consequential operating question: how much confidence should an organization place in the way it evaluates an AI agent?

The tentative program connects measurement methods, evaluation infrastructure and real-world systems. For teams deciding whether to put an agent to work, that sequence reaches beyond model selection into the evidence behind a deployment decision.

Registration is free and approval-based, and the published schedule runs from 9 AM to 6 PM Eastern. The exact venue address is reserved for confirmed attendees, so the practical first step is an application through the official listing.

When a passing score becomes an operating decision

An AI agent's success criterion can quietly become a product decision. If an evaluation rewards a plausible answer while the business needs a completed action, a strong result can leave the buyer with the wrong kind of confidence. The uncomfortable conversation begins when someone asks which outcome the measurement actually establishes.

Anthropic's explanation of agent evaluations describes why the mechanics matter: agents can call tools, change an environment and adapt over multiple turns. The company distinguishes the record of an agent's actions from the resulting state of the environment. A convincing completion message and an independently confirmed outcome are different pieces of evidence.

For an operator, that distinction changes what belongs in a purchase conversation. A score needs a task definition, a grading method and some account of the conditions under which it was earned. Without those details, comparing two systems risks comparing labels that happen to share a percentage sign.

That is the useful tension behind the New York symposium. The event can give participants a place to examine measurement choices together; it cannot promise that an afternoon of discussion will settle them. Builders preparing to attend should arrive with a particular failure case and the criterion that failed to catch it.

A program built around the measurement chain

EvalScience's program moves through frameworks and measures, benchmarks and infrastructure, and evaluation in real-world systems. Its structure creates a way to follow the same question across different responsibilities: what is being measured, how is that measurement produced, and where does it stop supporting the decision?

The listed speakers include Hanna Wallach of Microsoft Research, Jason Weston of Meta and NYU, and Yoav Artzi of Cornell University and Cornell Tech. Peter Henderson of Princeton University and Nouha Dziri of Cohere Labs are also scheduled, along with Olawale Salaudeen, whose UIUC assistant professorship is explicitly listed as an incoming fall 2027 appointment.

The fireside conversation brings together Ofir Press, Rachneet Kaur, Ida Momennejad and Saurabh Jha, with organizer-listed affiliations at Meta, J.P. Morgan, Microsoft Research and IBM Research, respectively. The relevance for deployment teams lies in the conversation's focus on evaluating agents in real settings. Institutional names establish who is involved; they do not establish an endorsement of any particular evaluation method.

The agenda remains tentative, which matters for anyone planning around a particular session. The stronger reason to pay attention is the chain of questions across the day. A useful discussion about a benchmark should connect to the work a grader rewards and the environment in which a team expects that behavior to hold.

Who has a reason to bring difficult evidence

Evaluation engineers can use the gathering to interrogate criteria that look straightforward until two graders disagree. Agent builders have a related interest in the relationship between a system's observed behavior and the conditions that produced it. Neither conversation benefits from sanding the awkward cases out of the evidence.

Enterprise operators have another stake: deciding what evidence belongs behind the permission to act. A team considering an agent for a workflow needs to know how failure will be recognized, who will inspect it and which actions warrant additional checks. Those are operating questions that can inform the evaluation brief before a technical team chooses its tests.

Founders and investors following agent infrastructure can listen for where measurement creates practical work. A proposed tool becomes easier to assess when its user can explain which uncertainty it removes and what remains for a human reviewer. The symposium offers a setting for that discussion without establishing that any particular company or investor will attend.

The published format includes contributed research, posters and networking alongside the invited sessions. Participants can compare approaches and discuss failure analyses, but no source can guarantee the quality of a connection in advance. The preparation within a team's control is to make its question concrete enough for someone else to challenge.

The people building the room, and the details still pending

The organizing committee includes Yining Hua, Hongbin Na, Manuel Cherep, Stephan Rabanser, Levi Lian, Yilun Hua and Abhinav Kumar. Their event roles are separate from the speaker roster and advisory board. The official site also distinguishes institutional affiliations from sponsorship, with Fleet AI linked separately as a sponsor.

EvalScience describes itself as an independent nonprofit working on evaluation science for agentic systems and broader AI. That focus gives this gathering a specific identity: a research and practitioner forum around the evidence used to assess agents. It leaves room for disagreement about methods rather than requiring participants to arrive with the same answer.

The venue page currently identifies downtown Manhattan and says detailed check-in, building access, accessibility and travel information will follow for confirmed attendees. Prospective participants should consult the official registration and program pages for updates. Research submissions closed on October 25, so older language inviting contributions should not be read as a current open call.

Between now and November 20, an interested team can choose the result it finds hardest to explain and assemble the evidence around it. Bringing that record into the discussion gives another researcher something concrete to examine, and gives the team a better chance of returning to its deployment decision with a sharper question.