Benjamin Feuer

Benjamin Feuer

Hello! I am a Member of Technical Staff at OpenAthena, leading post-training efforts on the Marin project (fully open-source foundation models) in collaboration with Percy Liang at Stanford University, where I am also a Visiting Scholar. I completed my Ph.D. in Computer Science and Engineering at NYU as a member of the DICE Lab. Previously, I received a BA in Film Studies from Wesleyan University, an MFA in Screenwriting from Columbia University, and an MS in Computer Science from New York University. My awards include an NYU Outstanding Dissertation Award, a NeurIPS Spotlight award, and the Deborah M. Rosenthal Award (Best CS Qualifying Exam).

Research: I have wide-ranging research interests; some of my recent topics include data-centric factors in machine learning systems, robust LLM benchmarking, evaluation and alignment, and scalable data integration for very large databases.

Education: I am currently a Visiting Scholar at Stanford University, working with Ludwig Schmidt's group, and was previously a Postdoctoral Researcher there. I completed my PhD in Computer Science at New York University, advised by Chinmay Hegde. Previously, I studied at Columbia University and Wesleyan University. Other frequent collaborators include Micah Goldblum and Colin White and John P Dickerson and Juliana Freire.

News

  • 2026/8/27 New benchmark (+ code). Terminal-Bench-Science 0.1 evaluates AI agents on expert-curated scientific research workflows spanning the life, physical, Earth, mathematical, and engineering sciences. Its 70 tasks were selected from 920 proposals by 376 contributors across 22 countries; the best model (Claude Opus 5) resolves just 30%. Led by researchers at Stanford and built by the Terminal-Bench team.
  • 2026/7/30 New benchmark (+ code). Terminal-Bench 3.0 measures agent abilities at the frontier: 74 diverse, difficulty-calibrated tasks across 7 domains, built as a continuously improved open community benchmark; the best models achieve ~34%.
  • 2026/6/29 New benchmark (+ paper). Harbor-Index is a compact, high-signal meta-dataset for agentic evaluation: 82 tasks distilled from 6,627 candidates spanning 54 benchmarks integrated as Harbor adapters, with rigorous auditing to separate genuinely difficult tasks from broken ones. No agent-model pair tested clears 30%.
  • 2026/6/23 New paper (+ blog) introducing OpenThoughts-Agent, a fully open data curation pipeline for training agentic models. 100+ controlled ablations and a 100K-example training set; fine-tuned Qwen3-32B averages 44.8% across seven agentic benchmarks, surpassing the strongest existing open-data agentic model.
  • 2026/3/10 New first-author paper (+ code) introducing average bias-boundedness (A-BB), a framework that formally guarantees reductions of harm from measurable bias in LLM judges, while maintaining 61-99% correlation with original rankings.
  • 2025/9/29 New first-author paper (+ code) showing how design failures in LLM judge benchmarks silently undermine validity, turning judgment into noise.
  • 2025/8/1 New first-author paper (+ code) introducing WILDCHAT-50M, the largest public chat dataset to date, extending WildChat to include responses from over 50 open-weight models. The resulting SFT mixture outperforms recent Tulu-3 benchmarks using 40% fewer samples. ICML 2025.
  • 2025/7/15 New first-author paper (+ code) introducing MARVIS (Modality Adaptive Reasoning over VISualizations), a training-free method that enables even small vision-language models to predict any data modality with high accuracy..
  • 2025/6/24 New paper (+ code) introducing Open Thoughts, introducing strong public data recipes for reasoning models. ICML 2025 (Oral).
  • 2025/5/19 New paper on large reasoning models for agriculture, introducing AgReason, the first expert-curated open-ended science benchmark with 100 questions for agricultural reasoning.
  • 2025/3/1 New invited survey paper published in IEEE Data Engineering Bulletin on data systems and engineering, describing challenges and opportunities in the use of LLMs for data integration and discovery.
  • 2025/01/31 New paper (+ code). LiveBench is a leading benchmark for foundation models; unlike traditional benchmarks, questions and categories evolve over time. Featured in Gemini, Qwen and Deepseek technical reports. ICLR 2025.
  • 2025/01/31 New first-author paper (+ code). Do LLM-judge preferences translate to progress on other, more concrete metrics for alignment? No, because LLM judges exhibit implicit biases; without being prompted, they reweight existing criteria and implement their own standards by which they evaluate provided judgment criteria. This work emphasizes the importance of blending ground truth and LLM judge evaluations for foundation models. ICLR 2025.
  • 2025/01/31 New paper. We demonstrate a distortion-free watermarking method for images, based on a combination of a diffusion model's initial noise and generated Fourier patterns. ICLR 2025.
  • 2024/9/28 New first-author paper (+ code) describing BioTrove, the largest publicly accessible dataset designed to advance AI for biodiversity applications. We also release a suite of CLIP models trained using a subset of 40 million captioned images. We introduce several new benchmarks for rigorous assessment, report accuracy for zero-shot learning, and evaluations. NeurIPS 2024 (Spotlight), USDA Highlighted Project.
  • 2024/9/28 New first-author paper (+ code) describing TuneTables, a novel tabular classification and regression model which is competitive with boosted trees, and can scale to problems of any size. NeurIPS 2024.
  • 2024/6/24 New paper (+ code) introducing LiveBench, a benchmark for LLMs designed with test set contamination and objective evaluation in mind. LiveBench is designed to limit potential contamination by releasing new questions monthly, as well as having questions based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. Each question has verifiable, objective ground-truth answers, allowing hard questions to be scored accurately and automatically, without the use of an LLM judge. Featured in Venture Beat.
  • 2024/2/13 New paper (+ code) benchmarking the performance of tabular algorithms on the largest suite of datasets to date. NeurIPS 2023 (Datasets and Benchmarks).
  • 2023/11/07 New first-author paper studying the effects of two important dataset-level constituents: label set design, and class balance. NeurIPS 2023 (1st Workshop on Attributing Model Behavior at Scale) .
  • 2023/10/28 New first-author paper investigating sketching and feature-selection methods for prior-fitted networks. NeurIPS 2023 (Second Table Representation Learning Workshop) .
  • 2023/10/27 New first-author paper (+ code) introducing ArcheType, a simple, practical method for context sampling, prompt serialization, model querying, and label remapping, which enables large language models to solve CTA problems in a fully zero-shot manner. VLDB 2024.
  • 2023/08/01 New first-author paper introducing JANuS (Joint Annotations and Names Set), a collection of four new training datasets with images, labels, and corresponding captions, and conducting controlled investigations of factors contributing to robustness in image classification. TMLR 2023.

Publications

A full list of my publications can be found here.