All pull requests

Pull requests

• Fix DummyModel rolling likelihood for choice-free documents

  #1428·AaravMehta-07 opened Oct 13, 2026·

  ·

• Fix JudgeLLM crash with huggingface_hub 2.x for transformers/vllm judges

  #1427·li12345678990 opened Oct 12, 2026·

  ·
  2

• Use rouge-score-rs for ROUGE when it is installed

  #1426·kholdrex opened Oct 11, 2026·

  ·
  1

• Add KMMLU (Korean-native MMLU, 45 subjects)

  #1425·CY-HYUN opened Oct 10, 2026·

  ·
  1

• Add the TauBench retail benchmark with a tool-calling agent and an LLM user simulator

  #1423·zhongerdeajie opened Oct 10, 2026·

  ·
  1

• Fix Metrics.drop corpus aggregation to use mean instead of max

  #1422·salignatmoandal opened Oct 9, 2026·2 of 3 tasks·

  ·
  1

• Fix broken code examples in custom model docs

  #1421·singularity-14 opened Oct 9, 2026·

  ·

• Reject a missing details file when reloading an eval

  #1420·b423016 opened Oct 8, 2026·3 tasks done·

  ·

• Show scored sample counts in the results table

  #1419·b423016 opened Oct 8, 2026·2 tasks done·

  ·
  1

• fix: show a single vLLM data-parallel progress bar on the driver

  #1418·Steeve-Crypto opened Oct 7, 2026·2 tasks done·

  ·
  1

• fix(metrics): ignore boundary whitespace in word-unit counts

  #1416·March-7 opened Oct 4, 2026·

  ·

• Don't change global torch matmul precision when importing swiss_legal metrics

  #1415·jayzuccarelli opened Oct 2, 2026·

  ·
  1

  3

• fix(tasks): bypass deprecated dataset scripts with parquet data files (#1406)

  #1413·irvallensar opened Oct 1, 2026·

  ·
  1

• fix(metrics): raise when pass@k is called with k > n

  #1412·woodforestsen opened Oct 1, 2026·

  ·
  1

• fix: litellm judge backend silently scores API failures as 0 instead of surfacing them

  #1411·johnny-glitch12 opened Sep 30, 2026·

  ·

• Add sample count column to markdown results table

  #1409·manan45 opened Sep 29, 2026·1 of 2 tasks·

  ·
  1

• Fix DROP exact match treating a repeated span as one copy

  #1408·SashaMIT opened Sep 29, 2026·3 tasks done·

  ·

• Fix LiveCodeBench scoring every generation as -2 under spawn and forkserver

  #1401·David-Wu1119 opened Sep 27, 2026·

  ·
  1

  1

• math tasks: pass math_normalizer to maj@n as normalize

  #1399·no-hup opened Sep 25, 2026·

  ·

• Fix model arg parsing for endpoint URLs

  #1398·saptarshihalder opened Sep 25, 2026·

  ·

• fix(metrics): let pass_at_k_letters grade boxed / chain-of-thought answers (fixes #1094)

  #1392·Linxiushen opened Sep 22, 2026·

  ·
  1

• docs: explain how to run built-in multilingual tasks

  #1391·buyan201430-code opened Sep 19, 2026·

  ·
  1

• Allow configuring the LiteLLM cache directory

  #1390·lindicaphxag-tech opened Sep 19, 2026·

  ·
  1

  1

• Bump the actions group with 8 updates

  dependenciesgithub_actions
  #1388·dependabot[bot] opened Sep 17, 2026Bot·

  ·
  1

• Fix logging call that hides why torch.compile failed

  #1387·David-Wu1119 opened Sep 16, 2026·

  ·
  1