A defined starting point
Each attempt starts from a pinned repository snapshot and task specification. Agents inspect the actual source and work through explicit file and command tools.
FLAMKI LABS / RESEARCHResearch 01 · Preliminary
HorizonSWE
- tasks
- 5
- repositories
- 3
- model aliases
- 10
- attempts
- 42
Longer work.
Clearer evidence.
Researching how coding agents navigate repositories, make changes, and verify their work over multiple steps.
RESEARCH 01
Published 14 October 2026. This preliminary study documents Pilot 01, a five-task repository-level evaluation: eight models in the core study, plus two in a compact extension. Long-horizon evaluation is the research focus; task horizon length has not yet been independently validated.
01 / OBSERVED OUTCOMESOpen the results.
Download results CSV Open the results.
See what happened.
A useful evaluation records the whole attempt, including limits and errors. This release reports outcomes and resource use without assigning a model ranking.
8 models × 5 tasks · one attempt per cell · 40 turns · 16,384 output tokens per request
40Recorded attempts
8Patches graded
0Full passes
5Tasks represented
Attempt outcomes
Terminal outcomes across five tasks per model. Listed in run order.
Graded failureNo patchBudget limitEmpty responseService errorFormat error
Claude Sonnet 5
5 attempts
Gemini 3 Flash
5 attempts
GPT-oss 120B
5 attempts
Qwen3 Next 80B
5 attempts
DeepSeek R1
5 attempts
GPT-5.4 Nano
5 attempts
Gemini 3.1 Flash Lite
5 attempts
Granite 4.0 Small
5 attempts
A test failure means a submitted patch was graded. Other outcomes describe generation or service limits. Partial test passes are not a score.
Model records
8 models · core pilot| Model / alias | Test-graded | Full passes | Reported credits* | Avg. turns | Inspect model attempts |
|---|---|---|---|---|---|
Claude Sonnet 5anthropic/claude-sonnet-5@default | 1 / 5 | 0 | $4.856 | 34.6 | |
Gemini 3 Flashgoogle/gemini-3-flash-preview | 1 / 5 | 0 | $0.798 | 35 | |
GPT-oss 120Bopenai/gpt-oss-120b | 0 / 5 | 0 | $0.000421 | 1 | |
Qwen3 Next 80Bqwen/qwen3-next-80b-a3b-instruct | 0 / 5 | 0 | $0.112 | 20.6 | |
DeepSeek R1deepseek-ai/deepseek-r1-0528 | 1 / 5 | 0 | $0.850 | 7.6 | |
GPT-5.4 Nanoopenai/gpt-5.4-nano-2026-03-17 | 1 / 5 | 0 | $0.224 | 32.6 | |
Gemini 3.1 Flash Litegoogle/gemini-3.1-flash-lite-preview | 4 / 5 | 0 | $0.265 | 23 | |
Granite 4.0 Smallibm/granite-4.0-h-small | 0 / 5 | 0 | $0.033 | 3.2 |
*Sum of inference credits reported for the listed attempts, including unsuccessful runs when usage metadata was returned. Account usage of $8.88 also includes diagnostics and access probes. Alias versions and downstream provider identity require confirmation.
HOW TO READ THIS PILOT
Eight patches were test-graded. None passed every required check. The remaining attempts ended without a patch, at a budget limit, or with an endpoint/interface error. These results describe this scaffold and these limits; they do not establish general model capability or task difficulty.
02 / THE EVALUATION PIPELINEThe task is only
The task is only
the beginning.
Context. Execution. Verification.
Each needs its own evidence.
An isolated environment
Commands run in fresh offline Linux containers with the current edits. Task solutions, hidden tests, host files, and model credentials are withheld from the agent.
A verifiable outcome
Submitted patches run in a separate evaluator. Both existing tests and task-specific checks must pass. The generation status and test results are recorded separately.
GRADING CONTROLS
Check the evaluator, too.
Before model inference, the reference and negative controls were run locally. These checks validate the pipeline; they do not prove exhaustive test coverage.
5/5Reference patches passed
5/5Comment-only patches failed
Inspect the protocol and its limits
Core pilot
Five Python-focused tasks across three repositories. Eight model aliases, one attempt per cell, up to 40 turns and 16,384 output tokens per request. Generation has a 20-minute loop limit; commands have a 60-second limit.
Native tools request low reasoning, except GPT nano, which uses none. DeepSeek and Granite use JSON actions with provider-default reasoning. Sampling stays at provider defaults. These interface differences limit comparability.
Compact extension
Two additional accessible Gemini aliases attempt one task, with up to 20 turns and 2,048 output tokens per request. This exploratory protocol is reported separately and is not comparable to the core pilot.
Of 36 additional catalog aliases checked, 34 returned access errors. Those probes are not coding evaluations. Models unavailable through this endpoint have no task score.
Budget and context
The run uses a $10 daily Kaggle credit ceiling. A conservative cost reserve is checked before each request. Earlier complete tool exchanges may be pruned to retain the task and recent context; hard context limits remain enforced.
Every nonempty patch left by a credit cutoff receives local grading. Budget-limited generation remains visible in the data.
Research limitations
This is a small, selected sample with a single attempt per model/task. The same tasks informed harness development. There is no held-out development split, independent horizon validation, or contamination-free claim.
The proxy endpoint has a staging designation and mutable aliases. Independent task review, verified model revisions, repeated runs, and a larger task sample remain necessary before a reliable public leaderboard.
03 / TASK COVERAGERepository work.
Repository work.
Different demands.
Five task themes in this pilot.
Prompts and solutions remain private.
Tool registry lifecycle
STATE & LIFECYCLEBundle installation
PACKAGING & INTEGRATIONConfiguration overlays
CONFIGURATIONEvent inspection
OBSERVABILITYPer-step runtime configuration
AGENT RUNTIMEThese are high-level coverage descriptions, not public task specifications. The separate AUTH-001 demonstration ↗ provides an inspectable example of our task packaging and is not part of this pilot.
04 / TECHNICAL NOTERead the methods.
Read the methods.
Inspect the record.
The report documents task preparation, the agent scaffold, grading, interface revisions, observed outcomes, and threats to validity.
Read the reportFLAMKI LABSRESEARCH / 01
HorizonSWE
A pilot of repository-level
coding-agent evaluation.
OCTOBER 2026
PRELIMINARY TECHNICAL REPORT
PRELIMINARY TECHNICAL REPORT
Inspect a sample.
Scope your dataset.
Bring your target capability, language, and evaluation requirements. We’ll agree on a five-task scope, acceptance criteria, custom pricing, and a delivery schedule.
Each task includes a repository snapshot, reproducible setup, reference patch, tests, and recorded validation.