Real repositories.
Substantive tasks.
Agents need to trace dependencies, diagnose failures, and make changes across a codebase.
We design tasks around these demands, with explicit requirements and defined acceptance criteria.
✳ SOFTWARE ENGINEERING DATA & EVALUATION
Engineering data
for frontier AI.
Repository-level datasets, reproducible environments, and evaluations
for teams training and testing advanced coding agents.
01 / OUR APPROACH
CONTEXTENVIRONMENTREFERENCEEVALUATION
TASK / ENVIRONMENT / REFERENCE / EVALUATION+
Real repositories.
Controlled conditions.
Repeatable runs.
Each task brings together a repository snapshot, execution environment, reference patch, and test suite.
Pinned dependencies and documented resets give each run a consistent starting point.
Explicit criteria.
Verifiable outcomes.
We validate reference solutions against task requirements and package the results for inspection.
Research teams can review the task, reproduce the environment, and examine the evaluation criteria.
Inspect a task ↗02 / TASK SPECIFICATION
Inspect the task.
Review the evidence.
An original JavaScript task with source, tests, and a reference patch.
Run the validation locally and inspect the recorded results.
SPECIFICATION / AUTH-00101 — THE TASK
Fix concurrent token refresh
The baseline client refreshes independently for each unauthorized response. Preserve its API while sharing one in-flight refresh across concurrent requests.
01 Preserve request paths and responses
02 Share a single in-flight refresh
03 Reuse fresh tokens for late 401s
04 Propagate failures and allow recovery
VERIFIED SAMPLE / AUTH-001 / NODE.JS 22.14.0Inspect the sample repository ↗
Baseline: 6/9 tests pass → Reference: 9/9 pass. Verified locally on macOS arm64. No live authentication service or container run is claimed.
Original, AI-assisted demonstration task · MIT licensed · All tests public · No customer code
03 / SOLUTIONS
Data for training.
Evidence for evaluation.
For research labs, coding-agent teams,
and AI data providers.
{ }
Engineering datasets
Original repository-level tasks for training and post-training. Debugging, API integration, and multi-file changes grounded in practical engineering.
⌘
Reproducible environments
Containerized execution, pinned dependencies, documented resets, and structured task configurations. A consistent foundation for every run.
↗
Agent evaluations
Custom benchmarks with explicit acceptance criteria, hidden tests, and structured reports to examine how agents solve engineering problems.
04 / INITIAL PILOT
Start with five tasks.
Inspect every deliverable.
A proposed starting scope, tailored to your target capability.
Custom pricing and a delivery schedule agreed before work begins.
Scope
Five original repository-level tasks in an agreed language and engineering domain. Difficulty, task format, and evaluation requirements are defined together.
Deliverables
Each task includes its specification, repository snapshot, setup and reset instructions, reference patch, test suite, and recorded validation output.
Acceptance
Agree on checks before production: a reproducible baseline failure, a passing reference solution, coverage of the required behaviors, and complete delivery documentation.
05 / ABOUT FLAMKI LABS
INDEPENDENT AI DATA & EVALUATIONFounded on engineering.
Built for frontier AI.
Flamki Labs is an independent AI data and evaluation company focused on high-quality software engineering datasets, reproducible task environments, and benchmarks for advanced AI systems.
Founded by Ayush Singh, Flamki Labs brings hands-on experience in software engineering and AI data production to the development of coding datasets and evaluations.
WORK WITH FLAMKI LABS
Build your next
evaluation dataset.
Share your target capabilities, task domains, and evaluation requirements.
We’ll scope a pilot and prepare a technical sample for review.