What is measured?
Benchmarks define both a capability claim and a held-out evidence boundary.
Data-centric coding agents
From Issues to
Learning Signals
A Survey of Data for Coding Agents
Coding-agent data is executable experience, not an isolated prompt–response pair.
6work levels
7synthesis paradigms
4pruning levels
5learning routes
01 Landscape
The durable unit is an executable experience bundle.
A useful datum binds the repository, visible task, environment, action interface, verifier, rollout graph, learning labels, and lineage into one versioned object.
Landscape map
Four data questions, connected by quality, evidence, governance, and lineage.
What makes a task executable?
Prompt, environment, verifier, and rubric must be constructed and validated together.
Which decisions are reusable?
Rollout graphs preserve shared states, counterfactual branches, recovery, and cost.
Which signal trains what?
Demonstrations, matched preferences, process evidence, and arenas serve different objectives.
dataset ≠ prompt collection
test pass ≠ task validity
trajectory ≠ flat transcript
RLVR ≠ optimizer
02 Benchmarks
A benchmark is a measurement boundary—and a potential data mine.
The evaluation unit expands from isolated functions to repositories, issue repair, professional workflows, terminal state, and autonomous AI R&D. As the work unit grows, environment and verifier quality dominate prompt count.
Function
unit tests
Repository
context + dependencies
Issue repair
F2P + P2P
Workflow
feature · review · release
Terminal
final machine state
AI R&D
trained checkpoint
Evaluation role
Measurement instrument
- Frozen or rolling held-out tasks
- Versioned harness and verifier
- Model × scaffold × budget × date
Training role
Executable learning arena
- Prompts, tests, traces, and rewards are consumed
- Policy distribution changes through training
- A separate evaluation boundary becomes mandatory
03 Task Factory
Prompt synthesis is joint task–environment–verifier construction.
A natural-looking issue is not yet a valid agent task. The complete package must fail in the buggy state, pass under valid alternatives, reject near misses, replay stably, and expose no gold implementation.
Trusted seed
issue · commit · test · skill
Task spec
intent · constraints · difficulty
Environment
snapshot · runtime · reset
Verifier stack
functional · process · audit
Freeze
hash · version · lineage
Functional
Build, fail-to-pass, pass-to-pass, hidden behavior, mutation, and final-state assertions.
Process
Grounding, evidence, recovery, efficiency, minimality, and calibrated process reward.
Audit
Future-history access, grader edits, vacuous passes, reward hacks, and sandbox escape.
Generate→Execute→Diagnose→Repair→Freeze
04 Trajectories
The raw object is a rollout graph, not a successful transcript.
Shared checkpoints and divergent actions expose local counterfactuals. A passing trace can exploit a weak test; a failing branch can contain the evidence and recovery that teach robust behavior.
Rollout graph → learning views
Prune causal redundancy while preserving grounded evidence, recovery, and same-state alternatives.
select · reject · route
Outcome, policy audit, cost, and diversity decide the product—not success alone.
retain recovery
Preserve localization, diagnostic evidence, rollback, and corrected execution.
compare siblings
Create causally local chosen/rejected actions for preference or value learning.
mask · compact
Keep exact tools, paths, diffs, errors, and causal order while removing repetition.
05 Post-training
Algorithms consume different evidence; names alone do not define the data regime.
Experience source, supervision, policy relation, and credit granularity are independent axes. The data question comes before the optimizer name.
Demonstration→SFT / behavior cloningcompact + recovery traces
Verified success→Rejection samplingiterative self-training
Same-state pair→DPO / IPO / DMPOlocal counterfactual
Pointwise label→KTO-style learningdesirable / undesirable
Process evidence→PRM / value learningsegment or action credit
Task + env + verifier→PPO / GRPO / RLOOonline RL under RLVR
Do not collapse the taxonomy
RLVR is a reward regime.
PPO, GRPO, and RLOO are optimizer families. Rejection sampling is a data operator.
Credit granularity
Trajectory → segment → state/action → token
Finer credit is useful only when its verifier or intervention supports the attribution.
06 Lifecycle & Frontiers
A useful data flywheel must not consume its own evaluation.
Immutable raw artifacts feed versioned training views. A frozen system crosses the evaluation firewall; only aggregate failure categories return to task construction.
Leakage-resistant lifecycle
Immutable bundles, derived learning views, and a one-way held-out boundary.
Measurement validity
How should a benchmark handle multiple valid solutions, contamination, and revision without losing comparability?
Construction & replay
Can synthetic tasks transfer across repositories, languages, services, and naturally occurring maintenance work?
Causal credit
Which actions changed the reachable outcome, and which detours should be preserved as recovery?
Harness transfer
Does a learned workflow survive new tools, context policies, action schemas, and verifier distributions?
Governance & evidence
What lineage, rights, cost, privacy, and reporting standard makes agent-generated data auditable?
07 Paper explorer
Browse the literature by its role in the data lifecycle.
This compact index emphasizes what each work produces or consumes. It is a curated snapshot, not a claim of exhaustiveness.
0 works shown
No works match this filter.
08 Citation
Cite this survey.
Versioned draft · evidence frozen 18 August 2026 · website updated 19 August 2026.
BibTeX
@article{lin2026codingagentdata,
title = {From Issues to Learning Signals:
A Survey of Data for Coding Agents},
author = {JLULLM LAB},
year = {2026},
note = {Survey manuscript},
url = {https://linjh1118.github.io/coding-agent-data/}
}