K:Evaluation methodology and baselines
Evaluation methodology and baselines
Editorial topic mappings connect related questions across sources and book chapters.
SE@PoliMi bibliography 82
- W001 — AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
Agarwal · 2026 - W003 — The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers
Cui · 2026 - W004 — Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects
He · 2026 - W005 — Early Adoption of Agentic Coding Tools by GitHub Projects
Raida · 2026 - W008 — How Much Does AI Impact Development Speed? An Enterprise-Based Randomized Controlled Trial
Paradis · 2025 - W010 — Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools
Demirer · 2026 - W013 — Quantifying the Expectation-Realisation Gap for Agentic AI Systems
Lobentanzer · 2026 - W014 — Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
Murphy-Hill · 2026 - W016 — The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study
Vella · 2026 - W017 — Randomized Controlled Trials for Conditional Access Optimization Agent
Bono · 2025 - W018 — Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
Kumar · 2025 - W020 — Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity
Becker · 2026 - W022 — AI Coding Agents: Adoption Trends
JetBrains Research · 2026 - W023 — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Becker · 2025 - W025 — Estimating AI productivity gains from Claude conversations
Tamkin · 2025 - W027 — Beyond the Copilot: Scaling the Agentic Product Development Life Cycle
Relyea · 2026 - W030 — Workspace Topology as an Attack Vector in Agentic Coding Assistants
Day · 2026 - W032 — Understanding the Impact of AI Code Assistants on Security API Usage
Mousavi · 2026 - W033 — How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
Ogenrwot · 2026 - W034 — Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance
Pinna · 2026 - W036 — On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe · 2026 - W038 — Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation
Yan · 2026 - W040 — Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu · 2025 - W041 — Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
Schreiber · 2025 - W044 — An Empirical Study of Code Clones from Commercial AI Code Generators
Wu · 2025 - W047 — Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval · 2023 - W049 — The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
Churilov · 2026 - W051 — From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software
Fang · 2026 - W052 — What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code
Improta · 2026 - W054 — Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
Kraishan · 2026 - W055 — Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild
Liu · 2026 - W056 — A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories
Mao · 2026 - W057 — How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests
Mazloomzadeh · 2026 - W058 — Decoupling Code Complexity from Newcomer Participation: A Causal Study of AI Coding Agent Adoption in OSS
Xu · 2026 - W066 — Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
Gao · 2026 - W072 — AI-to-AI Code Reviews of GitHub Pull Requests
Selvanayagam · 2026 - W073 — The Effect of Complexity and Provenance on Code Review Decisions: Evidence from a Controlled Experiment
Singh · 2026 - W077 — Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
Zhao · 2026 - W078 — Beyond the Surface: Measuring Self-Preference in LLM Judgments
Chen · 2025 - W080 — Unit Testing Past vs. Present: Examining LLMs' Impact on Defect Detection and Efficiency
Ramler · 2025 - W081 — Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?
Tufano · 2025 - W090 — Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise
Floyd · 2017 - W097 — Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions
Kaufman · 2026 - W102 — Cross-Model LLM Code Review: Should You Use Claude to Review Codex or Vice Versa?
Xiang · 2026 - W103 — An Eye for Trust: An Exploration of Developers' Trust Perceptions Through Urgency and Reputation
Yabesi · 2026 - W104 — Coding with Enemy: Can Human Developers Detect AI Agent Sabotage?
Ye · 2026 - W107 — Human-AI Synergy in Agentic Code Review
Zhong · 2026 - W118 — How AI Impacts Skill Formation
Shen · 2026 - W120 — Automation, AI, and the Intergenerational Transmission of Knowledge
Ide · 2026 - W121 — Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task
Kosmyna · 2025 - W122 — AI Didn't Make Programming Easier. It Just Made It Differently Difficult
Osborn · 2026 - W123 — Firms' GitHub Copilot Adoption and Labor Market Outcomes for Software Engineers
Baird · 2026 - W125 — Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence
Brynjolfsson · 2026 - W130 — A New Look at AI's Impact on Jobs: Firm-Level AI Spending and Workforce Adjustment
Kharazian · 2026 - W135 — Generative AI as Seniority-Biased Technological Change: Evidence from U.S. Résumé and Job Posting Data
Hosseini · 2025 - W137 — The Labor Market Is Tilting Toward Seniority
Aidala · 2026 - W142 — AI Is Probably Not (Yet) the Reason for Labor Market Weakening
Nunn · 2026 - W143 — New data show no AI jobs apocalypse — for now
Kinder · 2025 - W145 — ROI of AI-Assisted Software Development
DORA · 2026 - W146 — McKinsey Technology Trends Outlook 2026
Chui · 2026 - W149 — Making AI Visible, Not Vanished: How AI Policies Reshape Developer Experience on GitHub
Chen · 2026 - W150 — We Permit the Use of AI, but [...]: The Landscape of AI Policies in Popular Open Source Projects
Hora · 2026 - W153 — Toward Agentic Software Engineering Beyond Code: Framing Vision, Values, and Vocabulary
Hoda · 2026 - W154 — Comprehension Debt in GenAI-Assisted Software Engineering Projects
Ahmad · 2026 - W161 — Examining ownership models in software teams: A systematic literature review and a replication study
Koana · 2024 - W163 — A Case Study of Onboarding in Software Teams: Tasks and Strategies
Ju · 2021 - W165 — Please Turn Your Cameras On: Remote Onboarding of Software Developers during a Pandemic
Rodeghero · 2021 - W169 — Simultaneous Measurement of Program Comprehension with fMRI and Eye Tracking: A Case Study
Peitek · 2018 - W170 — Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia · 2018 - W171 — Revisiting Turnover-Induced Knowledge Loss in Software Projects
Nassif · 2017 - W172 — Measuring neural efficiency of program comprehension
Siegmund · 2017 - W175 — Code Ownership and Software Quality: A Replication Study
Greiler · 2015 - W176 — I Know What You Did Last Summer — An Investigation of How Developers Spend Their Time
Minelli · 2015 - W178 — Ramp-Up Journey of New Hires: Tug of War of Aids and Impediments
Rastogi · 2015 - W180 — Using psycho-physiological measures to assess task difficulty in software development
Fritz · 2014 - W184 — Developer Fluency: Achieving True Mastery in Software Projects
Zhou · 2010 - W185 — Succession: Measuring Transfer of Code and Developer Productivity
Mockus · 2009 - W186 — Maintaining Mental Models: A Study of Developer Work Habits
LaToza · 2006 - W187 — Globalization by Chunking: A Quantitative Approach
Mockus · 2001 - W188 — The Ramp-Up Problem in Software Projects: A Case Study of How Software Immigrants Naturalize
Sim · 1998 - W192 — AIDev: Studying AI Coding Agents on GitHub
Li · 2026 - W193 — MirrorCode: Evidence that AI can already do some weeks-long coding tasks
METR / Epoch AI · 2026
Addy Osmani 2
- O029 — The reality of AI-Assisted software engineering productivity
Addy Osmani · 2025-08-16 - O051 — Agentic Verification
Addy Osmani · 2026-10-14
The Pragmatic Engineer 20
- P006 — How to debug large, distributed systems: Antithesis
Gergely Orosz; Elin Nilsson · 2024-11-12 - P011 — How GenAI is reshaping tech hiring
Gergely Orosz; Elin Nilsson · 2024-12-03 - P020 — Taking on Google search: Perplexity and Kagi
Gergely Orosz · 2025-01-28 - P029 — AI fakers exposed in tech dev recruitment: postmortem
Gergely Orosz · 2025-03-11 - P030 — Survey: What’s in your tech stack?
Gergely Orosz; Elin Nilsson · 2025-03-18 - P031 — What is Growth Engineering?
Gergely Orosz; Alexey Komissarouk · 2025-03-20 - P032 — AI Engineering in the real world
Gergely Orosz · 2025-03-25 - P033 — The Reality of Tech Interviews in 2025
Gergely Orosz; Evan King; Stefan Mai · 2025-04-01 - P035 — Tech hiring: is this an inflection point?
Gergely Orosz · 2025-04-15 - P039 — Building, launching, and scaling ChatGPT Images
Gergely Orosz · 2025-05-13 - P041 — The AI Engineering Stack
Gergely Orosz; Chip Huyen · 2025-05-20 - P052 — Cursor makes developers less effective?
Gergely Orosz · 2025-07-24 - P058 — How experienced engineers get unstuck in coding interviews
Gergely Orosz; Mike Mroczka · 2025-08-26 - P061 — How tech companies measure the impact of AI on software development
Gergely Orosz; Laura Tacho · 2025-09-16 - P069 — Comparing interviews at 8 large tech companies
Gergely Orosz · 2025-10-30 - P075 — A pragmatic guide to LLM evals for devs
Gergely Orosz; Hamel Husain · 2025-12-02 - P085 — Deepdive: How 10 tech companies choose the next generation of dev tools
Gergely Orosz · 2026-02-03 - P094 — What is inference engineering? Deepdive
Gergely Orosz (introduction); Philip Kiely (guest excerpt) · 2026-03-31 - P096 — Learnings from conducting ~1,000 interviews at Amazon
Gergely Orosz; Steve Huynh · 2026-04-21 - P111 — Pushing software engineering limits with “napkin math”
Gergely Orosz · 2026-07-21
Martin Fowler’s website 5
- F012 — What role does LLM reasoning play for software tasks?
Birgitta Böckeler · 2025-02-18 - F020 — Partner with the AI, throw away the code
Matteo Vaccari · 2025-07-31 - F028 — Viability of local models for coding
Birgitta Böckeler · 2026-07-07 - F029 — Experiences with local models for coding
Birgitta Böckeler · 2026-07-08 - F031 — TDD inside the agent loop - theater or actual value?
Birgitta Böckeler · 2026-08-10
Earlier Non-Deterministic books 46
- B001 — Introduzioni alle parti — Non-Deterministic Software Engineering
Enrico Papalini · - B003 — Summary
Enrico Papalini · - B009 — Measuring What Matters---The Data Behind AI's Real Impact
Enrico Papalini · - B010 — The Adaptation Imperative---What Companies Must Do Now
Enrico Papalini · - B013 — PART I: Core Practices
Enrico Papalini · - B016 — PART IV: Organizational Foundation
Enrico Papalini · - B017 — Pattern Recap: Quick Reference Guide
Enrico Papalini · - B018 — Antipatterns: What Not to Do
Enrico Papalini · - B019 — Tools \& Practices
Enrico Papalini · - B020 — Understanding Tool Maturity Levels
Enrico Papalini · - B021 — Getting Started: Choosing Your Primary Tool
Enrico Papalini · - B026 — Development Practices Enhanced by AI
Enrico Papalini · - B029 — Measurement \& Observability
Enrico Papalini · - B030 — Common Pitfalls to Avoid
Enrico Papalini · - B031 — First 90-Days Implementation Roadmap
Enrico Papalini · - B032 — Quick Reference Guides
Enrico Papalini · - B033 — Expert Insights
Enrico Papalini · - B034 — Implementation Guide and Solutions
Enrico Papalini · - B037 — Preface
Enrico Papalini · - B039 — The Process
Enrico Papalini · - B043 — Limitations and Judgment
Enrico Papalini · - B045 — Complete Implementation Examples
Enrico Papalini · - B046 — Playbook Quick Reference
Enrico Papalini · - B048 — Preface
Enrico Papalini · - B050 — The Process: Four Phases, Four Gates
Enrico Papalini · - B052 — The Ecosystem: A Map of Public Frameworks
Enrico Papalini · - B057 — The Comparison: Patterns, Divergences, and Convergences
Enrico Papalini · - B058 — Design Boundaries: What to Adopt, What to Avoid
Enrico Papalini · - B060 — The Agent Pipeline and Gates
Enrico Papalini · - B064 — Limitations and Professional Judgment
Enrico Papalini · - B066 — Installation and Setup
Enrico Papalini · - B069 — Framework Reference Cards
Enrico Papalini · - B076 — Spec Loop Engine
Enrico Papalini · - B084 — Preface
Enrico Papalini · - B086 — The Philosophy of Deterministic Failure
Enrico Papalini · - B088 — The Ralph Loop Pattern
Enrico Papalini · - B089 — Task Decomposition for Machines
Enrico Papalini · - B091 — Quality Gates and Backpressure
Enrico Papalini · - B097 — Complete Implementation Guide
Enrico Papalini · - B098 — Tool Reference
Enrico Papalini · - B099 — Troubleshooting and Debugging
Enrico Papalini · - B100 — Ralph Loops with VS Code and GitHub Copilot
Enrico Papalini · - B106 — The Economics of Loop Engineering
Enrico Papalini · - B108 — Building a Full Local AI Loop
Enrico Papalini · - B109 — Choosing the Right Loop
Enrico Papalini · - B111 — Glossary
Enrico Papalini ·