Measuring AI at Work: From Recorded Use to Reliable Outcomes

Worker surveys and chat logs capture different aspects of AI adoption
Corrections, verification and abandoned outputs remain poorly measured
Research should connect AI use to completed work and retained human capability

How should researchers measure a technology that can be used extensively without producing a useful result every time? Generative AI makes this question increasingly important. The Real-Time Population Survey analysis, covering nearly 14,000 US workers aged 18 to 64 through May 2026, reports that more than 80 percent of occupations have adoption rates of at least 20 percent. This indicates broad reach across the labour market. It does not establish how much completed work the tools improve. A conversation may produce a useful draft, a misleading answer that requires correction, or an output discarded after several attempts. These experiences have different consequences for productivity and understanding. Science needs measures that connect interaction with a system to what eventually happens to the work, while recognising the valuable information that existing adoption datasets already provide.

Broad Adoption Does Not Establish Broad Benefit

The survey links workers' occupations to tasks and asks which of their ten most important tasks AI regularly helps them perform. More than 40 percent of individual tasks have adoption rates above 20 percent, but fewer than 3 percent exceed 50 percent and none reach 70 percent. Broad occupational adoption therefore coexists with uneven use within particular tasks. This may reflect a technology still diffusing, differences in job requirements, organisational restrictions or workers' assessments of its usefulness. Disappointment and abandonment are also possible explanations, but these adoption figures do not identify their frequency. Treating shallow use as proof of either imminent expansion or widespread failure would go beyond the evidence. The scientific question is which mechanisms account for the observed pattern and what additional information could distinguish them.

The study also compares observed use with exposure scores estimating what models might perform in principle. Such scores explain up to half the variation in adoption across occupations and tasks, but less than 10 percent across individual workers. The distinction matters: people doing apparently similar work may have different data access, organisational permissions, experience or quality requirements. They may also judge the same output differently. A theoretical capability belongs to a model under specified conditions; a useful intervention belongs to a person working within a particular environment. Research that links these levels can explain where adoption succeeds and where it stalls. It should avoid treating task descriptions as complete accounts of the circumstances in which work is performed.

What Chat Logs Can and Cannot Observe

Conversation logs offer a much larger and more detailed view of interaction than a worker survey can provide. Providers classify exchanges into activities drawn from the US Department of Labor's O*NET system, allowing researchers to examine what users appear to request. However, the text of a request does not necessarily reveal the user's occupation or purpose. A professor seeking help with a dataset may resemble a data analyst in the classifier's categories. Across 332 intermediate work activities, the original study reports correlations of 0.34 between survey measures and Anthropic's conversation measures, 0.10 for Microsoft and 0.11 for OpenAI. The differences do not make logs useless; they show that the sources observe different objects. Logs describe recorded interactions among users of a platform, while surveys can relate use to workers, occupations and people who do not adopt the technology. The denominator is especially important. A platform can describe the distribution of requests it receives, but that distribution does not reveal what proportion of all workers performing an activity use the platform. Frequent prompting by a small group may dominate the records. Survey weights and information about non-users address a different part of the measurement problem, which is why one source cannot simply replace the other.

Concentration patterns reinforce this distinction. In conversation data, the largest activity accounts for 15 to 23 percent of recorded use and the top ten for 46 to 61 percent. The survey's corresponding figures are 4 percent and 22 percent. Editing is prominent in conversations, although O*NET may describe the worker's broader responsibility as preparing reports or managing operations. A short exchange about rewriting a paragraph does not reveal the decision the paragraph supports. Conversely, directing an organisation can involve many small interactions that never appear under a single management category. For research design, this creates a problem of interpretation rather than simply one of sample size. More records cannot resolve missing information about the purpose, context and outcome of the activity being classified.

The Missing Cost of Correction and Abandonment

An additional gap concerns the fate of the output. A log can record a request, a response and perhaps subsequent revisions, but usually cannot establish whether the final answer entered a deliverable or was abandoned. A survey asking whether AI regularly helps with a task captures the worker's assessment, yet may still leave individual unsuccessful attempts unrecorded. A user can receive meaningful help overall while spending substantial time on some failures. Adoption and net benefit are therefore separate dimensions. To connect them, research would need information about outputs accepted as delivered, corrected, partly reused or rejected. These categories also require care: a discarded suggestion can stimulate a better idea, while a copied answer may be wrong. Outcome measurement must consider quality as well as acceptance.

Correction and verification consume resources that interaction counts cannot directly describe. A worker may check a generated citation, compare an answer against a source, or repeatedly explain a requirement the system has misunderstood. Some of this effort is worthwhile and would also occur when reviewing human work. Some may exceed the effort saved by generation. The source article uses such experiences to motivate a measurement gap, rather than presenting a measured estimate of wasted time. A stronger study could record total task duration, review effort, final quality and the circumstances of abandonment. It would also compare assisted and unassisted workflows fairly. A long conversation is not necessarily a failure, and a short one is not necessarily productive. The relevant comparison is the complete process required to reach a reliable result. A comparison could follow a defined task from its initial brief through delivery, with quality assessed independently of how it was produced. Researchers would need to record which review steps are included and avoid comparing a checked human result with an unchecked AI draft. They would also need to distinguish immediate assistance from learning effects: a process that saves time today may develop or weaken capabilities needed for the next task.

Measuring the Work That Survives

The problem matters when adoption indicators inform public research and employment projections. The original analysis notes that the US Bureau of Labor Statistics uses conversation-based measures among its indicators of AI exposure, while acknowledging that they do not directly observe occupational use. It also describes the proposed Workforce Transparency Act as a route towards collecting task-level usage data. Whatever the eventual collection framework, definitions will shape the conclusions it permits. Asking whether AI was used establishes participation; asking what became of the output begins to establish its contribution. Researchers should also distinguish an individual's time saving from effects on colleagues who review, correct or integrate the work. A task can appear faster at one stage while shifting effort elsewhere. Transparent measurement should make these boundaries visible rather than implying that every recorded interaction contributes equally to productivity.

Rapid model improvement does not remove the need for this evidence. It makes the date, version and setting of evaluation more important. A result observed with tools available through May 2026 cannot be assumed to describe every subsequent system. Equally, a newer benchmark score does not establish improved outcomes in a particular workplace. Worker surveys and logs should be treated as complementary sources, with each linked where possible to observed task outcomes. Organisations can begin with proportionate internal evaluation: record completion time, relevant errors, review effort and the final disposition of generated material. Sampling representative tasks may be more informative than monitoring every conversation. The aim is to learn when assistance helps, including the training and oversight that make it effective, without turning measurement into another burden that obscures the benefit being studied. Within research teams, outcome evaluation could examine whether generated code runs correctly, whether cited evidence exists and supports the claim, and whether an analysis can be reproduced by a colleague. These are different tests, and success on one does not imply success on the others. A useful measurement framework would retain those distinctions while making comparison over time possible.

Reliable evidence about AI at work requires a chain from capability to use, from use to a completed result, and from that result to its consequences. Existing surveys help describe diffusion; conversation data reveal patterns in requests. Neither alone resolves what happens after an answer appears. The missing information includes correction, abandonment, quality and the human capability retained through the process. A scientific approach would investigate these outcomes without assuming that frequent use proves success or that an occasional failure disproves value. For institutions investing in AI and for workers adapting their practice, the useful question is when a tool makes the complete task better. Measuring that contribution would give public discussion a firmer basis than counting the conversations along the way.

This Science Forum article is adapted from material published within Gordon Institute’s network. It presents editorial analysis and does not constitute professional advice or an official institutional policy.