Monitoring Political AI: How to Study Systems That Keep Changing

Political AI extends beyond synthetic media to everyday information seeking
Repeated evaluation reveals changes that a single audit cannot capture
Scientific accountability requires transparent methods and preserved evidence

Political AI presents a measurement problem as well as a problem of trust. A fabricated campaign video can be identified as synthetic while still shaping attitudes through ridicule, repetition or emotional association. A chatbot response can look ordinary while selecting sources, framing a disagreement or omitting relevant uncertainty. These mechanisms require different forms of investigation. Published reporting describes AI-generated advertisements targeting Texas Senate candidate James Talarico in 2026, including a parody circulated by Citizens for Sanity. Such examples make synthetic media visible. Less visible is the growing use of conversational systems to obtain political information. Science Forum's question is how researchers can document what these systems do, distinguish errors from disagreements and detect changes over time. Public accountability needs a reliable record of behaviour before it can make credible claims about that behaviour's causes or consequences.

Synthetic Media and the Problem of Authenticity

Synthetic political material is not a single category. It includes satire, impersonation, misleading editing and content intended to be mistaken for an authentic recording. The Talarico advertisements examined here were designed as political spectacle; their possible influence need not depend on viewers believing that the depicted performance actually occurred. Other cases involve fabricated audio attributed to real public figures. Researchers should therefore specify the mechanism under examination rather than group every synthetic example under deception. Useful questions include whether viewers recognise the fabrication, whether recognition changes its persuasive effect and how repeated exposure influences memory. Advertising impressions establish distribution, but do not establish unique viewers, belief or electoral impact. The available platform records can provide a starting point for such work, with limits that should remain explicit.

The 2024 New Hampshire robocall imitating Joe Biden illustrates a different mechanism: impersonation used to deliver misleading voting instructions. The research discussed here describes a subsequent Federal Communications Commission penalty against Steve Kramer. For scientific analysis, the relevant distinction is between the authenticity of the speaker and the accuracy or effect of the message. Researchers may be able to establish that an audio recording is synthetic without knowing how many recipients followed its instruction. Conversely, a message can be harmful without a convincing impersonation. Investigating these stages separately avoids treating the existence of a fake as sufficient evidence of every claimed consequence. It also helps institutions decide whether a response should emphasise authentication, accurate information, distribution safeguards or public understanding of synthetic media.

Reporting also describes a 2025 attempt to contact officials using a cloned voice attributed to Marco Rubio. This broadens the problem from public persuasion to identity and information security. A voice that sounds plausible may be used to solicit access, establish a relationship or bypass an ordinary check. The scientific questions concern how people authenticate communication under pressure and which organisational procedures reduce mistakes. Technical detection may help, but an independent confirmation through a trusted channel addresses a different part of the vulnerability. These cases illustrate why research should describe the complete interaction between content, recipient and institutional setting. The same production tool can support very different activities, and its social effect depends on more than the realism of the audio or image.

Chatbots as an Information Environment

Chatbot-mediated information introduces a wider set of questions. Relevant evidence includes 2026 surveys in which substantial shares of Americans reported using AI for news or political understanding, while another poll asked about likely future use for candidate information. These measures should not be combined as if they described the same behaviour. Reported past use, frequency and intention have different meanings. Nevertheless, they motivate investigation of conversational systems as an information environment. A person asking about a candidate may receive a concise answer rather than a list of sources to inspect. Research should examine what information is selected, how uncertainty is communicated and whether users follow the supporting references. It should also consider how answers compare with the alternatives people would otherwise use, rather than assume that every chatbot response replaces high-quality journalism.

Persuasion and accuracy need separate measurement. The studies examined here find that language-model messages can persuade readers under experimental conditions, and that some interventions increasing persuasion may reduce factual accuracy. Such findings do not establish a universal relationship across models or political contexts. They show why a fluent, measured tone is an inadequate indicator of reliability. A study should specify its participants, topics, comparison messages and outcome measures. Changes in expressed agreement immediately after exposure are different from durable changes in belief or voting behaviour. Scientific evaluation can investigate each, but should resist converting a bounded experimental result into a broad claim about elections. A politically consequential system may be accurate in many responses while still producing influential errors in a smaller set of cases.

Positive findings also require boundaries. The Brennan Center's testing examined here found that several chatbots challenged the election conspiracy claims examined, including under repeated prompting. That is useful evidence about the tested systems and prompts at that time. It does not establish how they answer every related question, respond in other languages or behave after an update. Repetition across versions can show whether the result is stable. Researchers should preserve successful responses as carefully as failures, since a monitoring project designed only to collect alarming examples would provide a distorted picture. The aim is to estimate patterns of performance and uncertainty, including where safeguards work, rather than treat evaluation as a search for the most shareable mistake.

Why Evaluation Must Follow Change

A Carnegie analysis describes conversational systems as Dynamic, Ephemeral, Embedded and Personalised, using the acronym DEEP. Each feature changes the requirements of evaluation. Outputs can vary with model updates, system instructions, retrieved material and conversation history. Responses may disappear unless deliberately preserved. The same underlying model can behave differently through different interfaces or surrounding safeguards. Personalisation can make two users' experiences diverge. A named model is therefore an incomplete description of the treatment being studied. Researchers need to record the interface, date, available version information, prompt sequence and relevant settings. Some hidden conditions will remain inaccessible, but documenting the observable environment makes comparisons more meaningful. A one-time audit offers a snapshot of a changing system, not a permanent certificate of its behaviour.

The NewsGuard and BBC audits examined here illustrate the importance of test design. One examined responses to false news claims; another asked systems questions grounded in BBC reporting. Their reported error rates cannot be transferred directly to all news queries or voting questions. They depend on the selected material, scoring rules and systems tested. The scientific value lies partly in making those conditions inspectable so that others can reproduce or extend the work. Researchers should distinguish incorrect facts, misleading summaries, absent context and poor attribution. These are related but different failures. Repeating a stable test set can reveal change, while adding new questions can assess emerging issues. Both are useful, provided shifts in the test set are not mistaken for shifts in the model.

The AI Watchman project examined here repeatedly tested politically sensitive prompts and observed changes in refusal behaviour. Reported changes on Israel-related and abortion-related questions show the kind of pattern longitudinal evaluation can make visible. A temporal association with an event does not establish deliberate political intervention by a provider. Other explanations include software updates, retrieval changes, policy enforcement or sampling variation. Researchers should investigate those possibilities before attributing motive. Multiple runs, comparison topics and records of known product changes can strengthen the analysis. The important contribution is the preserved sequence of observations. It permits questions that a single isolated response cannot answer, including whether a change persists, affects several interfaces or appears only under a particular wording.

Different Standards Produce Different Scores

Providers' own evaluations are another useful but incomplete source of evidence. Public disclosures include internal impartiality and bias assessments released by Anthropic and OpenAI. Scores produced under different frameworks should not be read as a common ranking. They may assess response tone, factual balance, refusal patterns or compliance with company-defined behaviour. An evaluation can be carefully executed while leaving important questions outside its scope. Scientific scrutiny should examine the definition, sampling, annotation process and uncertainty behind a score. Public disclosure helps external researchers understand the intended standard, but independent testing remains valuable. It can investigate whether the stated behaviour appears in ordinary use and whether performance differs across issues, languages or user contexts not represented in the provider's examples.

Different standards also reflect different objectives. A system evaluated for willingness to discuss controversial subjects may score well while performing less well on accuracy or harmful-content handling. A system instructed to present competing positions may appear balanced while giving unsupported claims unwarranted equivalence. Refusal can protect against one risk and obstruct access to legitimate information in another setting. These tensions cannot be resolved by calling every product choice neutral or every difference biased. Researchers should state which property they measure and why it matters. A public comparison may need several indicators rather than one overall score. This allows users and institutions to see the trade-offs instead of assuming that the system highest on one dimension is the most reliable for every political question.

Synthetic impersonation and ordinary chatbot answers therefore require different monitoring methods. Authenticity checks examine whether a recording represents a real person or event. Answer audits examine factual support, source selection, framing, omissions and the handling of uncertainty. Neither replaces studies of audience response. A system can supply accurate information that users misunderstand, or unreliable information that users reject. Researchers should connect these stages when studying public effects, while keeping the evidence for each distinct. An answer that changes over time is an observation worth explaining, but not automatically a finding of manipulation. Transparent categories and careful interpretation are especially important in a political environment where evaluation itself can become a contested claim.

Building a Public Research Record

A sustained research programme could combine a stable panel of questions with samples of new issues. Universities, independent researchers and funders could maintain repeated observations, publish protocols and preserve responses in an accessible archive. The design should specify how prompts are selected, how often tests run and how disagreements between assessors are resolved. It should include quality checks and enough repeated samples to distinguish stochastic variation from more persistent changes. Costs also matter: access charges and restrictions can make longitudinal work difficult for smaller teams. Shared infrastructure may reduce duplication, but should not concentrate interpretive authority in a single organisation. The aim is a public evidence base that others can inspect, challenge and extend. Monitoring establishes what happened; explanations of intent or legal consequences require additional evidence and separate standards.

Political monitoring must also protect legitimate disagreement and user privacy. The purpose of longitudinal evaluation is to make behaviour observable, not to require a single approved answer to every contested question. Researchers can distinguish a verifiable factual error from a normative position and explain how each is scored. Public protocols should acknowledge where reasonable assessors may disagree. Collecting ordinary users' conversations introduces consent and confidentiality issues that synthetic test prompts can sometimes avoid. Where personalisation is important, a study needs an appropriate ethical design rather than unrestricted access to private histories. Independence from providers and political interests should be documented as part of the method. Credibility depends on the scrutiny the monitoring project accepts for its own choices.

Research evidence can inform governance without being identical to regulation. An observed pattern of errors may justify investigation, better documentation or changes to product design. It does not by itself establish a legal violation, and a regulatory requirement does not guarantee reliable performance. For scientific institutions, a practical contribution is to publish methods, preserve results and invite replication. Providers can assist by supplying clearer version information and explaining consequential changes. Public bodies can support access to evaluation resources and use evidence with appropriate attention to its limits. The central task is to build the capacity to learn how systems behave over time. A durable research record would help discussion move beyond competing anecdotes and incomparable internal scores, while leaving judgments about public policy open to informed debate.

Political AI should be studied as an evolving interaction between systems, people and institutions. Synthetic media highlights the authenticity problem; conversational answers raise questions about reliability, framing and access to information. Repeated evaluation can reveal both improvements and failures, but only if its design separates genuine change from differences in prompts, interfaces and scoring. The contribution of science is a disciplined record: what was asked, what was returned, under which observable conditions and with what uncertainty. Such a record will not resolve every disagreement about political neutrality. It can make those disagreements better informed and expose claims that exceed the evidence. For Gordon Institute's Science Forum, that is the foundation of accountability: independent inquiry that remains inspectable as the technology changes.

This Science Forum article is adapted from material published within Gordon Institute’s network. It presents editorial analysis and does not constitute professional advice or an official institutional policy.