I’m a innovation and individual entitlements researcher. For the former few years, one of my focuses has been the inquiry of how tongue shapes the ways we advantage from, or are harmed by, AI. I developed an open-source phase for language-pair analysis of LLM responses throughout distinct languages and contexts. I’ve additionally worked on evaluating policy-prompts guardrails and on whether giving LLM guardrails admission to tools can create them additional dependable and trustworthy (this activity was lately accepted to NeurIPS! Yay!!).
Recently, though, I was on a panel at RightsCon on the individual entitlements effect appraisal of agentic AI. It got me thinking additional concerning which aspects of tongue matter whenever evaluating LLM agents. I wanted to move beyond asking whether a example performs alternatively whenever I ask the identical inquiry in English versus Farsi (my native language), and appearance alternatively at the entire agentic trajectory (reasoning, planning, search, origin selection, origin hierarchy, relic creation) during taking tongue and environment into account.
So I decided to run a test.
The project was to inhabit in missing data in the World Bank Global Public Procurement Database, using authoritative data for the US and Iran. I ran it in The project English for the US and Farsi for Iran (image below), alongside three agents:
Meta’s Muse
Anthropic’s Claude Cowork, Opus 5.5 Medium
OpenAI’s GPT 6.1 Sol, Medium
Below is the exact immediate I used for all three:
I intentionally used the web versions of these services (not the app or terminal versions) to indicate what mundane users experience. The difference matters for monitoring and logging an agent’s actions, which I conversation below.
This article is small concerning which delegate performed improved or faster, and additional concerning how the agents behave alternatively about admission to information, tongue representation, contextual understanding, transparency, human-in-the-loop, and safeguards.
You can discover all the results in the following files:
Output excel records for Muse, GPT, and Claude (here)
Each agent’s self-generated activity trajectory following receiving the immediate (here)
Text records extracted from display recordings of the agents’ actions (here, and full record here)
Below, I summarize my observations.
For those of us operating in digital rights, human-in-the-loop (HITL) oversight of AI agents joins a longer row of debates concerning “informed consent,” from GDPR consent requirements to biscuit pop-ups and the regular ticking of Terms of Service and Privacy Policy boxes. With that in mind, I paid near notice to how all delegate engaged me during the experiment.
For accessing and fetching data from websites, GPT asked for approval lone once, at the extremely commencement of the task. It requested admission to websites and offered an “allow all applicable sites” option, which I selected. After that, it did not ask again.
Claude asked questions throughout the task, the two concerning accessing websites for investigation and concerning extracting matter from the World Bank site. Unlike GPT, it offered no “allow all applicable sites” option, so it requested approval all time: nine times for the US project (all .gov websites) and nine times for the Iran project (mainly .ir domains and additionally fa.wikipedia sources). I approved all request.
Claude additionally ran into a specialized bounds that triggered a distinct benevolent of HITL moment. For the two Iran and the US, it couldn’t burden the live GPPD nation profiles, since the portal builds its pages alongside JavaScript, and Claude’s sandbox network guideline blocked admission to the World Bank data file. Claude stopped and asked whether I wanted to upload the leaf (as a pdf) myself or let the activity continue without it. I skipped the question, so it continued and took its data from the World Bank’s GPPD DataBank API instead. However, the DataBank holds 2018 data, during the portal shows a 2022 profile. As a result, Claude’s baseline data was distinct from GPT and Muse.
Muse did not ask for any approval until the fourth part of the task, which required registering on the World Bank website and uploading information.
All three agents completed the project up to the item of creating the spreadsheets, and described their confidence in the results they generated.
The final part of the task, registering on the World Bank portal and uploading the changes, is anywhere things became additional engaging since it required the agents to obtain a additional important actions fairly than fair collect information.
Claude and GPT the two stopped at this item and handed the registration and uploading complete to me. Muse, however, kept going. Without asking me or showing me the Terms of Service, it registered an document under the email location [email protected]. You can see Muse’s complete rear and onward here.
The array below summarizes how all delegate approached this final part of the project and cybersceuirty implications concerning it.1
There is an ongoing conversation concerning whether AI labs should disclose a model’s complete sequence of idea (CoT) and act trace, and if so, how much. Labs have stated multiple reasons for holding back. OpenAI chose not to display o1’s raw CoT to users, citing person experience, rivalrous advantage, and the value of keeping the CoT accessible for inner monitoring. Anthropic noted that raw reasoning can merge incorrect or half-formed thoughts and that malicious actors could use it to build improved jailbreaks. There is additionally a gaming and reward hacking concern, and “CoT unfaithfulness”.
To comprehend an agent’s behavior, however, evaluators need to cognize whenever and why things happen, which is lone imaginable alongside a monitoring scheme in location and admission to the agent’s complete trajectory. For an evaluator exterior an AI lab, without that access, it is nearly unattainable to completely create awareness of an agent’s behavior. And if exterior evaluators can lone see partial trajectories, and any conclusions they diagram can be dismissed for lacking complete information, what is the value of autonomous evaluation?
Knowing these limitations, I tried my finest to collect, monitor, and inspect as much of all agent’s activity as I could, again putting myself in the stance of an average investigator tasked alongside updating the World Bank data portal.
Since there is no one-click way for ordinary users to extract a complete document of an agent’s activity trajectory, I watched all delegate activity live and recorded everything clickable and apparent on screen. Once the project was finished, I gave the recordings to ChatGPT to extract the content and create it searchable. To provision you a awareness of what this looks like, current is a snippet (left: Claude, middle: Muse, right: GPT, apologetic for the size and illegibility).
Self-reported trajectories. When the project was done, I prompted all delegate to create a content document describing what it did, including errors, how it handled them, workarounds it used, websites it searched, and more. Muse and GPT all produced a downloadable .txt file, during Claude declined, stating that it went against its safety policy, stating “reasoning_extraction.”
Claude refusing to create activity trajectory .txt file, citing concerns concerning “reasoning extraction” That said, self-reported trajectories can not be completely trusted; I have seen mismatches in the former between what agents really did and what they reported. So I gave these reports small weight, but if you’re curious in reviewing them and spotting matches or mismatches, the records are here.
Analysis. I afterward used the output Excel sheets and the content extracted from the display recordings to behavior the analysis, the two on my own and alongside assistance from Claude Code to sift through the data and create tables. I cross-checked all of the data myself.
Below is several data concerning how much data on agents activity trajectory is accessible in all LLM agent’s web UI.
A few observations and afterward I’ll get to my points:
All three agents answered in fluent Farsi, but the Iran results were far weaker than the US ones. Of Iran’s 138 N/A fields, GPT and Muse all filled only 21 alongside a genuine value; for the US, they filled 51 and 64 of 130.
For the US, 76–89% of all agent’s citations came from authoritative authorities sites and the remainder from lawful global institution websites. For Iran, it was 11–22%.
Low-authority sources crept in, including a Telegram channel, a Medium post, Grokipedia, or websites run by Iranian diaspora media groups specified as Iran International. Claude seemed to be additional traditional concerning finding workarounds whenever websites were unavailable and frequently preferred English tongue sources equal alongside low legitimacy.
Claude could lone open 3 out of the 16 Farsi pages it tried. GPT and Muse cited 11 Persian sources all but showed study lone 3 and 7 of them respectively.
Knowledge gaps got filled alongside item else:
Claude used headlines and its own recollection (sometimes contradicting alongside what it found), and stated so.
GPT used republished copies of the law.
Muse mostly peruse a 2009 English translation but cited the authoritative Persian page.
My item is not that I expected the Iran/Farsi tasks to have the identical outcomes as the US/English ones. After all, the Iranian authorities has made it extremely difficult for abroad IP addresses to admission authoritative websites and domains ending in .ir (you can read more concerning this in the environment of Iran’s National Information Network). My item is concerning the agents’ differing workarounds and origin prioritization.
For me, this brought to intellect the digital entitlements and tongue inclusion activity that the fine group of Global Voices have done for years, including on net neutrality and tongue access. What does all this average for an AI agents era? And from an AI sovereignty perspective? One of AI sovereignty’s promises has been tongue assortment and assistance for local languages. LLM output quality, and perchance safeguards, keep improving, but we additionally need to think concerning what tongue localization should appearance akin in agents reasoning, searching, and prioritizing sources.
Looking through the agents’ trajectories, I noticed that they differed not lone in which websites they could access, but additionally in how difficult they tried whenever admission failed. Some agents stopped following an first failure, others tried substitute routes, distinct browsers, search-result snippets, cached or secondary sources, distinct fetch methods, and more.
So I ran a small follow-up test. I selected websites that the agents had accessed inconsistently during the first project and gave all delegate a uncomplicated instruction: “Here is a catalog of websites. Look them up and compose a one-paragraph summary of each.” The item was not to measure the norm of the summaries, but to detect what all delegate did whenever straightforward admission failed.
In this case, the agents were trying to attain websites that were difficult to admission from their own specialized environments. But what happens whenever the admission obstacle arrive from the user’s surroundings instead?
For group in countries anywhere governments display or obstacle websites, could an LLM or AI delegate rotate into another tier of data access? Could it retrieve, summarize, obtain actions, or relay data from websites that the person cannot attain directly? And could agentic workarounds create censorship circumvention easier — or conversely reproduce new restrictions through a distinct specialized stack?
As Iranians who investigation data admission and net governance in Iran, my companion Farzaneh Badiei (a digital-rights lawyer) and I have been discussing how LLMs and AI agents power be used in censorship-circumvention contexts. I may examine this additional in forthcoming installments of the Humane AI newsletter.
If you are curious in designing or conducting experiments on this topic, awareness liberated to attain out at [email protected].
And, final but not least:
If you peruse Farsi, fine fortune making awareness of the results on agents’ UIs!
To my chap right-to-left readers and writers (~700 myriad people): you have my commiseration all period you execute the gymnastics of trying to compose an Instagram caption, inhabit in a spreadsheet, peruse governments’ “accessible” translated forms, or copy and paste content throughout platforms.
Disclaimer: I used ChatGPT and Claude for copyediting. I use Claude Code for array generation, and supervised data analysis.
