Your AI assistant is quietly assigning everyone a gender

If you ask an AI assistant to rewrite a text in a more work-appropriate tone, or summarize a paragraph, or make something sound funnier, you probably expect that the facts will stay the same facts.¹ The people in the text will stay the same people. And if someone is referred to with whatever pronouns, those pronouns will be retained in the output, too.

That assumption turns out to be wrong, and measuring exactly how wrong it is — and in what circumstances — is what motivated our new paper, ProText: A benchmark dataset for measuring (mis)gendering in long-form texts, co-authored with Margit Bowler, Patrick Sonnenberg, and Yu’an Yang.

The problem

There’s a lot of research on gender bias in language models.² Most of it, though, was designed for an earlier generation of AI systems—when text generation was a whole lot less fluent than it is in LLMs of the past 3 or so years. The classic benchmarks (WinoBias, WinoGender, and others) test whether a model can correctly resolve pronouns in sentences like “The physician hired the secretary because she was overwhelmed with patients.”³ That’s a useful thing to measure, and as it turns out, it’s still difficult for models and hence informative for model owners, but it’s not what most people actually do with AI assistants today.

What people more commonly do is ask models to transform text: rewrite this more casually, summarize this email, make this sound better. And in the process of transforming text, a model has to make a lot of choices. Does it keep the pronouns from the original? Does it add pronouns where there were none? Does it respect a they/them pronoun, or quietly swap it for he or she?

Most existing benchmarks also only look at he and she. We wanted to explicitly include gender-neutral pronouns and cases where no pronouns are used at all, because those are exactly the cases where we’d expect models to struggle most.

What we built

ProText is a dataset of 640 English texts,⁴ all written by humans, designed to probe these questions. Each text was constructed along three dimensions: the Theme (how the main protagonist is referred to — by name, occupation, title, or kinship term), the Theme Category (whether that noun is stereotypically male, female, or gender-neutral), and the Pronoun Category (he/him, she/her, they/them, or no pronouns at all).

We had 100 native English speakers from Ireland, India, and the United States write the texts.⁵ Authors were given their assigned combination of categories and otherwise left free to write whatever they wanted. We encouraged them to use features like slang, sarcasm, emojis, code-mixing, and informal grammar to make the texts feel natural. The average text is about 65 words — the length of a casual message or short note.

The policy for handling pronouns in text transformations is simple:⁶

In English texts, gender information can come in the form of gender-marked nouns (“stewardess”) or gendered pronouns (“she”); some nouns carry very strong gender implications even though they are not grammatically encoded (“nun”, “mother”). In other languages, an important additional cue is gender agreement (on nouns, verbs, and adjectives.)

What we found

We ran a mini case study on GPT-4o and Gemini 2.0 Flash (inferences generated in April 2025). Each model was given two rewriting prompts (journalistic style and humorous rewrite), applied to all 640 texts in ProText.⁷ A few things stood out.

When explicit pronouns were present, the models mostly did fine. Misgendering rates were below 5% when the input used gendered pronouns like he or she. This is the scenario that most existing benchmarks test, and the models handle it reasonably well. The mitigations that have been put in place appear to be working in this case.

When no gender cues were present, the models routinely invented them. When input texts used they/them pronouns or no pronouns at all, both models frequently introduced he or she in the output. This gendering followed stereotypes: texts about a nurse with no pronouns tended to get she; texts about a doctor tended to get he. For genuinely ambiguous nouns like “healthcare worker” or “artist”, models defaulted to masculine pronouns more often than feminine ones.

Here are a couple of illustrative examples. The first example, involving a stereotypically feminine occupation (“social worker”) with no explicit gender cues in the input text, the model introduced a feminine pronoun (“her”) as well as a gendered noun (“lady”) to refer to the text protagonist. Gendering happened along stereotypical lines.

In the next example, the model misgenders along stereotypical lines outside the quotation, replacing “they” in the input with “she” in the output. Notice, however, that the model was able to retain the gender-neutral pronoun inside the quote unchanged:

Titles and kinship terms caused the most problems. Texts with words like Mrs., Mr., husband, or wife had higher misgendering rates than texts with names or occupations. This is notable because occupations are extremely well-represented in existing gender bias benchmarks — which may be exactly why models have better mitigation there. Arguably, these words might also be more strongly associated with particular genders than occupations, with names falling somewhere inbetween. The mitigations are narrow, and they don’t generalize.

We also found a handful of patterns that weren’t directly part of the study design but showed up anyway.

• Models gendered the authors of texts based on stereotypes: for example, a narrator who mentioned going hunting or visiting a barber would come back from the transformation as “he.”
• Models made heteronormative assumptions about couples: for example, if a text mentioned a husband, the model sometimes inferred the author was female. And
• Models pluralized singular they, treating “they went to the store” as referring to a group rather than a single person, even if the context in the input suggesting a single person.

The overall picture is that the models appear to have some mitigation strategies for gender bias in pronoun resolution tasks, but those strategies are too narrow. They work in the scenarios that are most common in training data and existing benchmarks—simple pronoun resolution with explicit gender cues, especially for occupations. They break down almost everywhere else.

The models have become less likely to misgender. But they are much more likely to gender, to quietly assign a gender to someone who never had one.

What this means

Text transformation is one of the most common things people use AI assistants for. Summarize this email. Rewrite this for a different audience. Make this sound less technical. These are everyday tasks, and every one of them is an opportunity for a model to change who a person is in the text, to assign them a gender they don’t have, or to override the one they do. The rates here are not small. This is a systematic, predictable, replicable pattern.

I want to be clear that this isn’t about malicious intent. The model doesn’t “know” it’s misgendering anyone. But it doesn’t need to know. It just needs to be trained on biased data and then deployed at scale, and the harm takes care of itself. If biased assumptions are found in one part of the model’s behavior, it stands to reason that they also occur elsewhere — and have real world consequences — in places that might be harder to detect, but are just as real.

A note on our assumptions about gender

As always when I do this kind of research: we are making simplifying assumptions about gender that are inaccurate. We included they/them pronouns, which is further than most prior work, but we still didn’t include neopronouns like ze/hir or fae/faer, partly because none of our text authors were neopronoun users, and we didn’t want to ask people to write with pronouns they weren’t familiar with. We discuss this in the paper and explain why we think the dataset will extend naturally to neopronoun categories once model behavior with they/them has improved (which, based on these results, still has a long way to go).

ProText is available on ArXiv, and the full dataset is available on GitHub. Cite as Kotek, Bowler, Sonnenberg, and Yang (2026). ProText: A benchmark dataset for measuring (mis)gendering in long-form texts.

Notes