OpenAI has banned a set of accounts that were using its models to run romance scams. The report describes what the models were doing for the operators: opening messages, translation, working the victims emotionally, and steering them towards bait for fraudulent investment schemes.
The company sorts the pattern into three stages, and the tidiness of the framework is the first thing worth noticing. The hook, meaning cold contact. Engagement, meaning the production of strong feeling. And the payout, meaning the part where money leaves the victim’s account.
Read that list again with a straight face and tell me it is not a sales funnel. Lead generation, nurturing, conversion. It is the same diagram that appears on a whiteboard in any commercial department on earth, with the same three boxes and the same arrow. The only material difference is that the product on offer is a soulmate and the conversion event is a lonely person’s savings.
The grammar that used to save us
The most uncomfortable line in the whole document is not about capability. It is a throwaway observation: the operators used the model to generate messages that were potentially more engaging and less recognisable as written by non native speakers than the messages of traditional scams.
Sit with that for a second, because it describes the erosion of something a lot of people were quietly relying on.
For years the amateur defence against this kind of fraud was partly linguistic. The wrong article. The verb in the wrong place. The sentence that had obviously come out of a machine and been pasted in without being read. It was never the only warning sign, and plenty of fraudsters have always written perfectly well, starting with the ones working in their victims’ own language. But where the tell was there, it cost nothing. No training course, no awareness campaign, no budget line. Your grandmother did not need a security qualification to notice that her new Dutch surgeon boyfriend wrote Italian like a washing machine manual.
That particular clue is now cheaper to remove, and nothing free is arriving in its place. The measures usually proposed instead, awareness programmes and literacy initiatives, ask the target to become a linguist and a threat analyst in her spare time. The grammatical tell asked nothing of her at all. That is what made it useful, and that is the part being worn away.
Targeting written by the industry itself
Then there is the section that writes its own satire.
The pig butchering scam described in the report often went after American men in their forties working in medicine, by replying to their posts about golf on social media. Another romance operation bought social media advertising to reach young men in Indonesia. A different family of scams impersonates a celebrity and works the fan groups.
Strip the criminal context and that is a targeting brief. An age band, an occupation, a country, a declared hobby. The report documents the posts about golf on social media; it does not say how the rest of that profile was assembled, and I am not going to invent a method for it. What it does show is that the selection criteria are the ordinary currency of advertising, and that the advertising was bought the way all advertising is bought. It is the same segmentation logic an agency would use for a client selling watches, running on the same platforms with the same tools. The victims were not hunted down one by one. They were reached, through the pipes that reach everyone else.
Nobody had to invent an instrument for this. The instrument was already there, built and sold as marketing.
The sentence at the bottom of the report
The most useful line in the document arrives near the end, and it is easy to skim past because it is phrased as methodology.
An essential element of these scams, the report says, is the distribution network. The operations it disrupted pushed their hook messages through SMS, encrypted messaging apps, social media posts, online advertising, or some combination of those. Pre AI scams used email, telephone calls, and in the nineteenth century, letters and telegrams. The company is careful about what comes next, and the care deserves to be preserved rather than flattened: it says the fragmentary nature of the evidence makes reliable comparison between the scams it has disrupted difficult, and it presents the conclusion as an assessment. Its assessment is that the chosen distribution method, indiscriminate or targeted, significantly affects a scam’s ability to reach and exploit its victims, regardless of how much the operation used AI for its various functions.
Regardless of how much the operation used AI.
That is a judgement, not a measurement, and the report says so. It is still the most interesting thing in it, and it sits underneath a wall of absorbing process description about what the model was asked to write. On this reading, the variable that governs whether a scam reaches people is not the quality of the prose. It is the pipe. Letters, telegrams, email, cold calls, SMS, paid social advertising: the criminal business has always been a distribution business with a text generation problem attached, and the text generation problem is the half that has just become easier to solve.
What can and cannot be banned
Which sets up an asymmetry that a case study of this kind is not built to close.
The model provider can ban an account, and it did. That is a real action with a real effect, and publishing a detailed case study of your own product being abused is more than much of this industry volunteers about itself.
But an account is one of the more replaceable parts of the operation. Nothing in the report claims the operators were stopped; it says these accounts were. The same workflow can be attempted with another provider, with an open weights model on rented hardware, or somewhere that publishes no case studies at all. Meanwhile the distribution layer, the one the report’s own assessment points to as decisive, is not an abuse channel that anyone needs to sneak through. It is a functioning commercial market. The advertising that carried the bait to young men in Indonesia was bought, and the platforms that sold it do have rules against fraudulent advertising. What they do not have is much incentive to slow down a business whose economics reward volume sold fast, reviewed afterwards and refunded at worst.
So that layer does not get switched off in a press release. Regulators, researchers and consumer bodies have been arguing for years about how hard it should be to buy attention and who should have to prove they deserve it; you can gauge the progress of that argument by how much of your own feed is still for sale to whoever turns up with a card. Nothing in this case study is going to settle it.
So the account goes. The funnel stays. The person at the other end of it, who used to get one free clue and now gets a fainter one, is told to be more careful.
Raffaele Di Marzio
All my “insane” books on cybersecurity and governance are here 👇 https://cyberium.limited/bookshelf.html