net.wars

Performing intelligence

“Oh, great,” I thought when news broke of the release of GPT-4. “Higher-quality deception.”

Most of the Internet disagreed; having gone mad only a few weeks ago over ChatGPT, everyone’s now agog over this latest model. It passed all these tests!

One exception was the journalist Paris Marx, who commented on Twitter: “It’s so funny to me that the AI people think it’s impressive when their programs pass a test after being trained on all the answers.”

Agreed. It’s also so funny to me that they call that “AI” and don’t like it when researchers like computational linguist Emily Bender call it a “stochastic parrot”. At Marx’s Tech Won’t Save Us podcast, Goldsmith professor Dan McQuillan, author of Resisting AI: An Anti-fascist Approach to Artificial Intelligence, calls it a “bullshit engine” whose developers’ sole goal is plausibility – plausibility that, as Bender has said, allows us imaginative humans to think we detect a mind behind it, and the result is to risk devaluing humans.

Let’s walk back to an earlier type of system that has been widely deployed: benefits scoring systems. A couple of weeks ago, Lighthouse Reports and Wired magazine teamed up on an investigation of these systems, calling them “suspicion machines”.

Their work focuses on the welfare benefits system in use in Rotterdam between 2017 and 2021, which uses 315 variables to risk-score benefits recipients according to the likelihood that their claims are fraudulent. In detailed, worked case analyses, they find systemic discrimination: you lose points for being female, for being female and having children (males aren’t asked about children), for being non-white, and for ethnicity (knowing Dutch a requirement for welfare recipients). Other variables include missing meetings, age, and “lacks organizing skills”, which was just one of 54 variables based on case workers’ subjective assessments. Any comment a caseworker adds translates to a 1 added to the risk score, even if it’s positive. The top-scoring 10% are flagged for further investigation.

This is the system that Accenture, the city’s technology partner on the early versions, said at its unveiling in 2018 was an “ethical solution” and promised “unbiased citizen outcomes”. Instead, Wired says, the algorithm “fails the city’s own test of fairness”.

The project’s point wasn’t to pick on Rotterdam; of the dozens of cities they contacted it just happened to be the only one that was willing to share the code behind the algorithm, along with the list of variables, prior evaluations, and the data scientists’ handbook. It even – after being threatened with court action under freedom of information laws, shared the mathematical model itself.

The overall conclusion: the system was so inaccurate it was little better than random sampling “according to some metrics”.

What strikes me, aside from the details of this design, is the initial choice of scoring benefits recipients for risk of fraud. Why not score them for risk of missing out on help they’re entitled to? The UK government’s figures on benefits fraud indicate that in 2021-2022 overpayment (including error as well as fraud) amounted to 4%; and *underpayment* 1.2% of total expenditure. Underpayment is a lot less, but it’s still substantial (£2.6 billion). Yes, I know, the point of the scoring system is to save money, but the point of the *benefits* system is to help people who need it. The suspicion was always there, but the technology has altered the balance.

This was the point the writer Ellen Ullman noted in her 1996 book Close to the Machine”: the hard-edged nature of these systems and their ability to surveil people in new ways, “infect” their owners with suspicion even of people they’ve long trusted and even when the system itself was intended to be helpful. On a societal scale, these “suspicion machines” embed increased division in our infrastructure; in his book, McQuillan warns us to watch for “functionality that contributes to violent separations of ‘us and them’.”

Along those lines, it’s disturbing that Open AI, the owner of ChatGPT and GPT-4 (and several other generative AI gewgaws) has now decided to keep secret the details of its large language models. That is, we have no sight into what data was used in training, what software and hardware methods were used, or how energy-intensive it is. If there’s a machine loose in the world’s computer systems pretending to be human, shouldn’t we understand how it works? It would help with damping down imagining we see a mind in there.

The company’s argument appears to be that because these models could become harmful it’s bad to publish how they work because then bad actors will use them to create harm. In the cybersecurity field we call this “security by obscurity” and there is a general consensus that it does not work as a protection.

In a lengthy article at New York magazine, Elizabeth Weil. quotes Daniel Dennett’s assessment of these machines: “counterfeit people” that should be seen as the same sort of danger to our system as counterfeit money. Bender suggests that rather than trying to make fake people we should be focusing more on making tools to help people.

The thing that makes me tie it to the large language models that are producing GPT is that in both cases it’s all about mining our shared cultural history, with all its flaws and misjudgments, in response to a prompt and pretending the results have meaning and create new knowledge. And *that’s* what’s being embedded into the world’s infrastructure. Have we learned nothing from Clever Hans?

Illustrations: Clever Hans, performing in Leipzig in 1912 (by Karl Krali, via Wikimedia.

Wendy M. Grossman is the 2013 winner of the Enigma Award. Her Web site has an extensive archive of her books, articles, and music, and an archive of earlier columns in this series. Follow on Mastodon or Twitter.

Gap year

What do Internet users want?

First, they want meaningful access. They want usability. They want not to be scammed, manipulated, lied to, exploited, or cheated.

It’s unlikely that any of the ongoing debates in either the US or UK will deliver any of those.

First and foremost, this week concluded two frustrating years in which the US Senate failed to confirm the appointment of Public Knowledge co-founder and EFF board member Gigi Sohn to the Federal Communications Commission. In her withdrawal statement, Sohn blamed a smear campaign by “legions of cable and media industry lobbyists, their bought-and-paid-for surrogates, and dark money political groups with bottomless pockets”.

Whether you agree or not, the result remains that for the last two years and for the foreseeable future the FCC will remain deadlocked and problems such as the US’s lack of competition and patchy broadband provision will remain unsolved.

Meanwhile, US politicians continue obsessing about whether and how to abort-retry-fail Section 230, that pesky 26-word law that relieves Internet hosts of liability for third-party content. This week it was the turn of the Senate Judiciary Committee. In its hearing, the Internet Society’s Andrew Sullivan stood out for trying to get across to lawmakers that S230 wasn’t – couldn’t have been – intended as protectionism for the technology giants because they did not exist when the law was passed. It’s fair to say that S230 helped allow the growth of *some* Internet companies – those that host user-generated content. That means all the social media sites as well as web boards and blogs and Google’s search engine and Amazon’s reviews, but neither Apple nor Netflix makes its living that way. Attacking the technology giants is a popular pasttime just now, but throwing out S230 without due attention to the unexpected collateral damage will just make them bigger.

Also on the US political mind is a proposed ban on TikTok. It’s hard to think of a move that would more quickly alienate young people. Plus, it fails to get at the root problem. If the fear is that TikTok gathers data on Americans and sends it home to China for use in designing manipulative programs…well, why single out TikTok when it lives in a forest of US companies doing the same kind of thing? As Karl Bode writes at TechDirt, if you really want to mitigate that threat, rein in the whole forest. Otherwise, if China really wants that data it can buy it on the open market.

Meanwhile, in the UK, as noted last week, opposition continues to increase to the clauses in the Online Safety bill proposing to undermine end-to-end encryption by requiring platforms to proactively scan private messages. This week, WhatsApp said it would withdraw its app from the UK rather than comply. However important the UK market is, it can’t possibly be big enough for Meta to risk fines of 4% of global revenues and criminal sanctions for executives. The really dumb thing is that everyone within the government uses WhatsApp because of its convenience and security, and we all know it. Or do they think they’ll have special access denied the rest of the population?

Also in the UK this week, the Data Protection and Digital Information bill returned to Parliament for its second reading. This is the UK’s post-Brexit attempt to “take control” by revising the EU’s General Data Protection Regulation; it was delayed during Liz Truss’s brief and destructive outing as prime minister. In its statement, the government talks about reducing the burdens on businesses without any apparent recognition that divergence from GDPR is risky for anyone trading internationally and complying with two regimes must inevitably be more expensive than complying with one.

The Open Rights Group and 25 other civil society organizations have written a letter (PDF) laying out their objections, noting that the proposed bill, in line with other recent legislation that weakens civil rights, weakens oversight and corporate accountability, lessens individuals’ rights, and weakens the independence of the Information Commissioner’s Office. “Co-designed with businesses from the start” is how the government describes the bill. But data protection law was not supposed to be designed for business – or, as Peter Geoghegan says at the London Review of Books, to aid SLAPP suits; it is supposed to protect our human rights in the face of state and corporate power. As the cryptography pioneer Whit Diffie said in 2019, “The problem isn’t privacy; it’s corporate malfeasance.”

The most depressing thing about all of these discussions is that the public interest is the loser in all of them. It makes no sense to focus on TikTok when US companies are just as aggressive in exploiting users’ data. It makes no sense to focus solely on the technology giants when the point of S230 was to protect small businesses, non-profits, and hobbyists. And it makes no sense to undermine the security afforded by end-to-end encryption when it’s essential for protecting the vulnerable people the Online Safety bill is supposed to help. In a survey, EDRi finds that compromising secure messaging is highly unpopular with young people, who clearly understand the risks to political activism and gender identity exploration.

One of the most disturbing aspects of our politics in this century so far is the widening gap between what people want, need, and know and the things politicians obsess about. We’re seeing this reflected in Internet policy, and it’s not helpful.

Illustrations: Andrew Sullivan, president of the Internet Society, testifying in front of the Senate Judiciary Committee.

Review: Survival of the Richest

A former junior minister who’d been a publicity magnet while in office once told me that it’s impossible to travel on the tube when you’re famous – except in morning rush hour, when everyone glumly hides behind their newspaper. (This was some while ago, before smartphones.)

It was the first time I’d realized that if you were going to be famous it was wise to also be rich enough to buy yourself some personal space. The problem we face today is that we have multi-billionaires who are so rich that they can surround themselves with nothing *but* personal space, and rain in the form of other people never falls into their lives.

In fact, as Douglas Rushkoff writes in Survival of the Richest, this class of human sees the rest of us as an impediment to their own survival. Instead, they want to extract everything they can from us and then achieve escape velocity as completely as possible.

Rushkoff came to realize this when he was transported far out into the American southwestern desert by a pentangle of multi-billionaires who wanted advice: what, in his opinion, was the best way to hide from out and survive various prospective catastrophes (“The Event”)? Climate change, pandemics, mass migration, and resource depletion – where to go and for how long? Alaska, New Zealand, Mars, or the Metaverse: all their ideas about the future involved escaping humanity. Except: what, one wanted to know, would be the best way to keep control of their private security force?

This was the moment when Rushkoff discovered what he calls “The Mindset”, whose origins and development are what the book is really about. It is, he writes, “a mindset where ‘winning’ means earning enough money to insulate themselves from the damage they are creating by earning money in that way. It’s as if they want to build a car that goes fast enough to escape from its own exhaust”. The Mindset is a game – and a game needs an end: in this case, a catastrophe they can invent a technology to escape.

He goes on to tease out the elements of The Mindset: financial abstraction, Richard Dawkins’ memes that see humans as machines running code with no pesky questions of morals, technology design, the type of philanthropy that hands out vaccines but refuses to waive patents so lower-income countries can make them. The Mindset comprehends competition, but not collaboration even though, as Rushkoff notes, our greatest achievement, science, is entirely collaborative.

‘Twas not ever thus. Go back to Apple’s famous 1984 Super Bowl ad and recall the promise that ushered in the first personal computers: empower the masses and destroy the monolith (at the time, IBM). Now, the top 0.1% compete to “win” control of all they survey, the top 1% scrabble for their pocket change, and the rest subsist on whatever is too small for them to notice. This is not the future we thought we were buying into.

As Rushkoff concludes, the inevitability narrative that accompanies so much technological progress is nonsense. We have choices. We can choose to define value in social terms rather than exit strategies. We can build companies and services – and successful cooperatives – to serve people and stop expanding them when they reach the size that fits their purpose. We do not have to believe today’s winners when they tell us a more equitable world is impossible. We don’t need escape fantasies; we can change reality.

Re-centralizing

But first, a housekeeping update. Net.wars has moved – to a new address and new blogging software. For details, see here. If you read net.wars via RSS, adjust your feed to https://netwars.pelicancrossing.net. Past posts’ old URLs will continue to work, as will the archive index page, which lists every net.wars column back to November 2001. And because of the move: comments are now open for the first time in probably about ten years. I will also shortly set up a mailing list for those who would rather get net.wars by email.

***

This week the Ada Lovelace Institute held a panel discussion of ethics for researchers in AI. Arguably, not a moment too soon.

At Noema magazine, Timnet Gebru writes, as Mary L Gray and Siddharth Suri have previously, that what today passes for “AI” and “machine learning” is, underneath, the work of millions of poorly-paid marginalized workers who add labels, evaluate content, and provide verification. At Wired, Gebru adds that their efforts are ultimately directed by a handful of Silicon Valley billionaires whose interests are far from what’s good for the rest of us. That would be the “rest of us” who are being used, willingly or not, knowingly or not, as experimental research subjects.

Two weeks ago, for example, a company called Koko ran an experiment offering chatbot-written/human-overseen mental health counseling without informing the 4,000 people who sought help via the “Koko Cares” Discord server. In a Twitter thread. company co-founder Rob Morris said those users rated the bot’s responses highly until they found out a bot had written them.

People can build relationships with anything, including chatbots, as was proved in 1996 with the release of the experimental chatbot therapist Eliza. People found Eliza’s responses comforting even though they knew it was a bot. Here, however, informed consent processes seem to have been ignored. Morris’s response, when widely criticized for the unethical nature of this little experiment was to say it was exempt from informed consent requirements because helpers could opt whether to use the chatbot’s reponses and Koko had no plan to publish the results.

One would like it to be obvious that *publication* is not the biggest threat to vulnerable people in search of help. One would also like modern technology CEOs to have learned the right lesson from prior incidents such as Facebook’s 2012 experiment to study users’ moods when it manipulated their newsfeeds. Facebook COO Sheryl Sandberg apologized for *how the experiment was communicated*, but not for doing it. At the time, we thought that logic suggested that such companies would continue to do the research but without publishing the results. Though isn’t tweeting publication?

It seems clear that scale is part of the problem here, like the old saying, one death is a tragedy; a million deaths are a statistic. Even the most sociopathic chatbot owner is unlikely to enlist an experimental chatbot to respond to a friend or family member in distress. But once a screen intervenes, the thousands of humans on the other side are just a pile of user IDs; that’s part of how we get so much online abuse. For those with unlimited control over the system we must all look like ants. And who wouldn’t experiment on ants?

In that sense, the efforts of the Ada Lovelace panel to sketch out the diligence researchers should follow are welcome. But the reality of human nature is that it will always be possible to find someone unscrupulous to do unethical research – and the reality of business nature is not to care much about research ethics if the resulting technology will generate profits. Listening to all those earnest, worried researchers left me writing this comment: MBAs need ethics. MBAs, government officials, and anyone else who is in charge of how new technologies are used and whose decisions affect the lives of the people those technologies are imposed upon.

This seemed even more true a day later, at the annual activists’ gathering Privacy Camp. In a panel on the proliferation of surveillance technology at the borders, speakers noted that every new technology that could be turned to helping migrants is instead being weaponized against them. The Border Violence Monitoring Network has collected thousands of such testimonies.

The especially relevant bit came when Hope Barker, a senior policy analyst with BVMN, noted this problem with the forthcoming AI Act: accountability is aimed at developers and researchers, not users.

Granted, technology that’s aborted in the lab isn’t available for abuse. But no technology stays the same after leaving the lab; it gets adapted, altered, updated, merged with other technologies, and turned to uses the researchers never imagined – as Wendy Hall noted in moderating the Ada Lovelace panel. And if we have learned anything from the last 20 years it is that over time technology services enshittify, to borrow Cory Doctorow’s term in a rant which covers the degradation of the services offered by Amazon, Facebook, and soon, he predicts, TikTok.

The systems we call “AI” today have this in common with those services: they are centralized. They are technologies that re-advantage large organizations and governments because they require amounts of data and computing power that are beyond the capabilities of small organizations and individuals to acquire. We can only rent them or be forced to use them. The ur-evil AI, HAL in Stanley Kubrick’s 2001: A Space Odyssey taught us to fear an autonomous rogue. But the biggest danger with “AIs” of the type we are seeing today, that are being put into decision making and law enforcement, is not the technology, nor the people who invented it, but the expanding desires of its controller.

Illustrations: HAL, in 2001.

Wendy M. Grossman is the 2013 winner of the Enigma Award. Her Web site has an extensive archive of her books, articles, and music, and an archive of earlier columns back to November 2001. Comment here, or follow on Mastodon or Twitter.