Trust & Safety (Is Not Actually) Fun

Enjoy playing through the peril of running trust & safety at a social media startup.

by

Published

Trust & Safety is the department in technology companies wrought with really challenging policy questions. How do you strike a balance between keeping users safe and happy, and allowing users to express themselves? Tradeoffs. And really hard decisions.

These are explored in the recent web game, Trust & Safety Tycoon, created by Mike Masnick, Randy Lubin, and Leigh Beadon. Masnick and Beadon are both from the technology blog Techdirt and the think tank Copia; Lubin is a game designer and consultant.

The three have been making games for some years. Their previous game, Moderator Mayhem, also about moderation, was about making quick decisions—think the 2013 video game Papers, Please.

Trust & Safety Tycoon—though not actually a tycoon game—is about making policy at a fictional social media site, Yapper. (‘Yapper’, like ‘yapping’.) The questions start out slow and simple, but progress towards more nuanced and complex ethical dilemmas. You’re also given a budget at regular intervals, and you have to strategically choose what to invest in to reduce pressure on your moderators.

The core of this game is educational/informative: the questions the game asks of you are somewhat representative of what real-world social media sites’ T&S teams have to deal with. (I have been a moderator for small communities for many years and have a deep interest in moderation.) Their decisions can have massive and dire consequences. Choose wisely.

Some obstacles intersect with the law. You and your team need to understand copyright law, for example. But you also need to be able to interface with law enforcement, like the FBI, who want to collect evidence that you have of users.

You get feedback from your decisions, in the form of ‘your users are angry’, mostly. Even if you do the right thing, people will get angry. Your metrics will go down, and you might be fired.

“We wanted to show how fraught trust and safety issues can be, from crafting policies to enforcing them,” Lubin, the game designer, said over email. “We tried to have the responses cover the spectrum of options that real world teams have, although the real world allows for much more nuance and creativity than we could model in the game.”

Masnick told the Washington Post that the idea for the game came to him when talking to someone who led a Trust & Safety team expressed the want for people to really get a grasp on the dilemmas their teams face. The game’s purpose is to show this interactively.

Masnick is a purveyor of public and company policy. He ferociously argues for “free speech” in his social media posts and blog posts on Techdirt. His “freedom of speech” is the one in the First Amendment, which includes the freedom to boot people off your private company, not the “freedom of speech” conservatives want where you are forced to carry speech you don’t agree with. (Masnick did not respond to requests for comment.)

Over time, according to Lubin, they spoke to around 40 people and experts working in T&S roles. “Most of the dilemmas, options, and consequences were inspired by real world issues that T&S teams have faced,” Lubin told me. “Many T&S workers reached out to us afterward saying that they ‘felt seen’ and that it was a good portrayal of the issues they face.”

The profession is unfairly maligned. A suspension, removal of content, or a demonetization gets people angry. The people doing that is typically T&S. The process is opaque, too: you don’t often get more than a vague reason for a punishment, and requests for clarification might not be possible. But behind the curtain are real, fleshy humans making decisions that aren’t clear-cut, making sense the best they can of possible violations.

Oh, and I won the game easily on my first try.