Loading...
In 2016, a team of researchers at Facebook's own internal division discovered something alarming: the platform's recommendation algorithm was systematically steering users toward increasingly extreme content. A user who watched a video about vegetarianism would be recommended videos about veganism, then about animal rights activism, then about eco-terrorism. A user who liked a post about immigration policy would be shown increasingly inflammatory content until they were consuming content from white nationalist groups. The algorithm wasn't ideological — it was optimizing for engagement, and extreme content held people's attention longer. The mechanism was elegant in its simplicity. Each click, each second of watch time, each comment told the algorithm what kept this particular user engag...
Popular framing: Facebook and YouTube are radicalizing people on purpose.
Structural analysis: The algorithm is not ideological; it optimizes for engagement, and outrage plus tribal identity are the most reliable engagement drivers across demographics. Gradient escalation produces a rabbit-hole effect emergent from local optimizations. Ad revenue scales with engagement, so every reform pits safety against the model that funds the company — moral hazard locked in by the business model.
The popular framing demands a villain (executives who chose profit) while the structural framing reveals a trap (the incentive architecture deterministically produces the harm regardless of intent). This gap matters because villain-framing produces accountability remedies — fines, leadership changes — that leave the underlying feedback loop intact. Only structural framing points to the actual lever: changing what the objective function optimizes for, which requires redesigning the engagement-reward loop rather than moderating individual outputs of it.
