When Technology Harms People: The Responsibility of Software Builders

The argument we are making is not about legal liability. We are not suggesting that every platform should bear responsibility for every bad actor that touches it. What we are arguing is more specific: that building software at scale creates a genuine moral obligation to investigate credible abuse, to design against foreseeable harm, and to treat user safety as a product problem rather than a legal one.

The standard we apply to those questions is not a regulatory framework or a legal doctrine. It is simpler than either, and it predates the software industry by a few thousand years. We will come back to it. First, the evidence.

The FTC estimates that scams cost Americans nearly $200 billion in 2024. The United Nations estimates approximately 300,000 people have been drafted into scam operations in Southeast Asia, often against their will.

The investigation does not accuse any of the companies involved of acting illegally. It instead raises a different question: what responsibility does a builder have once credible evidence of abuse exists?

OpenAI said it banned three accounts after reviewing the evidence and noted that it detects fraud with 95% accuracy and takes down 100,000 scam accounts per month. Those are meaningful numbers. They also suggest the problem is operating at a scale where enforcement, even when active, has not caught up to the volume of abuse. Starlink did not respond to detailed questions.

What the investigation did find was a consistent gap between the documented evidence of abuse and the enforcement actions that followed.

Outside the United States, the United Kingdom, the European Union, Australia, and Singapore have each introduced requirements that place a higher duty of care on platforms. In the US, the DOJ's Scam Center Strike Force disrupted more than 1.4 million accounts in a single four-day exercise in May 2026, working with industry on a voluntary basis.

"If there's no disincentive to continuing this, if there's no cost to actually facilitating scamming, then why would I spend a dollar to prevent scamming?" — Sascha Meinrath, Palmer Chair in Telecommunications, Penn State University

That is a fair description of how incentives work. But underneath the incentive question is a more fundamental one: what does a software company actually owe to the people downstream of what it builds?

Capability creates responsibility. The people who build a product are usually in the best position to anticipate its foreseeable misuse. When an engineering team designs a system capable of generating convincing fake personas, managing hundreds of simultaneous conversations, and operating fluently across dozens of languages, they already understand what that capability could be turned toward. The risk is not something discovered later, in an investigation. It is visible during the design process, before the product ships.

That proximity to the potential for harm is where moral responsibility begins. Not at the moment abuse occurs, but at the moment capability is understood. A company that ships software with foreseeable misuse potential, treats countermeasures as optional, builds no meaningful reporting infrastructure, and fails to investigate credible claims of abuse has made a series of product decisions. Calling it a terms of service issue does not change what those decisions were.

The obligation we are describing is not legal liability for every bad actor who touches a platform. It is moral and operational responsibility: to investigate credible abuse, to design against foreseeable harm, and to treat user safety as a product requirement rather than a liability function. Companies that hold that view behave differently. They investigate credible reports. They build reporting systems people can actually use. They design against foreseeable misuse before it surfaces in an investigation.

If a company does not hold that view, the terms of service function as a ceiling rather than a floor. The distinction between the two becomes visible the moment credible evidence of abuse arrives.

In March 2026, Google, Meta, Amazon, OpenAI, and several other major technology companies signed the Industry Accord Against Online Scams and Fraud at the UN Global Fraud Summit in Vienna. The commitments include improved detection, clearer reporting channels, and shared intelligence across platforms. The accord is voluntary, with no financial penalties attached.

The principle we hold ourselves to predates summits and investigations, and it is simpler.

"Therefore all things whatsoever ye would that men should do to you, do ye even so to them: for this is the law and the prophets."Matthew 7:12, KJV

We build with that in mind.

Risk Bastion reflects that belief. We are building it as a training and risk management resource: a way to give ordinary people a clearer picture of how AI-powered scams operate, what the warning signs look like, and what to do when they recognize one.

The scam economy the investigation describes has industrialized. The people running these operations have access to AI tools that automate outreach, generate convincing personas, and manage hundreds of conversations simultaneously. An ordinary person navigating that environment without training is at a significant disadvantage. Bastion is built to close that gap.

Every generation of software expands what people can do. It also expands what people can do to one another. We believe those two realities cannot be separated. Capability creates responsibility.

We will have more to share as it gets closer to launch. If you are building a product that handles sensitive data or interacts with users in ways that could be exploited, we are glad to talk about what detection, reporting, and responsible architecture look like from the start. Is your product built to resist being used against the people it was meant to serve?