I build with AI for a living. I am not a Luddite, and I am not surprised by where the technology is going. Text-to-video and ultra-realistic image generation are exactly the kinds of tools any honest roadmap would have predicted.
What I am surprised by is how casually they are being rolled out.
If Sora 2 and Google-style image/video generators (I’ll shorthand them as “Nana Banano” here) were pharmaceuticals, they would spend a decade in clinical trials before touching the public. Instead, they are being released like fun toys with some UX guardrails and a blog post about “safety.” The gap between the potential harm and the level of scrutiny is staggering.
This post makes a simple argument:
We should seriously consider banning or heavily restricting public access to Sora 2–class models and their Google equivalents until we have real societal safeguards in place.
Not permanently. Not because the technology is inherently evil. But because right now the externalities are being socialized onto the most vulnerable people in the world, while the benefits and profits are centralized.
“It Only Took Me Five Seconds to Clone Myself”
Let me start with something personal.
Using one of these new tools, it took me less than five seconds to generate a video avatar of myself. Not a glitchy cartoon. A moving, talking, blinking version of me that, at a glance, I cannot distinguish from my actual self.
Five seconds.
If you give that power to everyone — including scammers, abusers, and state-backed propagandists — you don’t need much imagination to see where this is going.
Today, it might still take a short video to create a convincing avatar. Soon enough (if not already), it will be possible from just a single photo. That means:
- Every time you walk past a camera in a city.
- Every time you appear in a school yearbook.
- Every old Facebook or Instagram photo.
All of that becomes raw material to synthesize you saying or doing anything.
The hypothetical scam is straightforward:
A “video call” from your mother saying she’s stranded at an airport, has lost her wallet, and needs $500 wired to a “Good Samaritan’s” crypto wallet right now.
We already have SMS scams and voice cloning attacks; realistic, real-time video just lowers the friction and raises the emotional pressure.
And this isn’t science fiction. We already have public models capable of generating highly realistic videos and images from text prompts, and even big entertainment and creator organizations are publicly worrying about deepfakes and misuse. oai_citation:0‡Reuters
Abkhazia, Balloons, and Panic
It’s tempting, in San Francisco or London or Berlin, to treat all of this as interesting “media literacy challenges” that we’ll solve with better labels and watermarking.
Let me tell you about a place where that is not enough: Abkhazia.
If you haven’t heard of it, Abkhazia is a region internationally recognized as part of Georgia (the country), but currently occupied and controlled by Russia. I’ve been living in Georgia; people here are not uniformly tech-savvy. Many have phones and social media, but they are not steeped in AI discourse. Some have barely heard of ChatGPT, let alone Sora.
Now imagine this environment:
- Stalin on horseback in a hyper-realistic video, shouting that war has come.
- Exploding balloons offshore, supposedly showing early attacks.
- Genghis Khan in a tank saying he’s en route to Sokhumi.
To a Western AI engineer, these are meme-tier absurdities. To people living in a geopolitically tense, partially occupied region with traumatic recent history, they are terrifying.
And here’s the key detail: many don’t know that “Sora” (or any other AI tag) in a caption means the video is generated. They may not know that video can be faked this well at all.
This isn’t a hypothetical. Troll networks already exploit lower-information environments. Add photorealistic video to that mix and you have the perfect accelerant for fear, rumors, and panicked behavior — long before any fact-check reaches them, if it ever does.
“We Put a Watermark On It” Is Not a Safety Plan
Major AI labs like OpenAI and Google talk about watermarking, content credentials, and detection tools. These are better than nothing. But they have structural limitations:
- Watermarks can be stripped or overridden. Once an open-source model approaches similar quality, bad actors will simply use that. Even with proprietary models, screenshots, screen recordings, and re-encodes can remove or degrade watermarks.
- Detection will always lag behind generation. The same arms race we’ve seen with spam filters and ad fraud will repeat here. But this time the stakes are not annoying emails; they are elections, wars, and personal reputations.
- “This video may be AI-generated” is too weak a warning. Emotionally compelling content bypasses critical reasoning. By the time someone sees the tiny “AI” label, the damage is done.
In other words: watermarking and disclaimers are necessary, but they are nowhere close to sufficient for the scale and power of these tools.
Corporate Culture: Ship First, Apologize Later
The culture around these releases doesn’t inspire confidence.
We see leaders of major AI companies making playful meme videos — Pokémon, Nintendo jokes, “please don’t sue us” — about their own tools generating obviously infringing content. The subtext is:
“Yes, we know this is questionably legal and ethically gray, but relax, it’s cool, and besides, it’s already out there.”
That’s not safety. That’s marketing.
The pattern looks worryingly familiar:
- Ship a powerful new capability with minimal external oversight.
- Allow misuse to surface in the wild (often at the expense of victims who did not consent).
- Announce new “safety measures” in response to public outrage.
- Repeat.
We already see this with deepfake tools: public figures and unions complain about unauthorized AI videos, and only then does the provider tighten policies or add opt-in rules for using someone’s likeness. oai_citation:1‡New York Post
This is reactive ethics, not proactive governance.
If this were a new psychoactive drug or a gene-editing therapy, regulators would never permit this approach. You don’t roll out a new molecule to hundreds of millions of people first and then see what happens.
“But the Genie Is Out of the Bottle”
I can already hear the counterargument:
“The technology exists. Hackers will build it anyway. Banning Sora 2 or Google’s models won’t stop bad actors.”
I agree with the premise and disagree with the conclusion.
Yes, the genie is out of the bottle in terms of possibility. But who has access, at what scale, and with what defaults still matters enormously.
- If only a small number of state-level actors can do something, that’s one kind of threat model.
- If everyone with a phone can do it in five seconds via a slick consumer UI, that is a very different world.
We routinely regulate powerful technologies even when they cannot be fully contained:
- You can’t buy explosives at the supermarket.
- You can’t run an unlicensed nuclear reactor in your backyard.
- You can’t manufacture and sell new drugs without clinical trials.
None of these bans stop determined bad actors completely. But they raise the bar and limit the blast radius.
Right now, Sora-class tools are being rolled out more like a casual social app than a dual-use technology with profound implications for security, democracy, and mental health.
What Would a Responsible Path Look Like?
I don’t pretend to have a complete solution. But I can sketch what a more responsible approach might include, especially for Sora 2–level and “Nana Banano”-level models:
Temporary moratorium on general-public access to photorealistic text-to-video and face-swap tools, until:
- We have robust, independently audited provenance standards.
- Legal frameworks for non-consensual deepfakes are clear and enforced.
- Major platforms have agreed protocols for handling AI-generated media during crises and elections.
Licensing and tiered access.
- High-fidelity identity-manipulating tools treated more like controlled technologies.
- Access granted to vetted entities (e.g., studios, research labs, regulated companies) under strict contractual terms, not just “everyone with a credit card.”
Default identity protections.
- Hard rules that prevent generating specific real people (not just celebrities) without strong consent mechanisms.
- Swift takedown obligations and liability for providers who fail to act on abuse reports.
Stronger duties for big AI companies.
- External red-team testing focused on geopolitical misinformation in low-information regions, not just U.S. domestic politics.
- Mandatory transparency about known failure cases and misuse, not just cherry-picked success stories.
- Coordination with regulators before rollout, not after scandals.
These are not trivial steps. They will slow down product roadmaps. They will add friction to “move fast and break things.” That is exactly the point.
Why I’m Calling for a Ban (For Now)
To summarize my position as clearly as possible:
- The current generation of text-to-video and hyper-realistic image tools is extraordinarily powerful.
- The costs of misuse fall hardest on:
- People in lower-information environments (like Abkhazia),
- Vulnerable individuals who can be impersonated,
- Democracies and conflict zones where trust is already fragile.
- The safeguards in place today are reactive, fragmented, and largely voluntary. They are not commensurate with the risks.
- The corporate attitude — ship first, then patch safety — is unacceptable for technology that can rewrite reality on a screen.
Given all of that, I believe:
It is reasonable, even necessary, to consider banning or tightly restricting public access to Sora 2–class and “Nana Banano”-class tools until we have built legal, technical, and social infrastructure that can handle them.
This is not a plea to roll back progress. It is a plea to align progress with responsibility.
Words — including the words generated by language models like the one you’re reading this through — have always been tools for propaganda and persuasion. We’ve had centuries of practice dealing with that.
Photo-realistic, on-demand synthetic video is different. It hijacks our visual trust systems in ways we are only beginning to grasp. Releasing it to billions of people with essentially no democratic debate and minimal oversight is, in my view, reckless.
We can do better. And we should demand better — from OpenAI, from Google, from Meta, and from every other company racing to own this new visual layer of reality.
Until then, a pause — or even a ban — on Sora 2–level public tools is not overreaction.
It is the bare minimum of care.
https://youtube.com/shorts/Fr5MpyREpJI