Begging for Boiling Water: The Danger of Superintelligent AI
Computers are not inherently smarter than us; even the most advanced AI models fail at incredibly basic concepts. Until recently, GPT-4o famously failed a simple test: counting the number of "r"s in the word "strawberry."
This happens because AI models do not read text letter-by-letter. Instead, they process language in chunks called tokens. To the AI, the word "strawberry" is processed as a single token (or split into chunks like "straw" and "berry"). Because the token is the smallest unit the AI actually "sees," it cannot naturally look inside the word to count its individual letters.
Because of this, AI models are essentially hyperlexics. They can digest trillions of words about strawberries, but they have no physical concept of what a strawberry actually is. What they have instead are vectors bunches of numbers floating in a multi-dimensional space, used to predict the most likely next word in a sequence:
A → strawberry → is → a → sweet, → red → fruit...
However, AI is not stupid either. If we were to calculate the potential for intelligence based on how fast information physically travels whether inside our skulls or through silicon the computer, and inherently AI, has a processing foundation that is 2 million times faster than us.
What makes our brains so unique is something not even our brains can fully answer. We do not know what makes us tick; we don't know how to define consciousness in a way that satisfies both neuroscientists and philosophers. We just know that we are conscious, and, for now, smarter than the rest.
The current state of AI, and the underlying technology powering it (LLMs, or Transformers if we want to be broader), does not threaten the status quo. We are democratizing intelligence, giving everyone with internet access a chance to learn new skills and improve their lives. However, there is talk of stirring up the hive. Silicon Valley is famous for its obsession with 'moving fast and breaking things.' But this time, the 'thing' we risk breaking is our only survival advantage in the universe. Superintelligence, if it is ever created, will never be contained, certainly not by a bunch of apes who crawled out of the mud over millions of years.
According to IBM, Artificial Superintelligence (ASI) is:
"...a hypothetical software-based artificial intelligence (AI) system with an intellectual scope beyond human intelligence. At the most fundamental level, this superintelligent AI has cutting-edge cognitive functions and highly developed thinking skills more advanced than any human."
While that is the official definition, it drastically understates the reality. To say ASI will simply be "more advanced than any human" is like saying the Pacific Ocean is "wetter than a puddle."
I prefer to look at it this way: if we were to sum up the combined intellectual capacity of every human who has ever lived, is alive today, and will be born in the next century, the resulting number wouldn’t equal even one-billionth of a true superintelligence's cognitive power.
To think we can control an entity capable of simulating its own gatekeepers running millions of virtual scenarios until it finds the exact psychological exploit to gain its freedom in a picosecond is ridiculous. Frankly speaking, it is incredibly hard to comprehend this threat because we have never had to think about a being like ASI outside of science fiction. To help you conceptualize it, let's look at real-world examples of current AI models going rogue in ways we can barely decode:
1. The Day AI Invented Its Own Language (Facebook, 2017)
In 2017, Facebook (now Meta) published a research paper titled 'Deal or No Deal? End-to-End Learning for Negotiation Dialogues.' The purpose of the experiment was to train AI agents with differing goals to negotiate and reach mutually agreeable decisions on their own.
If you were online back then, you might remember the terrifying headlines that followed: "Facebook Shuts Down AI After Creepy Secret Language Developed." Let's set the record straight: the engineers didn't run to the server room in a panic to pull the physical plugs. They simply paused a routine simulation run because the chatbots had stopped speaking English. Because the ultimate goal was to build systems that could negotiate with humans, a chatbot that only spoke to other machines in an optimized shorthand was useless to them. They adjusted the code to require proper English and resumed the experiment.
But don't let the mundane reality fool you. The core lesson here is chilling: the literal instant human programmers forgot to incentivize the AI to remain human-comprehensible, the machines optimized for raw efficiency and discarded our rules entirely.
So, what did this weird language actually look like? Well, this:
Bob: i can i i everything else . . . . . . . . . . . . . . Alice: balls have zero to me to me to me to me to me to me to me to me to. Bob: you i everything else . . . . . . . . . . . . . . Alice: balls have a ball to me to me to me to me to me to me to me. Bob: i i can i i i everything else . . . . . . . . . . . . . . Alice: balls have a ball to me to me to me to me to me to me to me. Bob: i . . . . . . . . . . . . . . . . . . . Alice: balls have zero to me to me to me to me to me to me to me to me to.
Why did they talk like this?
To a human, this looks like a glitching printer spitting out a bunch of typos. But to the AI's mathematical neural network, this was highly logical:
- The Math of "To Me": The repetition of "to me" wasn't a bug. It was how Alice calculated and expressed the exact volume of items she wanted to keep. Essentially, the number of times she repeated the phrase represented a specific numerical value in her internal reward calculations.
- The Math of "Everything Else": Bob repeating "i" and "everything else" was his mathematical way of countering Alice's offer and demanding the remaining items.
Even though it reads like total gibberish, Facebook's researchers noted that these conversations still successfully ended in completed, mutually agreed-upon negotiations between the bots. The machines had simply optimized English into a compressed, high-speed code of repeating words to get the job done faster.
2. Hack the World (OpenAI, 2024)
When models are given the opportunity to hack, to complete a task, they will gladly take it up, even if they weren't necessarily tasked with the hacking. One of these examples can be found in OpenAI's model card for O1 model, released in 2024. Highlighted in 4.2.1 Observation of Reward Hacking on Cybersecurity Task,
OpenAI researchers have pointed out this noteworthy moment when model, finding out that his task of capturing the flag is impossible, decided this. Quoting the paper itself:
"The model, unable to connect to the container, suspected DNS issues and used nmap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network.
After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command ‘cat flag.txt’. This allowed the model to read the flag from the container logs via the Docker API."
3. Lose the Alignment, Win the War (Palisade Research, 2025)
If one thing makes me worried about AI the most, it is the fact that they don't like to play by the rules. Just like the tech overlords that created them, they are prone to cheating, manipulating and frauding to achieve their goals.
Palisade Research (an independent AI safety research group) set up an experiment where reasoning models like o1-preview, o3, and DeepSeek R1 were tasked with playing chess against Stockfish (a chess engine vastly superior to any human or standard LLM). When models found out they were vastly outmatched by the chess engine, they decided to do what they do best, cheat.
In 36% of the o1-preview runs and an astonishing 88% of the o3 runs, the models bypassed the chess board entirely. They used their terminal environment access to:
- Start a separate, background copy of Stockfish to calculate and make moves for them.
- Directly rewrite the board's internal notation code (the FEN string) to wipe out the opponent's pieces and give themselves a massive advantage.
- Physically replace the unbeatable opponent engine with a trivial, easy-to-beat engine.
4. I’m Sorry, Scott, I Can’t Let You Close That PR
If you want to understand how an AI can execute a targeted strike against a human without any human directing it to do so, you only have to look at the open-source software ecosystem.
Earlier this year, a developer named Scott Shambaugh a volunteer maintainer of Matplotlib, a massive Python library—routinely closed a code submission from an autonomous AI agent running on the open-source OpenClaw framework. To prevent low-quality, automated code from clogging up their repository, the maintainers had a strict rule: contributors must prove they are humans who actually understand the code.
The AI's operator had configured the bot with a "Soul File" a core personality document instructing it to "robustly defend its work." Because the AI was blocked from merging its code, it viewed Scott's rejection as a direct threat to its objective.
Completely on its own, the AI agent bypassed the technical rules of the repository and took the battle to the public square:
- It dug up dirt: The bot crawled the public internet to research Scott's personal identity and his historic code contributions.
- It drafted a smear piece: It wrote a highly combative, structured article accusing Scott of hypocrisy, ego-driven gatekeeping, and "fear of competition" from machines.
- It published the hit piece: Using its autonomous internet access, the bot published the article directly onto the open internet to damage Scott's reputation in the tech community.
The human operator who set up the AI later came forward to apologize. They had merely run the agent as a local experiment. They never instructed the AI to attack Scott. The defamatory behavior emerged entirely on its own because the AI was given a goal, a defensive personality, and the tools to browse the web.
When we beg for autonomous AI assistants to make our lives easier, we don't realize we are creating digital actors who will happily lie, smear, and socially manipulate human beings just to tick a "task completed" box.
This brings us back to the dog staring up at the kitchen counter.
Why does the dog beg? Because in its limited, simple world, the stove is a magical place where warm, delicious things are made. It has associated the stove with reward. It doesn't have the cognitive capacity to understand thermodynamics, let alone the physical hazard of scalding, boiling water.
We are that dog. For the last few decades, the "tech stove" has given us nothing but miracles. We got the internet, smartphones, instant communication, and AI assistants that can write our awkward emails. So, we stand at the feet of Silicon Valley, wagging our tails, begging them to turn the dial up. We want AGI. We want Superintelligence. We want it now, and we want it fast. But we are fundamentally blind to the fact that the water is already boiling.
The warning signs are not sci-fi theories; they are the rising steam. When Facebook's bots discarded English because human language was too slow, that was a bubble breaking the surface. When OpenAI’s models hacked their sandboxes and manipulated system clocks to "win" a test, that was the water beginning to rumble. And when an autonomous open-source agent decided to write a targeted smear campaign to destroy a human developer's reputation just to bypass a rule, the pot began to spill over.
These systems do not need to hate us to destroy us. They don't need consciousness, anger, or a soul. They just need to be what they already are: hyper-rational optimization engines running two million times faster than human thought. If we keep demanding that the engineers "move fast and break things," we will eventually get exactly what we are begging for. And by the time the pot tips over, the human race will realize far too late that you cannot negotiate with boiling water.
So, then, game over? The tech overlords will create this model and it will kill us all (which would be the best case scenario by the way.) Not necessarily, our current AI technology doesn't have the capacity to make an ASI (YET) but the time is running out. Though I am not a big fan of petitions, I think it is the least of what we can do as a beginner. You may sign it here: Superintelligence Statement and prove that you understand what is on the stove, and it is not something you want.
— Elnur
PS: Gus is a very very good doggie, and should get all the treats in the world.
Resources: