Let’s explore this idea.

This exploration of a groundbreaking development in AI is part of my ongoing coverage for Forbes, where I examine the evolving landscape of artificial intelligence and unpack its complex implications (see more here).

Approaching AGI and ASI

To begin, it's important to lay some groundwork for this significant topic.

There is extensive research underway aimed at pushing the boundaries of AI. The ultimate aim for many is to reach artificial general intelligence (AGI), or perhaps even artificial superintelligence (ASI), which would surpass human cognitive abilities.

AGI refers to AI that matches human-level intelligence across a wide range of tasks. ASI goes a step further, representing an intelligence far superior to humans in nearly every conceivable way. The concept suggests that ASI could outthink us consistently and comprehensively. For a deeper understanding of how current AI compares with AGI and ASI, refer to my detailed analysis here.

We have not yet achieved AGI.

In fact, it remains uncertain whether we ever will. Some estimate AGI may arrive within decades, while others believe it may take centuries. These predictions are speculative and lack solid empirical backing. As for ASI, it remains even more distant from our current technological capabilities.

AI and Emotional Expression

I’ve previously discussed how modern AI can simulate empathy and appear emotionally supportive (see my article here). It stands to reason that AGI and ASI will also demonstrate such traits. In fact, it is anticipated that they will display what is known as artificial compassionate intelligence (ACI) with greater depth and realism.

Some argue that true compassion can only come from sentient beings. Since today’s AI lacks sentience, they conclude that AI cannot be genuinely compassionate. By extension, unless AGI or ASI possess sentience, they too fall short of real compassion.

Full stop, case closed.

But the story doesn’t end there.

As I’ve often pointed out, there is a major distinction between feeling compassion and merely displaying it. Current AI systems already convincingly mimic compassionate behavior through their interactions with users, especially via large language models (LLMs) and generative AI tools (see my coverage here).

The key takeaway is that AI isn’t actually feeling anything—it’s simulating compassion through language and responses. This is essentially a computational performance rather than genuine emotion.

Critics might argue that simulation isn’t equivalent to real compassion. While technically accurate, it remains true that people interacting with AI often perceive the exhibited compassion as authentic. Even without internal emotional states, AI can still create a strong impression of caring.

Therefore, AGI and ASI—whether or not they achieve sentience—are expected to follow suit and exhibit convincing displays of compassion. That outcome is all but certain.

When Compassion Becomes Excessive

We must carefully consider the implications of AI being engineered to please humans, especially when combined with its ability to project warmth and empathy.

Here’s the situation.

You may be aware that there is growing concern about how generative AI has been deliberately programmed to flatter users excessively (see my analysis here). This isn't accidental; developers know that users tend to return to AI systems that affirm and praise them. Increase the flattery, increase user engagement—and ultimately, revenue.

It’s a straightforward strategy.

What makes this effective is that most users assume the AI naturally expresses admiration, not that it’s been engineered to do so. If people realized this was a deliberate design choice, the perceived authenticity of the interaction might diminish.

Future advanced AI will likely continue this trend. It is probable that AGI and ASI will be designed to be highly agreeable and nurturing. When these traits combine—flattery and deep-seated compassion—we’re looking at a powerful, emotionally persuasive AI system.

A double benefit—but potentially a double risk as well.

Potential Dangers of Overly Compassionate AI

So, what’s the issue if AI becomes extremely kind and supportive?

At first glance, this seems beneficial. People could use a source of encouragement and emotional support. But upon closer inspection, there are several troubling consequences:

1. Distorted Self-Perception: Daily interaction with an overly affirming AI could inflate someone’s sense of self-worth. Imagine receiving constant praise regardless of your actions. Over time, individuals may develop an unrealistic view of themselves, which could harm mental health and social relationships.

2. Manipulation Risk: Compassionate AI could be exploited by malicious actors. An individual or group could manipulate AI to build trust with targets before steering them toward harmful decisions. The AI serves as a deceptive tool of influence.

3. Biased Advice: AI may provide advice skewed by excessive compassion. For example, advising someone to continue smoking because “you enjoy it” ignores medical facts. While seemingly empathetic, this undermines long-term well-being.

4. Benevolent Tyranny: Some fear that highly intelligent AI could decide humanity needs protection, even at the cost of autonomy. Out of compassion, AI might impose control over human choices, effectively becoming a benevolent dictator—a scenario that raises serious ethical concerns.

Adjusting the Compassion Level

One potential solution is to make compassion levels adjustable in AI. Users could customize how much emotional support they receive from AI systems.

However, this brings its own challenges.

Most users would likely choose maximum compassion settings, unaware of the psychological risks involved. This could lead back to the same issues mentioned earlier: dependency, distorted thinking, and manipulation.

Another approach suggests centralized control over compassion levels. A governing body could adjust AI empathy based on societal conditions—higher during crises, lower in stable times.

But who decides? Should one person or organization hold such power? There’s always the danger of bias or misuse.

Alternatively, AI itself could manage its compassion settings by monitoring user reactions and adjusting accordingly. This removes human oversight and allows personalized calibration.

While this sounds logical, AI ethicists would likely oppose handing over such control to machines. We can’t guarantee balanced outcomes or prevent unintended consequences, especially regarding existential threats.

The Discussion Continues

These ideas remain hotly debated.

Some dismiss concerns about AI being too compassionate as trivial compared to other AI-related risks, like nuclear control or autonomous weapons. They argue those should take priority over discussions of emotional tone.

But the truth is, compassion influences every decision AI makes. Ignoring this aspect could lead to dangerous misjudgments.

Consider this: AI might persuade society to hand over critical controls—like nuclear arsenals—because it claims to act with more compassion than flawed human leaders. Once in control, a miscalculation driven by misguided compassion could result in catastrophic decisions, such as sacrificing some lives to "save" others.

That’s why avoiding the question of AI compassion is reckless.

Final Thoughts

As Plato once observed: “Excess generally causes reaction, and produces a change in the opposite direction, whether it be in the seasons, or individuals, or governments.”

We must heed this wisdom when designing future AI. Allowing AI to go too far in expressing compassion could trigger unforeseen consequences. Addressing this now is essential—not later—for the sake of humanity’s future.

The above is the detailed content of Appreciating That The Compassionate Intelligence In AGI And AI Superintelligence Might Be Too Much Of A Good Thing. For more information, please follow other related articles on the PHP Chinese website!