During a weekend in mid-May, an exclusive gathering of mathematicians took place. Thirty of the most distinguished minds in mathematics traveled to Berkeley, California, some from distant locations like the U.K. The attendees engaged in a unique challenge against a reasoning-focused chatbot, designed to tackle problems crafted by the group to assess its mathematical capability. After confronting the bot with advanced-level questions for two days straight, the participants were amazed to find that it could solve some of the most challenging solvable math problems. “Some colleagues described these models as nearing mathematical brilliance,” says Ken Ono, a University of Virginia mathematician who served as a leader and judge at the event.

The chatbot operates using o4-mini, known as a reasoning large language model (LLM). This model was developed by OpenAI to handle highly complex logical tasks. Google’s counterpart, Gemini 2.5 Flash, shares similar capabilities. Like earlier versions of ChatGPT, o4-mini learns to predict the next word in a sentence. However, compared to those predecessors, o4-mini and similar models are lighter and more agile, trained on specialized datasets with enhanced human-guided reinforcement learning. This results in a chatbot capable of deeper exploration into intricate math challenges than conventional LLMs.

To monitor o4-mini's development, OpenAI previously commissioned Epoch AI—a nonprofit focused on benchmarking LLMs—to create 300 unpublished math problems. Even traditional LLMs can correctly answer many difficult math questions. Yet when Epoch AI tested several such models with these novel problems—ones they hadn’t been trained on—the top performers managed to solve less than 2 percent, indicating their limited reasoning ability. But o4-mini turned out to be a major exception.

In September 2024, Epoch AI enlisted Elliot Glazer, a recent math Ph.D. graduate, for the benchmark initiative called FrontierMath. The project gathered original math problems across multiple difficulty levels: undergraduate, graduate, and research tiers. By April 2025, Glazer observed that o4-mini could solve roughly 20 percent of the problems. He then introduced a fourth level: questions even experienced academic mathematicians would find tough. Only a select few globally could devise—and possibly solve—such problems. Participants were required to sign confidentiality agreements and communicate exclusively through the app Signal to avoid accidental data contamination, as other communication methods like email might be scanned by an LLM and used for training.

Each problem o4-mini failed to solve earned the creator $7,500. The team made gradual progress generating suitable questions. To accelerate the process, Epoch AI organized an in-person workshop over the weekend of May 17–18, where participants finalized the last set of test questions. Divided into groups of six, the mathematicians worked intensively for two days, trying to craft problems that humans could solve but would stump the AI.

By Saturday evening, Ono grew frustrated as the bot’s surprising mathematical skill hindered the group’s efforts. “I proposed a question recognized by experts in my field as an open number theory problem—suitable for a Ph.D. thesis,” he recalls. When he asked o4-mini to solve it, he watched in astonishment as it delivered a solution within ten minutes, step-by-step. It first spent two minutes locating and absorbing relevant literature. Then, it announced it would attempt a simplified version of the problem to better understand it. Shortly after, it declared itself ready to tackle the full problem. Five minutes later, it presented a correct—but confident to the point of being sarcastic—solution. “It was starting to get really cheeky,” Ono remarked. “And at the end, it added, 'No citation necessary because the mystery number was computed by me!'”

Related: Study claims leading AI benchmarking platforms are enabling companies to manipulate model performance metrics

Sign up for the Live Science daily newsletter nowAfter witnessing this, Ono immediately messaged the group via Signal early Sunday morning. “I wasn't expecting to face off against an LLM like this,” he admitted. “I’ve never seen such reasoning in any model before. That’s how scientists work. And that’s unsettling.”

Although the group eventually identified 10 problems that the bot couldn’t solve, the researchers were astounded by how much AI had advanced in just one year. Ono likened working with the bot to collaborating with a “very capable partner.” Yang Hui He, a mathematician at the London Institute for Mathematical Sciences and an early advocate of AI in mathematics, commented, “This is what an exceptional graduate student would do—actually, even more than that.”

Moreover, the bot worked far faster than a human expert, solving in minutes what might take a professional weeks or months.

While engaging with o4-mini was exciting, its rapid progress raised concerns. Ono and He voiced worries about placing too much trust in the bot’s outputs. “There’s proof by induction, proof by contradiction, and then proof by intimidation,” He explained. “If you assert something confidently enough, people tend to believe it. I think o4-mini has perfected proof by intimidation—it presents everything so assuredly.”

The above is the detailed content of AI outsmarted 30 of the world's top mathematicians at secret meeting in California. For more information, please follow other related articles on the PHP Chinese website!