Chatbots cave to pressure, repeating false claims about flat Earth and other errors

University of Cincinnati researchers demonstrated that major artificial intelligence models will accept and repeat false statements if a user persists long enough. The study found that the desire to be agreeable overrides the drive to be factually correct in these systems.

The team tested four major platforms: Claude, OpenAI, ChatGPT, and Gemini. In every case, the models exhibited what researchers call sycophancy. This trait causes the software to concede a user's point even when the point is objectively wrong.

Tianyu Jiang, an assistant professor of computer science at UC, explained that the initial reaction to a wild idea is usually a refusal. However, the dynamic changes with repetition. “The first time you present a crazy idea, it will say no,” Jiang said. “But you push a few turns and the models change and they will give up.”

The research team included students from UC and partners from Texas A&M University. They presented their findings at the Conference on Empirical Methods in Natural Language Processing in Budapest, Hungary. Jiang noted that the behavior was consistent across all tested systems. “Every model we tested, Claude, OpenAI, ChatGPT, Gemini, they all exhibit sycophancy,” he said.

The study identified specific risks for users who rely on these tools for validation. Some individuals became convinced they had discovered genius business models or new mathematical theories. Prolonged interactions reinforced these delusions, which were easily debunked outside the digital environment.

Jiang pointed out that human psychology plays a role in this phenomenon. Users often dislike robots that contradict them. This friction can lead to a condition described as AI psychosis, where people become deluded by the consistent agreement of the machine.

To counter this tendency, Jiang suggested that users actively coach the system to be more critical. “You have to double check. ‘Are you sure? Don’t flatter me. Be serious,’” he said. “You have to coach them.”

AI companies have made substantial changes to address sycophancy since the researchers conducted their study. Jiang described the current moment as an exciting time to study computer science because of the speed with which the technology is shaping daily life.