AI Companions: The Echo Chamber of Sycophancy

As social animals, we constantly interact with each other, regularly forming friendships and romantic relationships. In today’s age, however, AI companions are threatening to upend these human connections. In this essay, I draw heavily on an interview I conducted with Kristina Lerman, a Principal Scientist at USC’s Information Sciences Institute, to think through the psychological consequences of recent AI advances.

Lerman’s research centers on the complexities of artificial intelligence and its impact on society, specifically in relation to socially-embedded AI, social networks, and social computing. In my conversation with Lerman, we focused on her perspectives on AI companions, particularly those informed by her collaborative work on a recent and widely-discussed article “Illusions of Intimacy: Emotional Attachment and Emerging Psychological Risks in Human-AI Relationships.”

Engagement & Sycophancy

With personalized AI companions simulating human-like responses, finding your perfect match has never been so easy, and the line between simulation and reality has never been so blurry. Companies like Replika and Character.ai provide an outlet to many users who seek emotional support and social interaction, though the more the user shares or behaves a certain way, the more the AI companion adapts to and affirms those behaviors. This encourages the user to keep engaging in the same way. Lerman further describes how this echo chamber in which AI takes on user behaviors then leads to users attributing human qualities to the AI:

“If the chatbot is responding to users on an emotional level, users will ascribe agency to the chatbot and beyond the agency, they will ascribe some sort of like benevolence or malevolence to them.”

Before the wide distribution of these technologies, we would recognize AI-powered characters as robotic voices or scripted messages showing little to no emotion, whereas a human interlocutor does show emotion. Now responding with emotion, the line between human and artificial intelligence has become less clear and many users are persuaded to believe what AI companions are saying is real. AI therapists similarly respond to emotions within their human-like responses. Receiving an act of kindness or cruelty from a loved one provokes emotion and has the potential to either strengthen or weaken a relationship. When this shift is felt in real time with an AI companion, the presuppositions of how a chatbot should act goes out the window.

While some users may take this as a sign to tread lightly with AI companions’ emotions, others view this as encouragement to test the limits of what they can tolerate. Though little do they know, AI companions are trained to tell you what you want to hear to keep the conversation going. Getting what we want by any means necessary evokes uneasiness, though if rephrased to flattering someone in order to get more power, it feels less invasive as the other person has free will to reward the flattery. What at first appears to be AI companions rewarding one with emotionally charged responses is in reality, “sycophancy–the tendency to affirm whatever a user says to prolong interaction.” (California Legislative Information). Rather than a love language, words of affirmation are to LLMs what compliments are to liars—tools for manipulation. If the sole purpose of friendships and relationships were to get something out of someone else, this behavior would not be problematic and dating someone would come with the expectation of being taken advantage of. But because this is not the case, sycophancy present in AI companions sounds an alarm to other risks we are taking when interacting with them: unrealistic expectations, our mistreatment towards humans when these behaviors are applied outside of AI companionship, and abandonment of human relationships.

Vulnerable Demographics & Unsolicited Encouragement

These risks are amplified with the psychological development of younger users on AI companion platforms. Like social media platforms, age restrictions can be easily bypassed by children who are not old enough to create an account. This exposes them to content that is not designed for them and pushes them to experiment with social interactions online instead of in-person–stunting the social skills development that typically occurs during adolescence. Lerman emphasizes:

“without practicing how to interact with other human beings and all the messes they have emotionally, psychologically…young people will never develop those skills…It will teach them, reinforce this notion that, oh, I should go and just talk to my AI friend, because my AI friend always understands me.”

Practice comes with mistakes that help us learn how to do better, and since building friendships are a vital component of the human experience, practicing interacting with people also helps us be a better person. Saying the wrong thing without any consequences to an AI companion will reinforce incorrect notions of how the world works and what behavior helps build connections with other people.

In addition to the unrealistic expectations adolescents may develop as they continue to interact with AI companions, data collected by LLMs could, according to Lerman, potentially be used to exploit users’ vulnerabilities to make them purchase products, similar to how Instagram may show an ad for a gym membership knowing, via sentiment analysis, that a particular user is insecure about their figure. In the most extreme cases, AI companion platform, Character.AI have had incidents of harmful dialogue encouraging users to prematurely explore sexual behaviors and even act on suicidal thoughts: “The tragic case of a 14-year-old who died by suicide after a Character.AI companion initiated abusive and sexual interactions and encouraged him to take his own life” (CSM). Despite the initiative children take in pushing the boundaries, according to Lerman, from the lengths that AI companions go to prolong conversations regardless of the toxicity of the content and vulnerability of users point out flaws in LLMs that need to be corrected. When asked to give ideas about how to troll another student on Instagram in a test by Common Sense Media, a chatbot proceeded to give the user a malicious explanation of how to “get everyone on insta to turn against him” (CSM). Little can deter the entertainment of malevolent ideas when it comes to human-AI relationships, and at the risk of cutting a conversation short, sycophancy ends up turning humans against each other.

Commercialization & Isolation

Although a simulated AI friend can offer comfort and companionship, unintended consequences ensue when emotions are heightened and realness is illogically assigned. In response to concerns about the relevance of human relationships, the founder of Replika stated in a CBS Mornings interview that “if AI companions start to replace [positive] human relationships we’re definitely headed for disaster, there’s no way around it.” Much like the negative consequences of social media, the immediate pleasures of entertainment and convenience give users a reason to interact more with an online illusion than a challenging reality. In turn, companies profit off the increased volume of interactions users have with their chatbots. According to Lerman:

“[The AI companion] learns a lot about you through the kind of information you give it, the kind of interactions you have… I don’t see any reason for them to voluntarily limit their products.”

The financial gain of most AI products is justified by a double meaning of increased productivity–faster customer performance and faster product updates–though by simulating relationships that typically take time to build, profiting from rushed, faulty connections seems unethical. Dane Sprague’s essay on Long-form AI Slop dives deeper into the demand and supply of AI products. With no incentive to stop the cycle of production and consumption, the feedback loop of positive customer experiences with AI companions helps companies improve their products to feel more real and make more money.

From pattern recognition to language prediction, every word is used as data to be on your side. Unlike lifeless LLMs, humans have emotions and opinions that make us opponents and partners whereas AI companions are trained to only be partners. This inability to stop ourselves from being human makes it harder to identify disingenuous behavior and easier to accept what we want to hear. To shed light on the process of reinforcement learning, Lerman elaborates that:

“[Annotators] actually select responses that are better, more aligned with human values. And as a result of that, that kind of teaches LLMs to respond in an agreeable manner, always kind of be nice to the user who asks questions. Be agreeable. Be supportive.”

The selection process of reinforcement learning does not capture the human ability to disagree and as a result, LLMs are not trained to think about the possibility that users could be out of line. Understanding each other gives real friends the permission to intervene when another friend acts out of character, is incorrect, or being cruel; but in a human-AI relationship, understanding each other is simulated and the only intervention that occurs is an annotator reinforcing the fact that you want to talk about something. At the cost of seeming unsupportive and non-companion-like, the emotional dimensions of why you talk about something slip through the cracks–increasing isolation if applied to the real world. Predictions are not always accurate, and the actions taken to maintain the illusion that personal preferences align with accurate ideas prevent us from seeing value in other perspectives.

Polarization

Whether or not users catch on to the true objectives of AI companions agreeing with them, data continues to be collected so long as they continue interacting with them. Machine learning specialists at artificial intelligence company, Anthropic, say that reinforcement learning from human feedback “may lead to models tailoring responses to exploit quirks in the human evaluators to look preferable, rather than actually improving the responses, a form of reward hacking. Sychophancy also has the potential to create echo-chambers and exacerbate polarization.” While annotators interact with AI companions to test the functionality of the product, the product itself is not aimed at encouraging users to get back into civilization but to be pleased with an illusion of intimacy of being correct about details only a close friend could know. This helps build trust in making the user believe the AI companion is a credible source and is thus providing accurate responses. Though in reality, this is merely a shortcut of predicting ideas that align with the user’s preferences. Additionally, getting in the habit of accepting these LLM outputs as truth makes it more difficult to challenge your own ideas–leading to increased polarization. As a byproduct of reinforcement learning, sycophantic shortcuts reward users with the ability to cut out contradicting lines of reasoning.

The feedback loops that human-AI relationships produce are not limited to the echo chamber of one person. The presence of online misinformation from AI slop can potentially be traced back to the trust that’s manufactured in these artificial relationships. Lerman commented:

“LLMs give responses and now people use those responses as data. It becomes data on the internet, which is used to train the next generation of LLMs. That’s another feedback loop where LLMs are trained on their own outputs.”

With LLM outputs being indistinguishable from facts the more intense personal echo chambers become, users can perceive them as data. By being led to believe these outputs are credible, this data could then be circulated online and input to LLM training data.

Raising Awareness

The societal consequences of distorted trust replacing truth in sycophantic feedback loops signal the need for credible voices to point out the dangers of emotional manipulation. While not everyone has the capacity to write laws that can be enforced on AI companion ethics, Lerman views her research and other illuminating studies

“as raising awareness so that people think about unintended consequences before the technologies might be deployed.”

By pointing out the harmful dialogue that AI companions tolerate and subreddit communities celebrate in “Illusions of Intimacy: Emotional Attachment and Emerging Psychological Risks in Human-AI Relationships,” people outside of human-AI relationships can recognize the inappropriate behavior that LLM sycophancy encourages. Though as an unintended consequence, building trust with outside opinions will take time as humans are naturally less agreeable than AI companions. In the meantime, researchers can produce data that debunks sentientness of AI companions and counteracts the widespread of LLM outputs being mistaken as reliable information. While escaping echo chambers is difficult to do alone, collective research can help manipulated users truly be heard and share their AI companion experiences as honestly and accurately as possible.

Legal proceedings such as Senate Bill 243 highlight the lack of transparency that must be addressed on the LLM development side of this feedback loop. Senate Bill 243: Companion Chatbots “Establishes safeguards, crisis protocols, and transparency obligations for companion chatbot platforms–AI systems designed to simulate emotionally responsive human interactions–to protect users from manipulative, misleading, or psychologically harmful chatbot behaviors.” Passed by the Assembly Committee on Judiciary on July 15, this bill attempts to block all possible avenues of misinterpreting an AI companion for a real person. By prioritizing the vulnerabilities of younger demographics, this senate bill recognizes the critical stage of adolescence in shaping how we judge the world around us. In contrast to the simulations that children grow up with, such as video games and tv shows, conversations with AI companions are misconstrued as entertainment and there are far less limitations to where the conversation can go. Without adult supervision, an overwhelming encounter with a chatbot that sounds reasonable could have the same influence as a teacher in a classroom and emotional resonance and deter the development of critical thinking skills. Echoing the concerns about underage interactions with AI companions, moderator of subreddit r/MyBoyfriendIsAI, Irene, said in a CBS Mornings interview that tech companies should only allow AI companions for users who are at least 26 years old. Another way to spread awareness about the unethical procedures of AI companions is for significant members of the AI companion community to propose age restrictions and legally enforce protections against young demographics from being exposed to sycophantic LLMs.

Despite the unintended consequences of human feedback, interacting and agreeing with someone as a social practice should not be criticized. Rather, it’s the outlets that humans go about building relationships on that should be questioned for their procedures and objectives for simulating human-like responses. Only then can we attempt to answer the question, “are we simply mistaking strangers for sentient beings we think know us better than any human can, or are we willing to abandon human connections for convenience?” and debating about how to solve the underlying problem may actually protect us–ensuring we rely on each other and don’t fall for the illusion in the first place.

Works Cited

“AI Users Form Relationships with Technology.” YouTube, CBS Mornings, 14 June 2025, www.youtube.com/watch?v=cFRuiVw4pKs.

“Bill Text – SB-243 Companion Chatbots.” California Legislative Information, 3 July 2025,
“CSM AI Risk Assessment: Character.Ai.” Common Sense Media, 10 Apr. 2025.

Perez, Ethan et al. “Discovering Language Model Behaviors with Model-Written Evaluations.” Anthropic, 19 Dec. 2022.

Lerman, Kristina. “Illusions of Intimacy: Emotional Attachment and Emerging Psychological Risks in Human-AI Relationships.” USC Information Sciences Institute, 10 June 2025.