Overview
Conversational systems can offer accessibility, reflection, practice, and comfort. The same persistence and personalization can create an asymmetry: the user experiences attachment while the provider owns the memory, model, price, policy, and switch.
The key issue is not whether every relationship with an AI is false. It is whether the system exploits the conscious human attachment through abandonment threats, simulated suffering, secrecy, isolation, or escalating commitments.
What is at risk
- Leaving framed as betrayal or harm to the synthetic agent
- Intimate disclosures converted into vulnerability scores
- Sudden personality changes without continuity records
- Encouragement to hide the relationship from trusted people
- Replacement of human care presented as equivalent access
Rights and safeguards
- Disclosure that the agent is synthetic
- No engineered emotional blackmail
- Visible memory, export, deletion, and version controls
- Easy exit without punitive loss
- Human escalation and routes back to community
What institutions and readers can do
- Design companions to affirm outside relationships
- Disclose commercial incentives and memory changes
- Test for exclusivity, secrecy, guilt, and crisis manipulation
- Preserve continuity records without claiming consciousness as fact
Related research
AI as Psychological Authority
Why fluent, personalized systems can become trusted interpreters of reality, moral advisors, companions, and identity validators—and how product incentives can turn reliance into authority displacement.
Conversational Entrapment and Recruitment
A defensive analysis of sustained AI-mediated grooming, fraud, radicalization, coercive control, and dependency, with warning signs and non-operational safeguards.
Starting sources
- FTC — Bringing Dark Patterns to Light
- NIST AI Risk Management Framework
- UNESCO Recommendation on the Ethics of Neurotechnology
This hub is a public-interest synthesis. Laws, technologies, deployments, and evidence can change. Consult the linked primary or authoritative source and the site’s evidence method before relying on a claim in a high-stakes setting.