I am very much sympathetic to this post and really like the paper too. But I stumbled at this: "Will these chatbots be safer in two years? Yes. Safe enough to stand in as a therapist? No." Sycophancy is a big issue, yes, but I am not convinced it is unsolvable. As a psychologist, I also worry that you are elevating human-delivered therapy beyond what is warranted. Research suggests that the effectiveness of therapy is modest on average. It appears like you've had a good experience with your therapist, but (a) despite the research you presented here, many people also report having a good experience with their AI therapists and (b) many people find human-therapy ineffective or disappointing too. So, while I beleive your paper is important, you haven't yet convinced me that LLMs could not be safe in the future or that they are ineffective right now.
Thanks for the comment, and thanks for reading the paper. I agree - they will be safer in two years. I am also not an dystopian AI skeptic. There is a very recent paper from Anthropic that talks about persona vectors, which may be the start to countering unhinged advice and sycophancy (https://www.anthropic.com/research/persona-vectors). (In this post, I also was already at 2500 words and decided to not work through the opportunities of AI to augment therapy to keep this from not being a mini-paper 😂
Let's do a thought experiment where, in 2 years, the egregious safety concerns we found about delusions and suicidal ideation disappear. There are still fundamental questions to address: a) what are the hardest safety issues that will still remain resistant and need scrutiny? b) What does human oversight look like for effective, patient-centered care? Who is responsible for "bad" therapy? Right now we have recourse for bad doctors, but what about harmful AI therapists? c) what parts of the human relationship are not able to be delegated to the chatbot? d) How do we evaluate "a good experience" from "an effective therapeutic experience"? The ways we measure good AI/UX experiences is vastly different than what good therapy needs.
Yes to all of this! So many more questions to address to get this to be safe (or at least as safe as with human-therapists). I also wonder if we need to think of AI as supplements, in between therapy, to work on things between sessions. Anyhow, I'm glad we connected. I might be putting together a small conference on relational AI (AI as friend, therapist, or coach) next year; so don't be surprised if you receive an email from me in the coming months!
Totally agree re: supplementation and addressing actual problems with medicine. CS folks sometimes jump ahead to having AI do these fancy activities - diagnosis, being a therapist, etc - but I wonder about what the actual problems are with care.
I had a really impactful conversation with a colleague after talking with them about the AI stuff - "why are you focusing on diagnosis? that's not the actual problem we have". It made me think of my HCI research training, and how, perhaps, we should be asking therapists and psychiatrists what their issues are that AI could solve. Notetaking? Caseload management? Billing? Compassion fatigue and burnout management? Training exercises? I don't think it's their actual care in-session that's the issue.
I am very much sympathetic to this post and really like the paper too. But I stumbled at this: "Will these chatbots be safer in two years? Yes. Safe enough to stand in as a therapist? No." Sycophancy is a big issue, yes, but I am not convinced it is unsolvable. As a psychologist, I also worry that you are elevating human-delivered therapy beyond what is warranted. Research suggests that the effectiveness of therapy is modest on average. It appears like you've had a good experience with your therapist, but (a) despite the research you presented here, many people also report having a good experience with their AI therapists and (b) many people find human-therapy ineffective or disappointing too. So, while I beleive your paper is important, you haven't yet convinced me that LLMs could not be safe in the future or that they are ineffective right now.
Thanks for the comment, and thanks for reading the paper. I agree - they will be safer in two years. I am also not an dystopian AI skeptic. There is a very recent paper from Anthropic that talks about persona vectors, which may be the start to countering unhinged advice and sycophancy (https://www.anthropic.com/research/persona-vectors). (In this post, I also was already at 2500 words and decided to not work through the opportunities of AI to augment therapy to keep this from not being a mini-paper 😂
Let's do a thought experiment where, in 2 years, the egregious safety concerns we found about delusions and suicidal ideation disappear. There are still fundamental questions to address: a) what are the hardest safety issues that will still remain resistant and need scrutiny? b) What does human oversight look like for effective, patient-centered care? Who is responsible for "bad" therapy? Right now we have recourse for bad doctors, but what about harmful AI therapists? c) what parts of the human relationship are not able to be delegated to the chatbot? d) How do we evaluate "a good experience" from "an effective therapeutic experience"? The ways we measure good AI/UX experiences is vastly different than what good therapy needs.
Yes to all of this! So many more questions to address to get this to be safe (or at least as safe as with human-therapists). I also wonder if we need to think of AI as supplements, in between therapy, to work on things between sessions. Anyhow, I'm glad we connected. I might be putting together a small conference on relational AI (AI as friend, therapist, or coach) next year; so don't be surprised if you receive an email from me in the coming months!
Totally agree re: supplementation and addressing actual problems with medicine. CS folks sometimes jump ahead to having AI do these fancy activities - diagnosis, being a therapist, etc - but I wonder about what the actual problems are with care.
I had a really impactful conversation with a colleague after talking with them about the AI stuff - "why are you focusing on diagnosis? that's not the actual problem we have". It made me think of my HCI research training, and how, perhaps, we should be asking therapists and psychiatrists what their issues are that AI could solve. Notetaking? Caseload management? Billing? Compassion fatigue and burnout management? Training exercises? I don't think it's their actual care in-session that's the issue.