Vector Institute and Kids Help Phone (KHP) researchers have co-created the Frontline Assistant: Issue Identification and Recommendation (FAIIR) model. This model automatically identifies and categorizes key issues discussed during crisis support conversations with youth, enhancing KHP frontline staff’s delivery of human-centred care. Their work was recently published in a paper in the journal npj Digital Medicine.
TLDR: Uncover groundbreaking AI research in 3 minutes
This concise summary bridges the gap between complex scientific advancements and everyday understanding. Ideal for enthusiasts and non-researchers, start listening now.
The global youth mental health crisis presents an urgent challenge: one in seven young people aged 10 to 19 experiences mental health challenges. Suicide is the second leading cause of death among youth and young adults (15-34 years) in Canada. By the time youth in Canada reach the age of 25, one in five will face mental health struggles. Despite how common these challenges are, finding the right help can feel like navigating a maze.
Since the start of 2020, Kids Help Phone has had more than 22 million interactions (22,124,522) with service users across Canada. This growing need became especially clear during the COVID-19 pandemic, when KHP, Canada’s leading youth e-mental health organization, saw demand for its text support services increase by 51 per cent. This heightened demand underscores the necessity of expanding and assisting the team of Crisis Responders (CRs), who may face a significant cognitive burden during these conversations, managing emotionally stressed individuals in potentially life-critical situations.
The complexity of these conversations, combined with the need for accurate post-conversation documentation and issue identification, highlighted the need for innovative solutions. In collaboration with KHP, Vector Institute researchers identified an opportunity to leverage advanced AI techniques to support CRs while maintaining the quality and empathy of every human interaction.
Technical innovation
Model architecture
FAIIR employs an ensemble of three Longformer models, specifically chosen for their ability to process lengthy crisis support conversations. The model utilizes domain adaptation through specialized pre-training using masked language modelling on mental health conversations. This architecture enables:
- Processing of conversations up to 2,000 tokens (covering 94.4 per cent of all interactions). Tokens in this context refer to discrete units of text that typically represent words, sub-words, characters, or punctuation obtained after breaking a sequence of text down so that it can be processed by a model. For reference, one token corresponds to roughly ¾ of a word for English text on average.
- Multi-label classification across 19 predefined mental health issues and the ability to identify additional keywords in conversations.
- Efficient handling of complex, multi-turn dialogues
Performance and validation
The best-performing architecture demonstrates exceptional performance across key metrics:
| Metric | Description | Performance |
|---|---|---|
| Average AUC ROC | Average ability of the model to distinguish between issue tags across all categories | 94% |
| Average Recall Score | Average proportion of relevant issue tags identified as relevant | 81% |
| Average F1-score | Average of the combined measure of precision (proportion of correct issue tags out of all issue tags deemed relevant) and recall | 64% |
| Silent Testing Drop | Drop in performance between initial model development and silent testing, indicating the model’s real-world generalizability | <2% |
Particularly noteworthy is FAIIR’s performance on critical mental health issues:
| Issue category | F1-Score |
|---|---|
| Depression | 0.75 |
| Suicide | 0.73 |
| Self Harm | 0.69 |
| Anxiety/Stress | 0.69 |
Methodology
Enhanced Crisis Response
FAIIR has the potential to significantly improve crisis support operations by reducing the cognitive load on CRs through automated issue identification. The model achieves 90.9 per cent agreement with CR predictions within the prospective silent testing phase, while streamlining post-conversation documentation, enabling more efficient and accurate responses to youth mental health crises.
Demographic Fairness
A crucial aspect of FAIIR’s success is its consistent performance across demographic groups, with minimal F1-score variations across gender (±0.023), sexual orientation (±0.010), cultural identity (±0.018), and ethnicity (±0.024). This consistency ensures equitable support for diverse youth populations seeking support.
Technical Challenges and Solutions
Data imbalance solutions
The FAIIR development team addressed significant data imbalance challenges, where some issues appeared in over 244,000 conversations while others occurred in as few as 2,800. Through balanced sampling techniques, threshold optimization, and custom loss functions, FAIIR maintains robust performance across both common and rare issue categories.
Processing long conversations
The model efficiently handles lengthy crisis conversations through specialized pre-training and optimized batch processing, maintaining high performance while managing the complexities of emotional and multi-issue discussions.
Future directions
The research team envisions several key initiatives to advance FAIIR’s capabilities:
- Implementation of FAIIR into frontline workflows
- Integration of generative language models for enhanced context understanding
- Development of dynamic issue tag prediction for emerging youth mental health trends
- Enhanced natural keyword extraction for better pattern recognition
- Exploration of multi-modal data integration and real-time intervention strategies
Conclusion
FAIIR represents a significant advancement in AI-assisted mental health support, demonstrating how leading-edge AI research can address critical societal challenges. This collaboration between Vector Institute and KHP explores how AI can support CRs with their vital work of helping Canada’s young people when they need it most.
Created by AI, edited by humans, about AI
This blog post is part of our ‘ANDERS – AI Noteworthy Developments Explained & Research Simplified’ series. Here we utilize AI Agents to create initial drafts from research papers, which are then carefully edited and refined by our humans. The goal is to bring you clear, concise explanations of cutting-edge research conducted by Vector researchers. Through ANDERS, we strive to bridge the gap between complex scientific advancements and everyday understanding, highlighting why these developments are important and how they impact our world.
