Vector Institute and Kids Help Phone (KHP) researchers have co-created the Frontline Assistant: Issue Identification and Recommendation (FAIIR) model. This model automatically identifies and categorizes key issues discussed during crisis support conversations with youth, enhancing KHP frontline staff’s delivery of human-centred care. Their work was recently published in a paper in the journal npj Digital Medicine.

The global youth mental health crisis presents an urgent challenge: one in seven young people aged 10 to 19 experiences mental health challenges. Suicide is the second leading cause of death among youth and young adults (15-34 years) in Canada. By the time youth in Canada reach the age of 25, one in five will face mental health struggles. Despite how common these challenges are, finding the right help can feel like navigating a maze.

Since the start of 2020, Kids Help Phone has had more than 22 million interactions (22,124,522) with service users across Canada. This growing need became especially clear during the COVID-19 pandemic, when KHP, Canada’s leading youth e-mental health organization, saw demand for its text support services increase by 51 per cent.  This heightened demand underscores the necessity of expanding and assisting the team of Crisis Responders (CRs), who may face a significant cognitive burden during these conversations, managing emotionally stressed individuals in potentially life-critical situations.

The complexity of these conversations, combined with the need for accurate post-conversation documentation and issue identification, highlighted the need for innovative solutions. In collaboration with KHP, Vector Institute researchers identified an opportunity to leverage advanced AI techniques to support CRs while maintaining the quality and empathy of every human interaction.

Technical innovation

Model architecture

FAIIR employs an ensemble of three Longformer models, specifically chosen for their ability to process lengthy crisis support conversations. The model utilizes domain adaptation through specialized pre-training using masked language modelling on mental health conversations. This architecture enables:

• Processing of conversations up to 2,000 tokens (covering 94.4 per cent of all interactions). Tokens in this context refer to discrete units of text that typically represent words, sub-words, characters, or punctuation obtained after breaking a sequence of text down so that it can be processed by a model. For reference, one token corresponds to roughly ¾ of a word for English text on average.
• Multi-label classification across 19 predefined mental health issues and the ability to identify additional keywords in conversations.
• Efficient handling of complex, multi-turn dialogues

Performance and validation

The best-performing architecture demonstrates exceptional performance across key metrics:

Metric	Description	Performance
Average AUC ROC	Average ability of the model to distinguish between issue tags across all categories	94%
Average Recall Score	Average proportion of relevant issue tags identified as relevant	81%
Average F1-score	Average of the combined measure of precision (proportion of correct issue tags out of all issue tags  deemed relevant) and recall	64%
Silent Testing Drop	Drop in performance between initial model development and silent testing, indicating the model’s real-world generalizability	<2%

Particularly noteworthy is FAIIR’s performance on critical mental health issues:

Issue category	F1-Score
Depression	0.75
Suicide	0.73
Self Harm	0.69
Anxiety/Stress	0.69

Methodology

Enhanced Crisis Response

FAIIR has the potential to significantly improve crisis support operations by reducing the cognitive load on CRs through automated issue identification. The model achieves 90.9 per cent agreement with CR predictions within the prospective silent testing phase, while streamlining post-conversation documentation, enabling more efficient and accurate responses to youth mental health crises.

Demographic Fairness

A crucial aspect of FAIIR’s success is its consistent performance across demographic groups, with minimal F1-score variations across gender (±0.023), sexual orientation (±0.010), cultural identity (±0.018), and ethnicity (±0.024). This consistency ensures equitable support for diverse youth populations seeking support.

Technical Challenges and Solutions

Data imbalance solutions

The FAIIR development team addressed significant data imbalance challenges, where some issues appeared in over 244,000 conversations while others occurred in as few as 2,800. Through balanced sampling techniques, threshold optimization, and custom loss functions, FAIIR maintains robust performance across both common and rare issue categories.

Processing long conversations

The model efficiently handles lengthy crisis conversations through specialized pre-training and optimized batch processing, maintaining high performance while managing the complexities of emotional and multi-issue discussions.

Future directions

The research team envisions several key initiatives to advance FAIIR’s capabilities:

• Implementation of FAIIR into frontline workflows 
• Integration of generative language models for enhanced context understanding
• Development of dynamic issue tag prediction for emerging youth mental health trends
• Enhanced natural keyword extraction for better pattern recognition
• Exploration of multi-modal data integration and real-time intervention strategies

Conclusion

FAIIR represents a significant advancement in AI-assisted mental health support, demonstrating how leading-edge AI research can address critical societal challenges. This collaboration between Vector Institute and KHP explores how AI can support CRs with their vital work of helping Canada’s young people when they need it most.