At a Glance
- AVA, Alaska’s AI probate chatbot, now over a year into development instead of the planned three months.
- The team has battled hallucinations, tone issues, and accuracy gaps before launch.
- Launch is slated for late January after a cost-effective 20-query test run.
Why it matters: It illustrates the challenges public agencies face when deploying generative AI in high-stakes legal contexts.
The Alaska Court System has spent more than a year developing a generative-AI chatbot called the Alaska Virtual Assistant (AVA) to help residents navigate probate. The project was intended to last three months, but delays from rigorous due diligence pushed it beyond a year.
From Promise to Reality
Aubrie Souza said:
> “AVA was supposed to be a three-month project. We are now at well over a year and three months, but that’s all because of the due diligence that was required to get it right.”
Balancing Empathy and Precision
Early versions of AVA were too empathetic, prompting users who were grieving to feel annoyed. Aubrie Souza noted:
> “Through our user testing, everyone said, ‘I’m tired of everybody in my life telling me that they’re sorry for my loss.'”
The team removed condolence messages to keep the chatbot focused on information.
Hallucinations and Legal Accuracy
The team struggled with hallucinations, where the chatbot would confidently give false information. Jeannie Sato explained:
> “We had trouble with hallucinations, regardless of the model, where the chatbot was not supposed to actually use anything outside of its knowledge base. For example, when we asked it, ‘Where do I get legal help?’ it would tell you, ‘There’s a law school in Alaska, and so look at the alumni network.’ But there is no law school in Alaska.”
Law professor and AI developer Tom Martin worked to restrict the chatbot’s references to probate documents only.
Cost and Maintenance
Martin highlighted the low cost of running AVA: Tom Martin said:
> “Under one technical setup, 20 AVA queries would cost only about 11 cents.”
He added that the system will need regular checks and updates as models evolve.
Looking Ahead
Stacey Marz said:
> “We did shift our goals on this project a little bit. We wanted to replicate what our human facilitators at the self-help center are able to share with people. But we’re not confident that the bots can work in that fashion, because of the issues with some inaccuracies and some incompleteness. But maybe with increasing model updates, that will change, and the accuracy levels will go up and the completeness will go up.”
The team refined the test set from 91 questions to 16, focusing on common and complex scenarios.
| Test Set | Questions | Purpose |
|---|---|---|
| Initial | 91 | Full coverage |
| Refined | 16 | Focused, high-impact |
| Queries | Cost |
|---|---|
| 20 | $0.11 |
Key Takeaways
- AVA’s development exceeded its original three-month timeline due to accuracy concerns.
- The chatbot faced tone and hallucination issues that required extensive user testing and data restriction.
- Each 20-query run costs roughly 11 cents, but ongoing maintenance will be needed as models change.
AVAs launch in late January will mark a milestone for Alaska’s use of AI in the justice system, but the experience underscores the need for careful oversight and continual improvement.
