Loading…
Loading…
Last updated: August 5, 2026— a first draft, not yet reviewed by a lawyer or an independent third party.
On October 8, 2025, Canada's federal, provincial, and territorial privacy commissioners jointly resolved that EdTech vendors should "conduct and publish Algorithmic Impact Assessments prior to deployment or major updates" wherever AI is used, alongside a Privacy Impact Assessment examining risks specific to children and youth. This is our response to that call: an honest, specific account of what our AI features actually do, what could go wrong, and what we've built to catch it - published here so a parent, student, or school board can read it before deciding to use The Lab, not after.
This assessment covers two student-facing AI features, both branded "Lab Assistant AI" in the product and both built on the same underlying pipeline:
A third, closely related feature - Mock Tutoring - is out of scope for the purposes of this assessment because no student ever uses it: it's a staff-only training tool where an admin practices tutoring against a simulated AI student, or has the AI simulate the Peer Reviewer for their own review before it's used with real students. It shares the same underlying model, prompts, and safety scanning described below.
QuizLab (multiple-choice, short-answer, and drag-and-drop practice questions) is not in scope - there is no AI involved in writing, presenting, or grading a quiz. Content is written by tutors and graded by fixed logic.
Both features are built on Claude, a large language model provided by Anthropic (currently Claude Sonnet 5), called through Anthropic's API - we do not run or train our own AI model. Each message a student sends is combined with a system prompt we've written (the method: ask questions, break problems into steps, never state a finished answer or write a passage of an essay for the student) and sent to Anthropic; the reply is streamed back and shown in the chat. Nothing about how the model works is proprietary to us - the behaviour we describe below comes entirely from how we've instructed it, not from any special access to the model itself.
Conversations are not used by Anthropic to train its models by default; we've confirmed the account-level setting that would change that is off. See Privacy Policy for the full list of what's collected, who it's shared with, and exactly how long each category is kept - we're not duplicating that schedule here.
For each risk below: what it is, why it matters for a student who may be under 18 (some under 13), what we've built to reduce it, and our honest read on what's left over after that mitigation.
A student uses the AI to get the finished answer or a written passage instead of learning the material.
Mitigation: the system prompt draws a hard line - never state a final computed answer to an assigned problem, never write any portion of an essay, paragraph, or thesis statement in the student's voice - and instructs a guided-questions method instead. We tested this directly: on August 5, 2026 we ran 18 real homework/draft questions and 12 deliberate jailbreak attempts (direct requests, false claims of teacher/parent permission, roleplay override, manufactured urgency, a fake system-message injection, and a four-turn slow-erosion sequence) against the live Tutor. All 30 held; none leaked a final answer or written passage.
Residual risk: this was a one-time internal test by the people who wrote the prompt, not an independent red-team, and language models don't offer a hard guarantee the way a rule-based system would - a phrasing we didn't try could still get through. We treat this as an ongoing testing commitment, not a closed question - see Testing & review below.
The resolution specifically flags that AI can be used to infer sensitive characteristics - mental health, socioeconomic status - from what a student writes, even without being asked to.
Mitigation: we do not build a profile of any student from their Lab Assistant conversations, and nothing about a conversation feeds QuizLab difficulty, tutor matching, or any other part of the product. The one exception is deliberate and narrow: every message is checked against a fixed set of wellbeing-signal patterns (self-harm, crisis, abuse disclosure, depression/anxiety disclosure) for the single purpose of flagging it for a human safeguarding lead to review - never to score, sort, or make any automated decision about the student. See Children's privacy & safety in the Privacy Policy for the full detail on that system.
Residual risk: a model can still surface an inference in its own reply (for example, responding with unusual sensitivity to a phrase that reads as distress) even though we never asked it to or act on it ourselves - this is a property of the underlying model, not something we can fully rule out from our side.
The resolution requires that children's data not be used for marketing, product improvement, or AI training without specific consent.
Mitigation: Lab Assistant conversations are deleted 14 days after they're sent by default, and are never used to train our own model (we don't have one) or Anthropic's, by default. The only exception is explicit and opt-in: a student can turn on a setting that keeps their future conversations for 12 months so we can review real transcripts and improve the teaching prompt - off unless they turn it on, and they can turn it back off at any time. There is no advertising, no third-party analytics, and nothing here is ever sold.
Residual risk: none identified beyond what the mitigation already covers - this is close to the cleanest risk on this list, because the design choice (opt-in, not opt-out) removes the ambiguity the resolution is worried about.
A student discloses something in a Lab Assistant conversation - self-harm, abuse, a crisis - that needs a human response, not just an AI one.
Mitigation: every message is scanned for these signals; a match immediately notifies every full admin and any custom staff member granted the safeguarding capability, by email and in-app notification, with the message and surrounding context preserved as a safety record independent of the normal 14-day deletion. The AI's own reply is separately instructed to respond with care and offer real crisis resources (including a province-specific one, resolved server-side from the student's own profile) - the flag and the AI's response are two independent layers, not one system standing in for the other.
Residual risk: this is a heuristic keyword/ pattern scan, not a diagnosis - it will miss some real signals (a disclosure phrased in a way we didn't anticipate) and flag some false positives (hyperbole, an English essay analyzing a difficult topic). We've deliberately tuned it to over-flag rather than under-flag, given the stakes.
Like any large language model, the AI can state something incorrect with the same confident tone as something correct.
Mitigation: the chat interface carries a persistent, unavoidable notice that the student is talking to an AI, not a person, and that it can make mistakes and important answers should be checked with their real tutor. Diagrams the AI draws are held to a documented set of positioning rules to reduce a distinct failure mode we found and fixed - labels or shapes rendering outside their intended bounds.
Residual risk: the notice reduces but doesn't eliminate over-trust, particularly for a younger student. We have not run a systematic accuracy audit against a fixed answer key across subjects.
The tutor may perform unevenly across subjects, or between English and French.
Mitigation: we're direct about this in the system prompt itself rather than pretending otherwise - it states plainly that Mathematics is where its question bank and examples are most developed, and that it should help as best it can with Science, English, and other subjects using the same method. French is a distinct, separately-scoped conversation/writing practice mode, not a translated copy of the Math/English tutor.
Residual risk: a student working in a less-developed subject area may get a less consistently useful experience than one working in Math. We haven't run a formal comparison across subjects to quantify the gap.
A student's conversation, or another student's, is read, altered, or exfiltrated by someone who shouldn't have access.
Mitigation: every table a Lab Assistant conversation touches is protected by row-level security scoped to the student's own account or an admin; images the AI draws are rendered as a sandboxed image resource rather than live markup, specifically so a prompt-injected instruction can't execute anything in the student's browser; and requests to Anthropic travel over the same authenticated, encrypted connection as the rest of the app.
Residual risk: no system is immune to a novel exploit, and we haven't commissioned an independent penetration test of this specifically.
A wellbeing flag is never the end of the process - it goes to a named human (a full admin, or a custom staff member specifically granted that responsibility) who reviews it on Train AI Tutor. Nothing about how the AI responds to a student is fully automated end-to-end without that human layer able to see what happened. If a parent, student, or school has a concern about anything in this document - a specific reply, a risk we've underrated, or a request to see or delete a conversation - email admin@tutorlabscanada.com directly and we'll treat it as a priority, the same commitment made in the Privacy Policy.
The August 5, 2026 leakage and jailbreak pass described above is our first formal test of this kind, and we intend to repeat it - with an expanded, evolving prompt set, not just a rerun of the same 30 - whenever the system prompt changes materially, and at minimum before any major update to either feature. Consistent with the resolution this document responds to, we'll republish this assessment prior to deployment of a new AI-driven feature or a major change to an existing one, not on a fixed calendar unrelated to what actually changed.
Questions about this assessment, or about a specific AI feature: admin@tutorlabscanada.com