Is it safe to use AI with patient information?

The honest answer to the title question is not yes or no: it is “it depends on what you verify.” No AI tool is safe or unsafe in the abstract; what can be is the combination of what a specific tool does with the data, what you agreed with your client, and how you use it. And one thing no tool changes: the person accountable for the clinical information is still you. This guide covers what data-protection law demands, what the WHO and APA frameworks propose, and the questions to put to any vendor — including us — before trusting it with a session.

Everything you record in a session is sensitive data

The starting point is not technical but legal. Data-protection regimes commonly treat health information as a specially protected category, and everything a psychotherapy practice produces qualifies: the clinical history, your notes, a session recording, a transcript, the presenting problem described in an email. The specifics vary by country, but the duties tend to rhyme. In Mexico, for example, the federal data-protection law in force since 2025 classifies present or future health status as sensitive personal data, requires the client’s express written consent to process it, obliges the practitioner to notify clients immediately of significant security breaches, and holds the practitioner accountable for making sure their privacy notice is honored by the third parties they engage — hiring a vendor does not export the responsibility. It also demands reasonable efforts to keep the processing period for sensitive data to the indispensable minimum: a useful yardstick for any tool that wants to retain session audio indefinitely. Whatever your jurisdiction, verify the rules where you practice; the questions below stay the same.

One structural idea from that law travels well: a vendor that processes data strictly on your behalf occupies a different legal position than one that uses the data for its own purposes — for instance, training its models. The moment its use stops being “on your behalf,” the legal analysis changes, usually not in your favor. That is why “is my data used to train your models?” is a legal question, not a technical one.

The US layer: business associates and BAAs

If you practice in the United States, HIPAA gives the vendor question a precise shape. Under HHS guidance, a company that creates, receives, maintains, or transmits protected health information on behalf of a covered entity is a business associate, and PHI may be disclosed to it only with “satisfactory assurances, in the form of a contract or other written arrangement” — the business associate agreement, or BAA. The cloud-computing guidance closes the loophole vendors sometimes hint at: even a “no-view” provider that stores only encrypted data and holds no decryption key is still a business associate — encryption alone does not settle the question. So the first question for any AI vendor becomes: will you sign a BAA? We are not asserting any product’s HIPAA status here, our own included; that is a claim to demand in writing from the vendor, not to read on a blog. BAAs are a US construct — elsewhere, the equivalent move is the written contract that keeps the vendor acting strictly on your behalf.

The international reference points

WHO: six principles and a warning aimed at those of us who document

The WHO published Ethics and governance of artificial intelligence for health in June 2021, with six consensus principles running from human autonomy to accountability. The first lands directly in your office: in health care it means “humans should remain in control of health-care systems and medical decisions; privacy and confidentiality should be protected, and patients must give valid informed consent through appropriate legal frameworks for data protection,” per the official release. The accountability principle adds that “it is the responsibility of stakeholders to ensure that they are used under appropriate conditions and by appropriately trained people.”

In January 2024 the WHO brought those principles to generative AI with its guidance on large multi-modal models (LMMs, the technology behind ChatGPT-style tools), carrying more than 40 recommendations. Two things matter here. First, among its five broad health applications it expressly lists clerical work, “such as documenting and summarizing patient visits within electronic health records”: your use case sits inside the WHO’s map. Second, its documented risks read like this article’s table of contents — false, inaccurate, biased, or incomplete statements; “automation bias,” where professionals overlook errors they would otherwise have caught; and cybersecurity risks that could endanger patient information.

APA: how to evaluate a tool, step by step

In October 2024, APA Services published a practical AI tool guide for practitioners in thirteen steps that end where they should: documenting your review and repeating it when policies change. Its questions are surgical. On data: “Take particular note of whether the company uses your user data to train the underlying AI model.” On consent: whether the company provides a sample form and requires an attestation that the client consented before the tool touches their information. The companion checklist asks the compliance question in terms that work anywhere — does the company attest the tool complies with “applicable data privacy laws and regulations in the jurisdiction in which you practice”? — plus, item by item: does the company encrypt personal data, where is data stored, how long is it retained, is user data used to train the underlying model, does it provide a BAA. And it closes with a disclaimer worth imitating: “The APA does not endorse any specific AI tools” — evaluating remains each practitioner’s job.

In July 2025 (its latest update) the APA added an ethical guidance document on AI in health service psychology — which states it does not represent official APA policy, though it draws on the Ethics Code. Three of its recommendations are immediately actionable: consider adding to your written informed consent “when, how, and the type of AI tool(s)” you use; tell clients they can opt out of certain AI-driven interventions; and know how client data are used, stored, or shared — with a clear exit: discontinue tools when security concerns arise. Peer-reviewed, mental-health-specific guidelines published in 2025 in the journal Healthcare add a simple principle: the confidentiality standards that bind you apply equally to your tools, and clients are told whenever AI is used in their treatment.

The questions to ask any vendor — including us

Distilled from all of the above, this is the minimum interrogation before a tool touches client data. Get the answers in writing: what is not in writing does not exist when something goes wrong.

What data does it process, and what does it keep? Audio, transcripts, notes, identifiers? For how long? Data-minimization duties give you the yardstick: the shortest retention that serves the purpose.
Is client data used to train its models? Ask — never assume, about any product. If yes: on what legal basis, with what de-identification, and with what genuine way to refuse?
Does it act only on your behalf? Does the contract confine the vendor to your purposes and exclude uses of its own? In the US: will it sign a BAA?
Can you delete? How is a client’s data erased when they withdraw permission, and what confirmation do you receive?
Who can access it? Which vendor staff see identifiable data, under what controls, and is access logged?
What happens after a breach? How fast are you notified, and in what detail? In many places the duty to inform affected clients is yours — without the vendor’s notice, you cannot meet it.
What security measures does it document? Do not settle for the word “encrypted” on a landing page: request the documentation and file it with your evaluation record, as the APA suggests.
Where does the human sit? Is the tool designed for you to review and approve every output, or to operate without you?

Two warnings. No vendor answer transfers your accountability — the frameworks above converge on that point from ethics and from law. And the list applies in full to gesell.ai: we are an AI vendor, and the only consistent position is that you ask us these same questions and weigh the answers with the same rigor you would apply to anyone else.

The honest limit: AI gets things wrong, and the signed note is yours

Even with the data chapter settled, the other risk remains: the content. Large language models hallucinate. A 2025 review in Healthcare Informatics Research describes them generating fluent, plausible-sounding content that is factually incorrect, unsupported by the source data, or entirely fabricated — and warns that using patient data for model training carries risks of memorization, leakage, and re-identification. A 2025 study in Communications Medicine planted a single fake clinical detail in prompts and measured how often models repeated or elaborated on it: between 50% and 82% of the time, with mitigation prompting lowering the average rate from 66% to 44% — better, but nowhere near zero. Read that number carefully: it comes from an adversarial clinical-decision-support scenario, not from routine note drafting, so it is not “your AI notes are mostly wrong.” The practical conclusion does travel, though: mitigation reduces error without eliminating it, so human review is not an optional step in the workflow. It is the workflow.

That conclusion has a legal echo wherever you practice: clinical notes are attributed and signed. In Mexico, for example, the clinical-record norm requires every note to carry the full name and signature of its author. The note a system drafted and you signed is, legally and ethically, your note — errors included. The APA’s 2025 guidance frames it as human oversight: “AI should augment, not replace, human decision-making,” and psychologists remain responsible for final decisions. In one sentence: AI drafts; you review, correct, and sign. And reviewing a draft well is the same discipline as reviewing your own session note: section by section, against what actually happened.

One piece no vendor contract can resolve: your client. Running an AI tool over session content — audio, transcript, or written record — is processing of sensitive health information, and the converging standard across the frameworks above is express, documented consent, obtained before you start and never assumed by default; in Mexico, for instance, the law demands it expressly and in writing for health data. The APA’s 2025 guidance says where to put it: in the written informed consent, specifying when, how, and what type of tool you use, with a genuine option to decline. Our informed consent guide covers that document and, above all, that conversation; the short version: the client knows beforehand, consents in writing, and can change their mind later.

“I use a tool that helps me draft my notes from my session record. Before using it with you, I will tell you what it processes, where that is stored, and how to ask for deletion. If you would rather I not use it, I will not — and your treatment does not change.”

So, is it safe?

Asked badly, the question has no answer: “AI” as a whole is neither safe nor unsafe. Asked well, it becomes three questions that can be answered: do I know what this specific tool does with the data, in writing? Did my client expressly consent? Do I review every output before it touches the record? With three yeses, AI can hold as defensible a place as any other part of your practice; with one no, the problem is not AI — it is that gap. gesell.ai is built around the third yes: it drafts your notes in SOAP, DAP, BIRP, or GIRP format from your session records and keeps each client’s chart structured, always with your review in the loop: you adjust, approve, and own every draft. The other two yeses — the written answers and your client’s consent — ask us for them exactly as you would ask anyone else.