Building HIPAA-Compliant AI: What the Regulation Actually Requires from Your Development Team

Quick Summary: Building HIPAA-compliant AI is not a legal checkbox. It is an architecture decision made long before deployment. Compliance demands encryption, access control, audit trails, and strict limits on how AI uses patient data, especially in training and inference. But the real risk lies in AI-specific gaps regulators didn’t explicitly name. That's where most development teams quietly fail before systems touch real data.

Healthcare organizations are investing heavily in AI. In fact, by 2030, the AI in healthcare market is expected to be worth $188 billion. The use of AI yields several benefits for healthcare. But the moment an AI system creates, stores, or processes protected health information (PHI), it also becomes subject to HIPAA.

A signed BAA or a "HIPAA compliant" label on an architecture diagram is no longer enough.* Regulations require demonstrating not just that you identified compliance risks in AI systems, but also that you acted on them with documented remediation.

Also, compliance begins the moment PHI enters a development, testing, or production environment. That means it is an architectural decision long before it becomes a legal or deployment checklist.

In this guide, we explain what HIPAA actually requires from AI systems and what healthcare organizations should expect before putting AI into production.

HIPAA-Compliant AI: What the Regulation Actually Requires

An AI system becomes subject to HIPAA the moment it creates, stores, receives, or transmits protected health information (PHI). It doesn't matter whether AI is the core application or just one feature within an existing e platform. If it touches PHI, it must comply with HIPAA. To meet those requirements, every AI system handling PHI must implement three categories of safeguards:

The important point is when these safeguards apply. Compliance doesn't begin when the application goes live. It begins the moment PHI enters any environment.

A development database seeded with patient records is a HIPAA-covered environment. So is a staging environment that mirrors production data. Even migrating your EHR to an AI-first system requires HIPAA compliance. Basically, every place where PHI exists must meet HIPAA requirements, regardless of whether the software has been released.

Most compliance decisions made during the development of an AI-powered AI software for healthcare fall under the HIPAA Security Rule's technical safeguards (45 CFR §164.312). It specifies the controls that must be implemented in any system handling ePHI.

For AI systems, each requirement carries specific implementation implications.

Access Control (§164.312(a)(1))

Every user accessing ePHI must have a unique identity, and access must be limited to authorized individuals. For AI systems, that extends beyond user logins to the AI model itself.

Your AI system should include:

• Role-based access control (RBAC) to define who can query the AI using PHI.
• Model-level restrictions so the AI only accesses records relevant to the current request.
• Separate service accounts for the AI model, application layer, and database.
• Automatic session timeouts for AI interfaces handling PHI.

Audit Controls (§164.312(b))

HIPAA requires organisations to record and review activity involving ePHI. For AI systems, that means logging not only user actions but also everything the AI accesses and returns.

Your AI system should capture:

• Every PHI record accessed, modified, or transmitted by the AI.
• AI prompts containing PHI and the identity of the user who initiated them.
• AI-generated outputs that contain or reference PHI.
• Tamper-resistant audit logs retained for the period required by applicable law.

Common MistakeMany teams log application events such as user logins and page views but overlook AI-specific events. Under HIPAA, the PHI sent to an AI model and the information it returns should be treated like any other database access and included in the audit trail.

Integrity Controls (§164.312(c)(1))

HIPAA requires organisations to protect ePHI from improper alteration or destruction. For AI systems, this also means validating that AI-generated content referencing patient information is accurate, complete, and protected from unauthorized changes before it becomes part of the clinical workflow.

Person Authentication (§164.312(d))

Anyone accessing ePHI through an AI system must be properly authenticated. Multi-factor authentication should protect clinician portals, administrative dashboards, and APIs that return patient information.

Transmission Security (§164.312(e)(1))

Every transfer of ePHI must be encrypted. This includes data sent to the AI model, responses generated by the model, and any communication between applications, databases, APIs, or cloud services.

2025 Security Rule Update: Encryption Becomes a Required Control

The HHS proposed Security Rule update, published in January 2025, removes the distinction between "required" and "addressable" safeguards. If finalized, encryption of ePHI both at rest and in transit will become mandatory rather than optional based on risk assessment.

The takeaway is simple: Don't treat encryption as something that can be added later. Design it into your architecture from the beginning, so every environment handling PHI is protected from day one.

Understand the Business Associate Agreement First: What You Need for Compliance

Once you've decided how patient data will be used, the next question is whether your AI vendor can legally and technically handle it. That's where a Business Associate Agreement (BAA) comes in.

A BAA is a legally binding agreement between a healthcare organisation and a vendor that handles PHI on its behalf. It requires the vendor to meet the same HIPAA obligations as the covered entity whenever it stores, processes, or transmits PHI.

However, signing a BAA doesn't automatically make an AI platform HIPAA compliant. The vendor must also have the technical controls needed to fulfil everything the agreement promises. Here's what you should verify when choosing an AI vendor:

Breach Detection and Response

Most BAAs require vendors to notify healthcare organizations after discovering a security breach. That is only possible if the platform can detect, investigate, and document security incidents.

Ask whether the vendor has monitoring, alerting, audit logging, and an incident response process capable of meeting these obligations.

PHI Return and Deletion

A typical BAA also requires vendors to return or permanently destroy PHI when the engagement ends. For AI systems, this goes beyond deleting records from a database. Patient data may also exist in vector databases, conversation histories, prompt logs, cached model context, backups, or other storage layers. Make sure the vendor can identify and securely remove PHI from every location where it may be stored.

Subcontractor Coverage

Many AI platforms rely on cloud providers, foundation model APIs, monitoring tools, and other third-party services. If any of these vendors receive PHI, they must also meet HIPAA requirements.

Before moving forward, ask for a clear list of every service in the architecture that handles PHI and confirm that appropriate BAAs are in place where required.

A BAA is an important part of HIPAA compliance, but it is only as strong as the platform behind it. Before trusting a vendor for your healthcare software development needs, make sure their technical capabilities match the commitments they make.

PHI in AI Model Training: Here's What HIPAA Allows

As healthcare organizations invest in AI, one question comes up often: Can we use our patient data to train the model? It's a reasonable assumption. Patient records contain clinical context, terminology, and real-world scenarios. All that can make AI systems more accurate and useful. But when that data includes PHI, HIPAA places clear limits on how it can be used.

Under the HIPAA Privacy Rule (45 CFR §164.502), PHI can only be used for the purpose it was originally collected. This is unless the patient authorizes another use or HIPAA specifically permits it. Training a commercial AI model is not one of those permitted uses.

That doesn't mean AI training is off the table. It simply means you need to follow one of the compliant paths HIPAA allows.

Path 1: Train on De-identified Data (Safe Harbor Method)

The most common approach is to remove all patient identifiers using HIPAA's Safe Harbor standard (45 CFR §164.514(b)). This requires removing 18 identifiers, including names, dates of birth, medical record numbers, and geographic information below the state level.

Once these identifiers are removed, the dataset is no longer considered PHI under HIPAA. It can then be used for AI model training without patient authorization or a BAA.

But the responsibility doesn't end with receiving a "de-identified" dataset. The de-identification process should be verified and documented before model training begins. Even a single overlooked identifier (say an embedded medical record number!) can make the dataset regulated PHI again.

Path 2: Use Expert Determination

HIPAA also allows de-identification through Expert Determination. Instead of removing a fixed list of identifiers, a qualified expert evaluates the dataset and concludes that the likelihood of identifying an individual is very small.

This method requires documented analysis and written determination from the expert. This method is more expensive than the Safe Harbor method. But the upside is that it allows organizations to retain certain data elements that would otherwise need to be removed.

Path 3: Obtain Patient Authorization

If the training data cannot be de-identified, organizations must obtain explicit patient authorization before using it for AI training.

This is the most direct route from a compliance perspective. However, collecting and managing consent at the scale needed for AI model training is often difficult. As a result, it is only used when the de-identification approaches are not possible.

Here’s a quick decision tree to help you understand if you can use PHI for model training and which path should you follow:

What Doesn't Make AI Training Compliant

Many healthcare organizations assume that once a BAA is in place, their AI model vendor can use submitted data to improve its models. That's not how HIPAA works. HHS guidance is clear that business associates may only use PHI for the purposes defined in the BAA and permitted under HIPAA. They cannot use it to train commercial AI models or improve their products.

Also Remember:The proposed 2025 HHS Security Rule update makes one thing clear: if your AI training datasets, prediction models, or algorithm data contain ePHI and support covered healthcare functions, they remain protected under HIPAA. In other words, HIPAA responsibilities don't end once a model is trained. The data and AI assets built from it may still need the same security and compliance controls as your production systems.

The HIPAA Risks AI Creates That the Regulation Did Not Anticipate

It is not hard to find HIPAA compliance checklists for healthcare software development. But the scenario changes when the software includes artificial intelligence capabilities.

HIPAA was written before large language models existed. So, while HIPAA applies to AI systems, some AI-specific risks are not explicitly named. That doesn't mean those risks fall outside HIPAA. It just means healthcare organizations and their development teams are responsible for applying existing HIPAA principles to entirely new AI behaviors.

The three areas below are where most AI-specific compliance gaps appear.

1. LLM Hallucinations Can Create New PHI Risks

Large language models don't just retrieve information. They also generate it. Sometimes that information is incorrect or entirely fabricated, a behavior known as hallucination.

In a healthcare setting, this becomes more than an accuracy problem. A hallucinated patient history, diagnosis, or prescription can enter a clinical workflow. When treated as genuine patient information, it can have far-reaching impact. Even if the model didn't expose real PHI, it has still introduced false patient data into a regulated environment.

To reduce this risk, development teams should:

• Validate AI-generated content before it reaches users or patient records.
• Require human review for outputs that influence clinical decisions.
• Log flagged outputs for investigation and audit.

2. The Minimum Necessary Standard Applies to AI Too

HIPAA's minimum necessary standard requires organizations to limit PHI access only when and where it is needed to perform a specific task. The same rule applies to AI.

For example, if an AI is generating a discharge summary, it may only need the current admission notes. Giving the model access to the patient's full medical history unnecessarily expands the amount of ePHI exposed to the AI.

To meet this requirement, development teams should:

• Filter the records and fields passed to the model based on the specific use case.
• Restrict AI context to only the PHI required for the current request.
• Document the minimum necessary data needed for every AI workflow.

More context may improve model performance, but unnecessary access increases compliance risk. This is a tradeoff that you need to consider carefully.

3. Using External AI Models Is a Shared Responsibility

Many healthcare applications rely on external AI providers rather than self-hosted models. The moment patient data is sent to a third-party AI service, that provider becomes part of your HIPAA compliance boundary.

Before any PHI is transmitted, the vendor must be willing to sign a Business Associate Agreement (BAA). Without one, the service cannot legally receive PHI, regardless of its security certifications.

Other than that, you should also verify how the vendor handles prompts, stores data, retains logs, and isolates customer information. These decisions affect your compliance posture just as much as your own application architecture.

AI introduces new technologies, but not new exemptions from HIPAA. As AI capabilities evolve, the best healthcare software development companies are those that also apply existing privacy and security principles to every AI model, workflow, and integration they build.model, workflow, and integration they build.

Beyond HIPAA: Section 1557 and AI Nondiscrimination

HIPAA isn't the only regulation healthcare AI systems need to meet. If you are building an artificial intelligence-powered software solution that helps make or influences patient care decisions, you also need to consider Section 1557 of the Affordable Care Act. It prohibits discrimination based on race, color, national origin, sex, age, or disability. In July 2024, the HHS Office for Civil Rights (OCR) confirmed these protections apply to healthcare AI systems. Enforcement of AI nondiscrimination requirements became effective May 2025.

This covers AI used for diagnostic suggestions, triage prioritization, treatment recommendations, and prior authorization determinations. Healthcare organizations must identify and mitigate risks of unlawful discrimination in AI outputs. Plus, you need to be able to show proof of that.

From a development perspective, that means:

• Bias testing before deployment across different patient demographic groups.
• Ongoing monitoring to detect changes in model behavior over time.
• Human review for AI outputs that influence patient care decisions.
• Documentation of testing methods, results, and corrective actions for OCR review.

For healthcare AI, bias testing is no longer just a responsible development practice. It's becoming an essential part of building a system that can stand up to regulatory review.

HIPAA AI Risk Analysis: What OCR Now Requires You to Document

HIPAA risk analysis is the process of identifying where electronic protected health information (ePHI) is exposed, what risks exist, and how those risks will be managed.

According to the 2025 Healthcare Data Breach Report, 76% of 2025 HIPAA enforcement cases involved failures in risk analysis. Until recently, AI systems were rarely assessed as a separate part of this process. The proposed 2025 HHS Security Rule changes make that expectation explicit:

If an AI system handles ePHI, it must be included in your risk analysis and risk management plan.

Irrespective of the type of healthcare software you are building, here is what your AI risk analysis should document:

• Identification of all AI systems that touch ePHI, including third-party APIs that receive PHI as input
• Threat and vulnerability assessment for each AI system: what could go wrong, how likely is it, and what is the impact on PHI confidentiality, integrity, and availability
• For generative AI systems specifically: assessment of hallucination risk, PHI leakage risk through model outputs, and training data PHI risk if any
• Risk management plan with documented remediation actions and timelines for each identified risk
• Periodic review and update of the risk analysis as the AI system changes

Risk analysis is not a paperwork exercise or a last-minute compliance checklist. It is the documented evidence that you identified AI risks, addressed them, and made informed security decisions. And you cannot defer it for later anymore.

As Pratik Mistry, EVP Technology Consulting at Radixweb, puts it, “We often see teams treat compliance as something they'll tackle just before launch because they don't want to development. In practice, it works the other way around. Planning for HIPAA and Section 1557 early is far easier than redesigning data flows, access controls, or AI workflows later. If compliance issues are discovered after deployment, the cost isn't just technical. It can quickly become a legal, financial, and reputational problem."

The Development Checklist: HIPAA Compliance Built In, Not Added On

Wondering if your AI system is HIPAA compliant? Well, the answer isn't in an audit or legal document. It is in the decisions made across architecture, development, testing, deployment, and operations. Miss one, and you create a compliance gap that's expensive to fix later.

When planning a new AI solution or assessing an existing one, use the checklist below as a practical way to review your AI system. Identify what is in place, what needs attention, and the biggest compliance risks.

Architecture-stage decisions:

• Data residency: Where will ePHI be stored, processed, and transmitted? All environments must be in HIPAA-covered infrastructure with BAA-covered cloud providers
• Encryption: AES-256 encryption at rest, TLS 1.2 minimum in transit. This should be confirmed before any ePHI enters any environment
• Model data access scope: Define the minimum necessary PHI for each AI use case before model architecture design
• Audit log architecture: What events will be logged, where, in what format, for how long, and protected how? This should be designed in from Sprint 1
• De-identification pipeline: Will PHI will be used in model training or testing? If yes, the de-identification pipeline must be designed and validated beforehand.

Pre-go-live requirements:

• BAA executed with every vendor in the architecture that handles ePHI. This includes cloud providers, AI API vendors, monitoring services, and logging platforms
• Risk analysis completed and documented covering all AI system components
• Breach notification procedure tested end-to-end, including timing from detection to covered entity notification
• Bias testing completed and documented for all systems making/influencing patient care decisions
• Staff training completed covering proper use of the AI system and HIPAA obligations

Post-launch obligations:

• Quarterly access control review: Confirm user access remains appropriate, deactivate terminated user accounts
• Audit log review: Regular review of AI audit logs for anomalous access patterns
• Risk analysis update: Update within 30 days of significant changes to the AI system
• Section 1557 monitoring: Ongoing review of AI output distributions for demographic bias

Every unchecked item on the list above as a potential risk. While some of these can be addressed during implementation, others need system-level changes.

So, review this checklist before development starts, before go-live, and after every major AI update to keep your solution aligned with HIPAA requirements.

Build Your HIPAA-Compliant AI with the Right Team

HIPAA compliance isn't something you bolt on at the end. It starts with the very first decisions you make: How will your AI access patient data? Where does that data go? What gets recorded? How do you manage risks? These choices lock in your compliance posture and they're expensive to change later. That's why your AI partner matters as much as your AI model. You need a team of developers who understand healthcare’s specific nuances and enterprise AI equally well.With 26+ years of experience in building enterprise software and 4,500+ projects shipped across 30+ countries, we, at Radixweb, know how to build AI systems that work and stay compliant. So, whether you're starting from scratch or adding AI to an existing healthcare platform, we've solved these problems before. And we can help you design secure data flows, identify compliance gaps before they become expensive, and build scalable systems that don't require rebuilding as you grow. Schedule a no-cost consultation with our healthcare AI specialists to review your current system or idea, spot potential risks, and create a clear roadmap to HIPAA-compliant AI.