In the past, complaint handling systems were suited for a slower pace of incoming reports with similar characteristics, but this has changed with the advent of high-volume product lines generating thousands of records monthly from various sources, including call centers, field service reports, and social media, with some being detailed and others vague, such as two lines written by someone who never saw the device fail.
Every record must be assessed: Is it a complaint? Does it need investigation? Is it reportable to the FDA? The timelines are not negotiable. Under 21 CFR Part 803, manufacturers must file a Medical Device Report within 30 calendar days of becoming aware of an event that reasonably suggests a death, serious injury, or a malfunction likely to cause one if it recurs. Certain events requiring remedial action to prevent an unreasonable risk of substantial harm to public health must be reported within five work days. The clock starts when any employee becomes aware of the event, not when the complaint file is opened.
AI Steps In as Manual Triage Falters
Manual triage cannot keep up with this volume. Senior reviewers read everything, backlogs build, and consistency suffers. Two experienced reviewers can score the same vague complaint differently, and the same reviewer can score it differently on a Friday afternoon than on a Monday morning. Triage was already the weakest link in post-market surveillance before volumes exploded. AI arrived as the obvious solution.
When used properly, AI eliminates blind spots inherent in manual triage. The clearest benefit is processing free text at scale. Natural language models can process every incoming record the day it arrives, normalize various formats into one structure, translate foreign-language reports, and suggest initial event codes. Nothing remains unread. For a process where the regulatory clock starts before the file is opened, this alone is valuable.
The second benefit is memory. A reviewer handling sixty complaints a day will not connect complaint number fourteen on Tuesday with a similar complaint from another region three months ago. An AI model will. Pattern detection across regions, lots, and time is where algorithms outperform humans, and it aligns with the signal detection work expected by post-market surveillance regulations. A cluster that might take a human team a quarter to notice can surface in days.
The third benefit is consistency. A model applies the same logic to the ten thousandth record as it did to the first. It does not tire, and its judgment does not waver on a Friday afternoon.
Systematic Errors Lurk in AI Triage
When a human reviewer misjudges a complaint, the error is typically random. Different reviewers catch each other’s mistakes, second looks occur, and the process self-corrects. When a model misjudges a complaint, the error is systematic. If it has learned a pattern that scores a certain type of event as low priority, it will score every instance of that event as low priority, quietly and at scale, until someone notices. This goes unnoticed quickly because the records the model buries are precisely those no one is reading anymore.
Models also drift. Products change, new failure modes emerge, and users invent new language to describe them. A model trained on last year’s complaints can overlook this year’s problems. Additionally, people drift too. Reviewers who watch an algorithm sort the queue correctly for months begin to trust the queue order itself. This is known as automation bias, and it weakens the human check as the team becomes complacent.
Regulatory Scrutiny of AI Tools Intensifies
Under the Quality Management System Regulation, which incorporates ISO 13485:2016, clause 4.1.6 applies: software used in the QMS must be validated before use, with rigor proportionate to the risk it carries. A tool influencing complaint prioritization and reportability decisions carries significant risk. Validation involves defining the tool’s scope, testing it against known outcomes, and setting acceptance criteria before live use.
Inspection and compliance
FDA inspections focus on Medical Device Reporting, and investigators examine complaint trends and reporting history. If an algorithm is used for prioritization, inspectors will request validation evidence, monitoring methods, and decision-making accountability.
The EU Medical Device Regulation also emphasizes post-market surveillance, and notified bodies ask similar questions about system validation and decision ownership. An audit trail must show model suggestions, human decisions, and individual accountability for overrides.
Ongoing Monitoring Ensures AI Reliability
A defensible AI triage program requires ongoing monitoring to ensure reliability. This includes sampling the model’s output by having qualified reviewers periodically re-evaluate records it flagged as low priority, without knowing the model’s score. Performance tracking is essential, measuring how accurately the model identifies reportable events over time. Retraining triggers should be defined, such as product launches, design changes, shifts in complaint language, or drops in sampling performance, to ensure the model remains valid.
Before deployment, acceptance criteria must be established. The model should demonstrate the ability to catch all known reportable cases in a challenge set. This test set should be retained for reference. These practices align with standard quality processes, treating the model as any other critical process. By systematically monitoring and maintaining the model, companies can mitigate risks associated with systematic errors and ensure compliance with regulatory timelines.
Leave a Reply