Priority area 5: Safety, ethics and quality

Maintain safe and ethical healthcare delivery

The integration of AI across NSW Health requires a focus on safety and expected benefits.

AI systems that are effectively designed, implemented and monitored have the potential to revolutionise healthcare delivery, enhance safety and improve health outcomes. Monitoring and assessing risks are vital to ensure AI supports fair, safe and effective healthcare.

Principles

Fairness and minimising bias

Use of AI should uphold human rights and human-centred values over the lifecycle of the implementation.

Prioritise human agency, monitoring and authority over AI system approval and use.
Value the continuous role of monitoring and auditing to ensure the use of AI aligns with human rights and ethical standards.
AI should be rigorously evaluated, and data should include diverse and representative datasets, where appropriate.
Implement strategies and safeguards to detect and correct AI system biases.

Safety first

Clinical AI systems should support safety, and operate reliably and responsibly to protect consumers and communities.

Ensure any AI system that operates as a medical device is appropriately listed on the Australian Register for Therapeutic Goods and is approved for the intended use.
Ensure traceability to help understand AI system outcomes, identify potential errors and improve trustworthiness of the system.
Maintain safety mechanisms to enable an AI system to be overridden, repaired or decommissioned to prevent misinformation and harm.
Foster reliability, repeatability and reproducibility.

Consumer and community benefit

AI systems should benefit the health outcomes of consumers and communities, and may also deliver broader benefits across the health system.

Determine the AI system is the most appropriate solution to address key needs in service delivery.
Establish benefits that can be clearly identified and justified.
Identify if discrimination or bias is occurring in an AI system and correct prior to use, if appropriate, or discontinue use.

Policy and guidance

Below are the key considerations for integrating AI, along with current policies and guidance that outline healthcare and technology obligations around safety, ethics and quality care.

Topic	Current policies and guidance
Clinician responsibilities	Meeting Your Professional Obligations When Using Artificial Intelligence in Healthcare Overview of the responsibilities of health professionals when using AI systems. Source: Australian Health Practitioner Regulation Agency Artificial Intelligence in Primary Care Position statement on the use of AI in primary care. Source: Royal Australian College of General Practitioners AI Clinical Use Guide This guide and associated safety scenarios support clinicians, together with their patients, in using AI safely and responsibly in patient care. It is structured to support the steps of ‘before you use’, ‘while you use’ and ‘after you use’ AI tools. Source: Australian Commission on Safety and Quality in Healthcare Artificial Intelligence Information and considerations for how AI will impact clinical practice. Source: The Royal Australian and New Zealand College of Radiologists Virtual Care in Practice Guide Information to support healthcare providers to deliver virtual care, including consumer considerations, ways to deliver virtual care, technology and equipment, training and education, and implementation. Source: Agency for Clinical Innovation
Clinical governance	National Safety and Quality Health Service Standards Standard 1 focuses on clinical governance and Standard 6 focuses on communicating for safety. Source: Australian Commission on Safety and Quality in Health Care Clinical Governance in NSW Policy (PD2025_032) Outlines key requirements for effective clinical governance to ensure the best clinical outcomes possible. Source: NSW Health Incident Management Policy (PD2020_047) Provides direction for consistency in managing and effectively responding to clinical and corporate incidents, and acting on lessons learnt. Source: NSW Health
New technology assessment	New Health Technologies and Specialised Services (GL2024_008) Outlines the process for reviewing and assessing health technologies that are new to the NSW public health system. Source: NSW Health Local new health technology processes Outlines suggested processes for local health districts and specialty networks to introduce new interventions, and only applies to local implementation. Source: NSW Health Artificial Intelligence (AI) and Medical Device Software Information for software manufacturers about how the Australian Government regulates AI medical devices. Source: Australian Government, Therapeutic Goods Administration
Meeting accreditation requirements	Australian Health Service Safety and Quality Accreditation Scheme Ensures the minimum standards for the delivery of clinical care are maintained. Source: Australian Commission on Safety and Quality in Health Care

Practice

Practice areas	Considerations
Clinical effectiveness and efficiency	Consider how the AI system will integrate with existing workflows (including clinical workflows) in real-world practice to achieve improved health outcomes across diverse populations and settings Evaluate resourcing, such as time, personnel, cost and environmental sustainability considerations, e.g. energy consumption and carbon footprint Determine how the AI system was trained and whether this included use of localised datasets. In the case of clinical AI systems, appropriate and specific health data that is representative of the NSW Health patient population Customise off-the-shelf systems, where required, to ensure best fit for the local environment
Risk and incident monitoring	Establish a risk management plan to avoid secondary or cumulative harm Ensure continuous monitoring to maintain quality, effectiveness and human oversight Maintain ongoing technical support processes and feedback mechanisms to allow for regular updates and bug fixes Ensure repeatability, reliability and reproducibility where the model provides consistent outcomes multiple times in the same environment, or when applied in different settings by different teams Review outputs for potential bias or hallucination
Post implementation and quality assurance	Consider how clinical safety and efficacy is achieved and maintained over the lifetime of the AI system Determine performance parameters and plan audit and monitoring frequency to ensure alignment with current policies and guidelines, and identify potential model drift Engage with ethics and clinical governance committees to seek workflow integration and safety recommendations Ensure clinical oversight of any AI system used in the provision of healthcare.

Challenges and opportunities

Risks associated with implementing AI systems can be addressed by designing robust, consumer-centred systems that prioritise safety, equity and accessibility. Clear pathways for monitoring and escalation must be integrated.

Safely harnessing AI to complement existing healthcare delivery has the potential to deliver many benefits, such as:

early and accurate disease detection and diagnosis
streamlined workflows
personalised treatments
enabling care closer to home
capacity planning.

Previous:Priority area 4: Governance and regulation

Next:Priority area 6: Research and development