Priority area 4: Governance and regulation

Prioritise responsibility, transparency and compliance

Healthcare delivery across NSW Health is highly regulated with a range of existing regulatory levers to manage various risks associated with AI systems.

As AI adoption grows, a considered approach is needed to ensure existing guidance remains adequate and up to date, and new policies are developed, as required.

Accountability

Decision making and proper functioning of approved AI systems is the responsibility of NSW Health entities, and will be enabled by allocating of individual roles and responsibilities.

Ensure clear governance and accountability structures that identify who is accountable for AI governance, who procures AI systems and who implements AI systems, as well as how AI systems will be monitored and overseen (including AI informed decisions within NSW Health).
Acknowledge humans hold ultimate responsibility for any AI-supported or informed decisions.
Ensure clear roles and responsibilities for:
- policies, practices, and procedures associated with AI systems
- the development and deployment of AI systems, including ongoing human control and oversight
- oversight of the development and use of AI systems by third parties
- testing AI systems across NSW Health
- oversight of concerns, challenges, and requests for redress
- the performance and continual improvement of AI system management across NSW Health.
Remain responsible for all AI supported decisions and monitoring as required.
Develop appropriate AI literacy within NSW Health to facilitate adequate supervision of AI systems by individuals with relevant expertise and support methods of human intervention.
Use appropriate AI systems.

Transparency

AI systems must operate with clear, accessible and meaningful transparency, and need to clearly define the function of the AI system and the learning that it does.

Identify and document key types of stakeholders, e.g. personnel or consumers of NSW Health entities, that may be impacted by the AI system.
Identify and prioritise stakeholder needs to be addressed in policies and procedures.
Engage with stakeholders and identify and document the potential benefits and harms for each AI system, including use of personal information, impact on vulnerable groups, and risk of unwanted bias.
Conduct appropriate stakeholder impact analysis for each identified risk of harm.
Communicate NSW Health's commitment to preventing or mitigating AI-related harms.
Document the scope of each AI system, including intended use cases, foreseeable misuse, capabilities and limitations.
Inform the workforce and consumers about the use of AI systems, and when the individual is interacting with AI-generated content and/or may be impacted by an automated decision. Determine clear objectives, communicate how safeguards have been put in place, and inform individuals of potential harms and planned outcomes of the AI system in use.
Ensure AI systems allow for transparent explanation of the factors leading to a decision or insight.
Identify vulnerable groups who may be adversely impacted by AI system use and provide appropriate guardrails to limit risks.
Provide a mechanism for workforce and consumers to query and seek reviews of AI-based decisions, as appropriate, and allow concerns to be raised and addressed by use of a simple and accessible process.

Policy and guidance

Below are the key considerations for integrating AI, along with current policies and guidance that outline healthcare and technology obligations around AI governance and regulation.

Topic	Current policies and guidance
Key legislative regulation of AI in NSW	Civil Liability Act 2002 (NSW) Privacy Act 1988 (Cth) Health Records and Information Privacy Act 2002 (NSW) Government Information (Public Access) Act 2009 (NSW) Privacy and Personal Information Protection Act 1998 (NSW) Copyright Act 1968 (Cth) Patents Act 1990 (Cth) Competition and Consumer Act 2010 (Cth) Online Safety Act 2021 (Cth) Therapeutic Goods Act 1989 (Cth) Therapeutic Goods (Medical Devices) Regulation 2002 (Cth)
Procurement standards	AI Procurement Essentials Provides guidance around the safe and effective procurement of AI. Source: Buy NSW NSW Health Procurement (Goods and Services) Policy (PD2024_044) Policy outlining principles and processes NSW Health entities must apply when procuring or disposing of goods or services, and managing resulting contracts. Source: NSW Health Procurement Policy Framework Applies to the procurement of goods and services of any kind, including construction. Agencies must test compliance with the framework on a regular basis. Source: Buy NSW Approved procurement arrangements PBD 2021-04 Sets out approved procurement arrangements for NSW government agencies. Source: NSW Procurement Board Use of Artificial Intelligence by NSW Government Agencies (DCS-2024-04) This circular outlines the requirement for agencies to adhere to the Artificial Intelligence Ethics Policy and Artificial Intelligence Assessment Framework when procuring AI. Source: NSW Department of Customer Service
Intellectual property	NSW Health Commercialisation Framework Provides accessible information to NSW Health employees engaged in the commercialisation of intellectual property. Source: NSW Office for Health and Medical Research Intellectual Property Arising from Health Research Policy (PD2023_007) Outlines obligations of public health organisations and employees regarding intellectual property arising from health research. Source: NSW Health
Regulatory compliance	Australian Health Service Safety and Quality Accreditation Scheme Ensures the minimum standards for the delivery of quality and clinical care are maintained. Source: Australian Commission on Safety and Quality in Health Care Artificial Intelligence (AI) and Medical Device Software Information for software manufacturers about how the Australian Government regulates AI medical devices. Source: Therapeutic Goods Administration
Responsibility	Understanding Responsibilities in AI Practices Provides guidance for aligning responsibilities and accountability within NSW Government agencies based on ISO/IEC standards and the NSW AI Assessment Framework. Source: Digital NSW Transparency in decision making Information about why and how to be transparent about how you decide to improve and change your product or service. Source: Digital NSW

Practice

Practice areas	Considerations
Responsibility	Ensure each AI system has documented accountabilities for managing risks, and providing evidence for decisions and actions Ensure record-keeping for use of, and decisions related to, AI systems, including approval, risk assessment, use and ongoing monitoring Clearly define and communicate AI-related responsibilities and authorities within the organisation
Intellectual property	Determine the commercialisation approach to intellectual property (IP) using NSW Health guidance, and in accordance with law Determine what IP rights NSW Health wants over the AI system if NSW Health data is being used to train the model and how these rights might be protected, including contractually. Engage expertise, as required, to better understand the IP considerations and impacts of AI system procurement and use, ensuring appropriate IP settings and exit rights
Identifying and registering AI systems	Notify the eHealth NSW AI Advisory Service of all new AI system projects, and complete a request form and preliminary assessment Maintain a registry of AI systems assessed and approved for implementation and use by NSW Health. Refer to Using AI in NSW Health
Evaluating AI systems	Fulfill mandatory assessment requirements, such as the NSW AI Assessment Framework, which determines the risk level and the pathway to implementation Seek expert advice to support evaluation Engage relevant governance groups within the organisation Ensure clarity around data ownership and IP considerations Escalate high-risk projects to the eHealth NSW AI Advisory Service, to be triaged accordingly to the NSW AI Review Committee
Monitoring and surveillance of AI systems	Monitor according to risk level Implement model documentation, version control and clear reporting mechanisms Use data quality checks, representative training sets and fairness audits as part of post-implementation surveillance Undertake policy enforcement, auditing and risk assessments over the lifecycle of AI systems Develop user feedback mechanisms to support continued improvement Enable processes to update or decommission AI systems, as needed

Challenges and opportunities

Good governance and regulation enable visibility of AI systems in use across NSW Health and awareness of the information required to review, implement and monitor these.

Previous:Priority area 3: Privacy and security

Next:Priority area 5: Safety, ethics and quality