Priority area 1: Consumers

Empower and prioritise consumers

Proposed AI systems should be safe, deliver meaningful benefits to consumers and communities, represent the most appropriate approach, and meet a clearly defined, patient-centred need.

Building trust and awareness among consumers is essential. This can be achieved in various ways, including through responsible AI governance, by openly discussing potential risks and benefits of AI systems prior to use, and by enhancing AI health literacy. Engaging consumers and other stakeholders in planning and co-design is critical to successful integration.

Principles

Trust and awareness

Consumers should be empowered to make informed decisions and be told when AI systems are integrated into their healthcare. NSW Health requires a valid legal basis to input personal information into AI systems.

Foster awareness and data privacy through clear, concise and accessible communication; and through appropriate legal privacy settings before using personal information in AI systems.
Ensure consumers are provided with information in plain language about the purpose of an AI system, and its function and performance. This is necessary for valid consent.
Build trust by demonstrating there will be human oversight and accountability over the lifecycle of the AI system.

Equity and accessibility

AI should be fair and should not cause bias and inequality. Consumers should have equitable access to AI systems.

Focus on implementing relevant AI systems that are accessible to all NSW Health consumers and communities, and are fit-for-purpose.
Advocate for, and collaborate with, priority populations to ensure AI proposals align with the needs of all communities, and ensure AI system training datasets are representative and appropriate.
Monitor for potential bias to ensure AI systems are inclusive and accessible.

Consumer-centred

AI systems should minimise harm and maximise benefit to communities and respect human rights and diversity.

Ensure the risk of potential AI harms has been evaluated, including reversable, significant, and possible secondary or cumulative harms.
Ensure the use of any AI system has a benefit to consumers and the community, and the proposed AI system use aligns with consumer laws.
Ensure clear and appropriate consent processes so that consumers are aware of how their care may be impacted by AI.
Support contestability, explainability and transparency to enable consumers and affected stakeholders to find out when and how AI systems have been, or will be, used in connection with the provision of their healthcare.
Provide a process to challenge AI system use and outcomes in circumstances where it impacts individual rights and opportunities, and maintain consumer autonomy by respecting human rights.

Policy and guidance

Below are the key considerations for integrating AI, along with current policies and guidance that outline healthcare and technology obligations when prioritising consumers.

Resources may be relevant to multiple areas for consideration.

Key considerations	Current policies and guidance
Foster AI health literacy	National Safety and Quality Health Service Standards Refer to Standard 2 for partnering with consumers. Source: Australian Commission on Safety and Quality in Healthcare AI Clinical Use Guide This guide and associated safety scenarios support clinicians, together with their patients, in using AI safely and responsibly in patient care. It is structured to support the steps of ‘before you use’, ‘while you use’ and ‘after you use’ AI tools. Source: Australian Commission on Safety and Quality in Healthcare Statewide Health Literacy Hub Provides practical tools and guidance to strengthen health literacy leadership, capacity and systems for clear, equitable access to healthcare services and information across NSW. Source: NSW Health Future Health Strategy Promotes building health literacy and access to information for consumers. Source: NSW Health Australian Digital Health Capability Framework A standard framework for digital health capabilities for people working in healthcare. It promotes workforce capability to effectively understand, use and apply digital technologies in healthcare. Source: Australian Digital Health Agency Meeting Your Professional Obligations When Using Artificial Intelligence in Healthcare An overview of the responsibilities of health professionals when using AI tools, including AI literacy. Source: Australian Health Practitioner Regulation Agency
Engage with consumers	Elevating the Human Experience Outlines what the NSW health system can do, in partnership with patients, families, carers and caregivers, to consistently create positive, personalised experiences. Source: NSW Health All of Us A guide to respectful engagement with consumers, carers and communities across NSW Health. Source: NSW Health NSW Aboriginal Health Governance, Shared Decision Making and Accountability Framework The 'gold standard' for Aboriginal governance, shared decision making and accountability across NSW Health services. Source: NSW Health Value Based Healthcare in NSW Best practice advice for designing and delivering consumer-focused healthcare. Source: NSW Health Strategic Framework for Integrating Care Supports better outcomes for consumers, health professionals and the broader health system. Source: NSW Health Working with Consumers Step-by-step guidance to support health staff to become more capable and skilled at working with consumers. Source: Agency for Clinical Innovation
Consider consent requirements	Consent to Medical and Healthcare Treatment Manual Operational guidance and procedures to support compliance with the NSW law on obtaining consent. Source: NSW Health

Practice

Practice areas	Considerations
Create a consumer engagement plan	Engage early with consumers in the development and implementation of AI systems Develop a communications strategy and related materials to inform consumers Ensure consumers have access to information and support, as needed
Adopt equity measures	Analyse datasets used to train AI systems to ensure they are culturally respectful and representative of the target population, and are trained on appropriate health data, as applicable Monitor and measure associated health outcomes across all demographic groups Address barriers to access Ensure benefits of AI systems are equitably realised across rural, regional and metropolitan areas
Conduct thorough user testing	Determine whether the AI system provides consumer-focused outcomes Ensure solutions are explainable, user-friendly and reliable through competitive user-testing Seek consumer feedback
Develop clear consent processes	Identify the specific AI component requiring consent, e.g. use as a diagnostic tool, decision support or data collection Determine when consent is needed, e.g. at admission, before specific AI use or when data is shared Develop consent resources that are user-friendly, in multiple languages and formats, to meet consumer needs, in alignment with privacy laws Consider consent requirements when an individual’s data may be used by, or in connection with, an AI system, including to train the model, and seek legal advice, as required

Challenges and opportunities

Consumers receiving care supported by AI technology face associated risks of harm, misuse, misinformation and bias. However, effective AI adoption has the potential to improve healthcare workflows and outcomes.

As AI advances, consumers will interact with systems aimed at:

earlier detection and diagnostics
personalised care and treatment
improved access to care
enhanced management of chronic illness
streamlined consumer experiences.

Previous:Using AI in NSW Health

Next:Priority area 2: Workforce