Singapore launches worlds first AI testing framework and toolkit to promote transparency

Joint Press Release by Ministry of Communications and Information, Infocomm Media Development Authority and Personal Data Protection Commission

1. Singapore launches A.I. Verify - the world’s first AI Governance Testing Framework and Toolkit for companies who want to demonstrate responsible AI in an objective and verifiable manner. This was announced by Singapore’s Minister for Communications and Information Mrs Josephine Teo at the World Economic Forum Annual Meeting in Davos. A.I. Verify – currently a Minimum Viable Product¹ (MVP), aims to promote transparency between companies and their stakeholders through a combination of technical tests and process checks. 

2. Globally, testing for the trustworthiness for AI systems is an emergent space. As more companies use AI in their products and services, fostering public’s trust in AI technologies remains key in unlocking the transformative opportunities of AI. 

Singapore remains at the forefront of international discourse on AI ethics

3. The launch of A.I. Verify follows Singapore’s launch of the Model AI Governance Framework (second edition) in Davos in 2020, and the National AI Strategy in November 2019. Having provided practical detailed guidance to industry on implementing responsible AI, A.I. Verify is Singapore’s next step in helping companies be more transparent about their AI products and services, to build trust with their stakeholders. A.I. Verify is developed by the Infocomm Media Development Authority (IMDA) and the Personal Data Protection Commission (PDPC).

Objective and verifiable testing process for industry

4. Developers and owners can verify the claimed performance of their AI systems against a set of principles through standardised tests. A.I. Verify packages a set of open-source testing solutions together, including a set of process checks into a Toolkit for convenient self-assessment. The Toolkit will generate reports for developers, management, and business partners, covering major areas affecting AI performance. The approach is to allow transparency, of what the AI model claims to do vis-à-vis the test results, and covers areas such as: 

a. Transparency: 
i. On the use of AI to achieve what stated outcome
ii. Understanding how the AI model reaches a decision
iii. Whether the decisions predicted by the AI show unintended bias

b. Safety and resilience of AI system.

c. Accountability and oversight of AI systems.

5. The Pilot Testing Framework and Toolkit:

a. Allows AI system developers/owners to conduct self-testing – to maintain commercial requirements while providing a common basis to declare results.

b. Does not define ethical standards. It validates AI system developer’s/owner’s claims about the approach, use, and verified performance of their AI systems.

c. Does not however guarantee that any AI system tested under this Pilot Framework will be free from risks or biases or is completely safe.

6. Minister for Communications and Information Mrs Josephine Teo said, “A.I. Verify is another step forward in Singapore’s AI development. In developing the world’s first product to demonstrate responsible AI in an objective and verifiable manner, we aim to help businesses become more transparent to their stakeholders in the A.I use. This will, in turn, promote greater public trust towards the use of AI. We invite industry partners from all around the world to join us in this pilot and contribute to building international standards in AI governance.”

7. Also commenting on the launch, Mr Chia Song Hwee, Deputy CEO, Temasek International and member of Singapore’s Advisory Council on the Ethical Use of AI and Data said, “I would like to congratulate IMDA for taking responsible AI to the next milestone with the launch of this testing framework and toolkit. Rapid digitisation has led to a proliferation of data and improved algorithms. As companies across sectors continue to innovate, this toolkit will enable them to turn concepts of responsible and trustworthy AI into practical applications that will benefit all stakeholders, from business owners to end users.”  

Building interoperability of trustworthy AI with partners and industry 

8. Developed under the guidance of the Advisory Council on the Ethical Use of AI and Data, 10 companies from different sectors and of different scale, have already tested and/or provided feedback. These companies are AWS, DBS Bank, Google, Meta, Microsoft, Singapore Airlines, NCS (Part of Singtel Group)/Land Transport Authority, Standard Chartered Bank, UCARE.AI, and X0PA.AI.

9. In addition, Singapore is engaging other like-minded countries and partners to enhance the interoperability of AI governance frameworks and to develop international standards on AI such as through Singapore’s participation in ISO/IEC JTC1/SC 42 on Artificial Intelligence. IMDA is also working together with the U.S. Department of Commerce to build interoperable AI governance frameworks.  Beyond the pilot stage of the MVP, Singapore aims to work with AI system owners/developers globally to collate and build industry benchmarks. This enables Singapore to continue contributing to the development of international standards on AI governance. 

Organisations invited to participate in pilot

10. As AI governance and testing is nascent, Singapore welcomes organisations to participate in piloting the MVP. Companies participating in the pilot will have the unique opportunity to:

a. Gain early access to the MVP and use it to conduct self-testing on their AI systems/models;

b. Use MVP-generated reports to demonstrate transparency and build trust with their stakeholders; and

c. Help shape an internationally applicable MVP to reflect industry needs and contribute to international standards development.

More details can be found: https://go.gov.sg/aiverify

Annex A-Quotes from industry and partners
Annex B- Background on Singapore's AI governance work

--------------------------------------------------------------------------------------------------------------------------
¹  _{MVP - a product with just enough features for early adopters to test and provide feedback for product development.}