Researchers have identified flaws in numerous tests that assess the safety and effectiveness of new artificial intelligence models. According to The Guardian, the study was conducted by specialists from the UK Government's Artificial Intelligence Security Institute, along with experts from Stanford, Berkeley, and Oxford universities.
They examined over 440 tests evaluating AI safety systems.
The experts found that these tests have shortcomings that, they argue, «undermine the credibility of the results obtained». They also noted that nearly all tests have «weaknesses in at least one area», and the results may be «irrelevant or even misleading».
Many of these assessments are used to evaluate new AI models released by major tech companies, noted researcher Andrew Bean from the Oxford Internet Institute.
In the UK and the US, there is a lack of national regulation for AI, so these tests are used to check whether new models are safe and align with human interests.
«Tests are fundamental to nearly all claims of progress in the field of artificial intelligence. However, without common definitions and reliable measurement methods, it is challenging to determine whether models are genuinely improving or merely giving the appearance of advancement,» emphasized Bean.
The study focused on publicly available tests, but leading AI companies also have their own internal tests that were not reviewed.
Bean remarked that «the shocking finding was that only a small minority (16%) of tests utilized uncertainty estimates or statistical methods to indicate how likely it is that the criteria would be accurate. In other cases, when criteria were set to assess AI characteristics, particularly its «harmlessness», the definitions were often controversial or vague, which reduced the utility of the test.
The study's conclusions highlight the «urgent need for shared standards and best practices» concerning AI.