A team of technology experts has launched a global initiative named \"Humanity's Last Exam,\" seeking the most challenging questions to test the limits of artificial intelligence systems. Organized by the Center for AI Safety (CAIS) and the startup Scale AI, the project aims to determine when AI reaches expert-level performance.
The call for questions comes shortly after OpenAI previewed its new model, OpenAI o1, which has significantly outperformed previous benchmarks. Dan Hendrycks, executive director of CAIS and advisor to Elon Musk's xAI startup, highlighted that AI systems are now effortlessly handling tests that once stumped them.
Historically, AI struggled with undergraduate-level exams and competitive math problems, often providing random answers. However, recent advancements have seen models like Anthropic's Claude improve their scores dramatically, rendering traditional benchmarks less meaningful.
To address the growing capabilities of AI, \"Humanity's Last Exam\" will feature over 1,000 crowd-sourced questions due by November 1. These questions will undergo peer review, and top submissions will earn co-authorship and prize money sponsored by Scale AI. The exam focuses on abstract reasoning and will keep certain questions private to prevent memorization by AI systems.
Alexandr Wang, CEO of Scale AI, emphasized the need for harder tests to accurately measure AI's rapid progress. The organizers have set a clear boundary by excluding questions related to weapons, ensuring the safety and ethical use of AI in this endeavor.
Reference(s):
AI experts ready 'Humanity's Last Exam' to stump powerful tech
cgtn.com
