Introduction to OpenAI’s Safety Evaluations Hub

OpenAI has introduced a new webpage, the Safety Evaluations Hub, to publicly share information about its models, including hallucination rates and the production of harmful content. This hub will also provide updates on how well models behave as instructed and attempted jailbreaks.

Background and Context

OpenAI, a company that has faced multiple lawsuits alleging the illegal use of copyrighted material to train its AI models, aims to provide additional transparency with this new page. Notably, the company has been accused of accidentally deleting evidence in a plagiarism case against it, as reported by The New York Times.

Purpose of the Safety Evaluations Hub

The Safety Evaluations Hub is designed to expand on OpenAI’s system cards, which currently only outline safety measures at launch. The hub will provide ongoing updates, allowing for a more comprehensive understanding of the safety performance of OpenAI systems over time. According to OpenAI, "As the science of AI evaluation evolves, we aim to share our progress on developing more scalable ways to measure model capability and safety."

Features of the Safety Evaluations Hub

The hub will be updated periodically, and interested parties can explore safety results for various models, including GPT-4.1 through 4.5. However, OpenAI notes that the information provided is only a "snapshot" and recommends consulting system cards, assessments, and other releases for further details.

Limitations of the Safety Evaluations Hub

One significant limitation of the hub is that OpenAI is responsible for conducting the tests and selecting what information to share publicly. As a result, there is no guarantee that the company will disclose all its issues or concerns with the public.

Conclusion

The introduction of the Safety Evaluations Hub marks a step towards greater transparency for OpenAI. While the hub has its limitations, it provides a valuable resource for understanding the safety performance of OpenAI systems and supports community efforts to increase transparency across the field. As OpenAI continues to work on developing more scalable ways to measure model capability and safety, the hub is expected to play a crucial role in promoting transparency and accountability.