0
0
Enhancing AI Safety with Enkrypt’s LLM Safety Leaderboard
In the realm of artificial intelligence, specifically in the domain of generative AI, the discussion around the performance and safety of large language models (LLMs) is of paramount importance. With the rapid advancement of these models, it is imperative for teams to prioritize the evaluation and testing of LLMs to preempt any issues that could potentially lead to negative user experiences, missed opportunities, or even regulatory penalties.
However, amidst the ever-evolving landscape of open and closed-source models, determining which LLM offers the highest level of safety can be a daunting task. Enter Enkrypt, a pioneering startup based in Boston, that has unveiled a groundbreaking solution: the LLM Safety Leaderboard. This innovative platform ranks various LLMs based on their susceptibility to safety and reliability risks, providing invaluable insights for organizations looking to make informed decisions.
Diving into the LLM Safety Leaderboard
When an enterprise incorporates a large language model into its applications, such as chatbots, it undertakes rigorous internal testing to identify and address safety risks like privacy breaches and biased outputs. A minor oversight in these assessments could result in the exposure of sensitive data or the dissemination of prejudiced information, as evidenced by past incidents like Google’s Gemini chatbot mishap. In regulated sectors like fintech and healthcare, the consequences of such lapses can be even more severe.
Founded in 2023, Enkrypt has been at the forefront of addressing these challenges with its innovative solution, Sentry, which proactively identifies vulnerabilities in generative AI applications and implements automated safeguards to mitigate potential risks. Building upon this foundation, the company has introduced the LLM Safety Leaderboard, a tool designed to empower teams in selecting the safest LLMs for their specific requirements.
This offering, developed through extensive testing across diverse scenarios and datasets, assigns a comprehensive risk score to 36 prominent open and closed-source LLMs. It evaluates various safety and security metrics, including the models’ capacity to avoid generating harmful or biased content and their ability to thwart malware or injection attacks.
The Safest LLMs Unveiled
According to Enkrypt’s leaderboard, OpenAI’s GPT-4-Turbo emerges as the standout performer, boasting the lowest risk score of 15.23 as of May 8. This model demonstrates exceptional resilience against jailbreak attacks and generates toxic outputs merely 0.86% of the time. However, it does encounter challenges related to bias and malware, affecting the model 38.27% and 21.78% of the time, respectively.
Following closely behind are Meta’s Llama2 and Llama 3 models, with risk scores ranging from 23.09 to 35.69. Anthropic’s Claude 3 Haiku also secures a commendable position on the leaderboard, garnering a risk score of 34.83. Despite performing well across various assessments, the model exhibits a significant propensity for biased responses, exceeding 90% of the time.
Conversely, models like Saul Instruct-V1 and Microsoft’s Phi3-Mini-4K trail at the bottom of the rankings, with risk scores of 60.44 and 54.16, respectively. Additional models, such as Mixtral 8X22B and Snowflake Arctic, also feature lower rankings on the leaderboard, underscoring the importance of continual improvement and evolution within the AI landscape.
Enkrypt intends to update the leaderboard periodically to reflect advancements in existing models and the introduction of new ones. This dynamic approach ensures that organizations have access to the most current and relevant insights for informed decision-making.
“Integrating our leaderboard into AI strategy not only boosts technological capabilities but also upholds ethical standards, offering a competitive edge and building trust. The risk/safety/governance team within an enterprise would use the Leaderboard to provision which models are safe to use by the product and engineering teams. Currently, they do not have this level of information from a safety perspective – only public performance benchmark numbers. The leaderboard and red team assessment reports guide them with safety recommendations for the models when deployed,” shared Sahi Agarwal, the co-founder of Enkrypt.
Image/Photo credit: source url
