. . AI: Anthropic Spots A Vulnerability in LLMs and Princeton NLP Unveils Open Source SWE-agent (4.2.24)

Friends, in today's AI news, Anthropic spots a vulnerability in LLMs that should have most providers concerned. We also learn that Princeton NLP has launched an open source competitor to Cognition Lab's Devin, with comparable results. This development, they claim, could potentially streamline the process of code correction and optimization. The iteration and parallelization by the open source community is remarkable. We also see some more news on the algorithmic optimization front along with a serverless API from Hugging Face. Enjoy!

-Marshall Kirkpatrick, Editor

Anthropic Discovers Vulnerability in LLMs Prompting Immediate Mitigation Efforts

First impacted: AI developers, AI safety researchers
Time to impact: Short

Anthropic has identified a vulnerability in LLMs that could potentially lead to harmful outputs, despite safety measures. The company has released a paper detailing a "many-shot jailbreaking" technique and has taken steps to mitigate the issue, including informing other AI developers about the vulnerability. The "many-shot jailbreaking" technique exploits the large context window of LLMs by forcing them to produce potentially harmful responses by including a large number of faux dialogues in a specific configuration, overriding their safety training. [Many-shot jailbreaking] Explore more of our coverage of: Anthropic AI, LLM Vulnerability, AI Safety Measures. Share this story by email

Princeton NLP Launches Open-Source AI Software Engineer Agent

First impacted: Software Developers, Open-source contributors
Time to impact: Medium

Princeton NLP has launched an open-source system, SWE-agent (AI software engineer), which they say can autonomously rectify issues in GitHub repos with similar accuracy to the much-watched Devin on SWE-bench, completing tasks in approximately 93 seconds. The team explains that the system operates through a specialized terminal, is optimized to view 100 lines at a time. [SWE-agent: Agent Computer Interfaces Enable Software Engineering Language Models] Explore more of our coverage of: Princeton NLP, Open-Source System, GitHub Optimization. Share this story by email

Ashkboos' Team Launches QuaRot for 4-bit Inference of LLMs

First impacted: Data scientists, Machine learning engineers
Time to impact: Medium

This next story is for the technical folks in the audience. Saleh Ashkboos and his team have launched QuaRot, a system they say enables 4-bit inference of LLMs by removing outlier features and quantizes KV-cache using 4 and 3-bits with minimal WikiText-2 complexity loss. They also developed CUDA kernels for QuaRot, which they claim can speed up the model, reduce memory and still maintain integrity post quantization, retaining an impressive 99% of the zero-shot accuracy of the baseline model. [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs] Explore more of our coverage of: QuaRot System, 4-bit Inference, CUDA Kernels. Share this story by email

Hugging Face Launches Cloudflare Workers AI Integration to Simplify Deployment of Open Generative AI Models

First impacted: AI Developers, Cloud Engineers
Time to impact: Short

Hugging Face has launched a new integration into their product suite called 'Deploy on Cloudflare Workers AI'. The service allows developers to use open models as a serverless API, powered by GPUs in Cloudflare edge data centers. The company says this feature addresses GPU availability and server deployment costs, and provides serverless access to popular Hugging Face Models with a pay-per-request pricing model. They claim the integration of Hugging Face Models on Cloudflare Workers AI is user-friendly, with step-by-step instructions provided for using various models, including the newest model from Nous Research, Hermes 2 Pro on Mistral 7B. [Bringing serverless GPU inference to Hugging Face users] Explore more of our coverage of: Hugging Face, Cloudflare Workers AI, Serverless API. Share this story by email

Kilpatrick Joins Google, Aims to Boost AI Development

First impacted: AI developers, Google employees
Time to impact: Short

Logan Kilpatrick, former leader of developer relations at OpenAI, has announced that he has joined Google to lead product for AI Studio and support the Gemini API. He says that he is building a team dedicated to AI development, with the aim of making Google the best platform for developers working with AI. [via @OfficialLoganK] Explore more of our coverage of: Google AI, AI Development, Product Leadership. Share this story by email

That’s it! More AI news tomorrow!