. . AI: Google's self-improving AI agent, OpenAI's safety drills, and more (12.18.23)

Friends, did you know you can search the archives of these posts for keywords very easily on the home page of this newsletter? I must have done so 10 times today while helping someone plan for how they could improve some of their AI processes. Grab those MEDPrompt write-ups, remind yourself of the Mistral and Perplexity API specifics! Easy to do.

Today we’ve got just 4 stories that rose to the top of our analysis of the AI community dialogue, and I’ve put them in a very specific order. I think they tell a fascinating story.

I hope they are interesting and useful to you.

And now here’s today’s news,

Marshall Kirkpatrick, Editor

Andrew Ng Questions Hypothetical Harm Focus

First impacted: AI specialists, AI industry professionals
Time to impact: Medium to long

AI community leader Andrew Ng has voiced concerns over the AI industry's disproportionate focus on potential future issues, arguing that it detracts from addressing present, tangible problems. According to Ng, while it's important to consider future problems, the primary focus should be on resolving current issues rather than speculative ones. This is something Ng has been exploring for some time and that many people agree with him on.

• Ng wonders if any other engineering field focuses more on theoretical issues rather than real, existing ones.

• He criticizes the AI sector for basing many of their future harm predictions on dubious theories about technology that doesn't yet exist. [via @AndrewYNg] Share by email

Google Develops Self-Improving AI Agent

First impacted: AI researchers, software developers
Time to impact: Short

Google Research and Google DeepMind have developed a Large Language Model (LLM) agent that can reason and act based on external knowledge, according to a research paper. The researchers say that after two iterations of their ReST-like method, they've produced a smaller model with fewer parameters that still performs well on challenging question-answering benchmarks.

• They say the model is capable of self-improvement through a method that repeatedly trains on previous trajectories.

• The team claims to have generated synthetic data that can be used to distill the agent into models one or two orders of magnitude smaller. These models reportedly perform comparably to the pre-trained teacher agent, and without the use of any human-labeled training data [ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent] Share by email

OpenAI Launches Preparedness Framework for AI Risk Management

First impacted: AI safety researchers, AI model developers
Time to impact: Medium to long

OpenAI has launched its Preparedness Framework (Beta), a system designed to manage potential risks from powerful AI models, according to a company blog post. The company says the framework includes a rigorous evaluation process, risk limits, safety measures, and a dedicated team to oversee safety decisions, with a focus on data-driven forecasts and tracking misuse in real-world scenarios.

• The Preparedness Framework plans to consistently update their model "scorecards", aiming to push these models to their maximum potential in order to evaluate risks and gauge the success of suggested risk reduction strategies. Interesting.

• The Preparedness Team intends to carry out regular safety drills to test resilience against the demands of their business and their own organizational culture, with plans to have audits performed by competent, independent third parties. Fun. [Preparedness] Share by email

Wolf: New AI Unicorn Companies to Emerge in 2024

First impacted: AI startup founders, Investors in AI technology
Time to impact: Short

Thomas Wolf predicts that in 2024, there will be at least 10 new unicorn companies focusing on state-of-the-art open foundation models, while older AI-unicorn companies may face challenges. He also suggests that evaluating model quality will become increasingly difficult, academia will regain prominence, annotation companies could face difficulties, and synthetic data will gain importance. [via @Thom_Wolf] Share by email

That’s it! More AI stories tomorrow.