. . AI: The 7 most-important stories of the past week

Author

Marshall Kirkpatrick
January 02, 2024

Friends, over the past holiday week, thousands of news stories have been discussed across the AI community. Our data and my editorial judgement says these were the 7 that were most important and most interesting.

Welcome to our new readers, including those of you joining us from global innovation expert Gianni Giacomelli’s New Year’s Resolution to “read only good newsletters.” It was an honor to be included in that short list.

One other link I want to share before getting to today’s news round up is this one to a very short story from Jack Clark about the use of LLMs to predict not the most likely events, but unlikely events that are worth preparing for. I found it inspiring and promptly asked ChatGPT to sketch out a list of 1st and 2nd order consequences of a scenario, including some 2nd order ones with only a 1/1000 chance of happening but of great significance if they did happen. That was fun.

And now here’s today’s top stories, a round-up of the 7 most significant and interesting stories in AI over the holiday week.

"Society of Mind" Wins NeurIPS2023 Best Paper

First impacted: AI researchers, AI developers
Time to impact: Short

Mingchen Zhuge, Haozhe Liu, and an international team, have developed a method inspired by Minsky's "society of mind" and Schmidhuber's "learning to think", winning the Best Paper award at NeurIPS2023. Schmidhuber's summary: "Up to 129 foundation models collectively solve practical problems by interviewing each other in monarchical or democratic societies." These "societies" tackled real-world AI challenges such as visual question answering, image captioning, text-to-image synthesis, 3D generation, egocentric retrieval, embodied AI, and general language-based task solving. [Mindstorms in Natural Language-Based Societies of Mind] Explore more of our coverage of: NeurIPS Awards, Neural Network Economies, Zero-Shot Reasoning. Share this story by email

First impacted: AI researchers, Data scientists
Time to impact: Short

Moscow researchers Artyom Eliseev and Denis Mazur have developed a strategy to operate large Mixture-of-Experts (MoE) language models on standard computer hardware, such as desktops and free-tier Google Colab instances, which are known for their limited accelerator memory. According to their work, they built upon parameter offloading algorithms and proposed a new strategy that accelerates offloading, allowing the operation of Mixtral-8x7B with mixed quantization on less powerful hardware, a task previously thought to require high-end GPUs. [Fast Inference of Mixture-of-Experts Language Models with Offloading] Explore more of our coverage of: Mixture-of-Experts Models, Parameter Offloading, Low-End Hardware AI. Share this story by email

First impacted: AI developers, AI researchers
Time to impact: Medium

Microsoft has launched an open-source library, PromptBench, which is designed to evaluate LLMs. According to a research paper by Microsoft, the tool, initially developed for adversarial prompt attack, now covers the entire LLM evaluation process and is expected to facilitate the development of new benchmarks, applications, and evaluation protocols. [PromptBench: A Unified Library for Evaluation of Large Language Models] Share this story by email

First impacted: AI researchers, Chatbot developers
Time to impact: Medium

Nous Research has launched its new "uncensored" AI model, the Nous Hermes 2 - Solar 10.7B, on HuggingFace. It's trained on 1,000,000 entries of primarily GPT-4 generated data. The model uses ChatML for the prompt format and its benchmarks are nearly as good as the group's "state of the art fine tuned Yi 34B model," while being only a third of the size. [NousResearch/Nous-Hermes-2-SOLAR-10.7B · Hugging Face] Explore more of our coverage of: AI Models, Yi, synthetic data. Share this story by email

First impacted: AI researchers, AI developers
Time to impact: Long

In a new blog post, podcaster Dwarkesh Patel predicts that by 2040, we could achieve AGI by scaling up LLMs, despite the ongoing debate about the feasibility and challenges of this approach. The post presents a hypothetical conversation between two characters, "Believer" and "Skeptic", where the Believer suggests that synthetic data and self-play could address data shortages and enhance performance, while the Skeptic questions whether these improvements truly indicate progress towards AGI.

  • The Skeptic character posits that AI would need 100,000 times more data than currently available to achieve human-level cognition, even considering the potential benefits of data efficient algorithms and multimodal training.

  • In response, the Believer character suggests that the synthetic data bootstrapping process, which mirrors human evolution in its ability to enhance model intelligence for interpreting complex symbolic outputs, could potentially address the data scarcity issue.

[Will scaling work?] Explore more of our coverage of:AGI. Share this story by email

First impacted: AI researchers, newcomers to AI research
Time to impact: Long

Stella Biderman, an AI specialist at Booz Allen, says she will present a new difficult research question each week in 2024, with the goal of demonstrating that influential LLM research is not exclusive to large laboratories or already crowded topics. People are into it. [via @BlancheMinerva] Explore more of our coverage of: LLM Research, Independent Researchers. Share this story by email

Sebastian Raschka:10 Noteworthy AI Research Papers of 2023

First impacted: AI researchers, Machine learning specialists
Time to impact: Long

ML researcher Sebastian Raschka writes: "This year has felt distinctly different. I've been working in, on, and with machine learning and AI for over a decade, yet I can't recall a time when these fields were as popular and rapidly evolving as they have been this year. To conclude an eventful 2023 in machine learning and AI research, I'm excited to share 10 noteworthy papers I've read this year." [10 Noteworthy AI Research Papers of 2023] Explore more of our coverage of: Research papers. Share this story by email

That’s it! More AI news tomorrow!