. . AI: Scale AI Reveals Overfitting in Models and Lamini Raises Series A (5.2.24)

Scale AI, Lamini, LM Studio

Today's news was light on breakthroughs for a change, but we see some cool reflections in the industry on model performance and costs. We also see a Series A announcement from Lamini, which is helping LLMs become enterprise-ready. Enjoy!

Sasha Krecinic

Scale AI Study Suggests Overfitting in Some LLMs

First impacted: AI researchers, data scientists

New research from Scale AI, detailed in their blog post, indicates that LLMs such as GPT and Claude show minimal overfitting on the GSM8k benchmark. In contrast, some models exhibit significant overfitting, with accuracy declines reaching up to 13%. The study also introduces a new benchmark, GSM1k, where some models performed poorly, suggesting these models might be memorizing data rather than genuinely reasoning, while GPT and Claude demonstrated consistent accuracy. [A Careful Examination of Large Language Model Performance on Grade School Arithmetic] Share this story by email

Lamini Secures $25 Million for AI Development For Enterprise-Specific Enhancements and Efficiency

First impacted: AI researchers, Enterprise

Lamini, an enterprise AI platform, says it has raised $25 million in a Series A funding round led by Amplify Partners to enhance its LLM capabilities and infrastructure efficiency. According to the company, they have reduced LLM training times from 1,000 hours to just 1 hour, and they claim to have reduced model hallucinations as well as enhancing secure deployment options. [Lamini Raises $25M For Enterprises To Develop Top LLMs In-House | Lamini - Enterprise LLM Platform] Share this story by email

Back-of-the-envolope Llama 3 Model Cost Breakdown

First impacted: AI researchers, Enterprise

An interesting post back-solving the cost of running Llama on an M2 Ultra. [via @awnihannun] Share this story by email

LM Studio Launches New Command-Line Tool

[GitHub - lmstudio-ai/lms: LM Studio in your terminal] Share this story by email