This is a article that explores the capabilities of OpenAI’s o3 model. HN

The impacts of AI reasoning are getting closer and closer to surpassing human capabilities.
But running it to solve a problem is extremely expensive.

Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of “programs” (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.

(programs is defined here as “My mental model for LLMs is that they work as a repository of vector programs”)

As far as i understand this and it’s implications i appears to me this process is more like a directed brute force with many candidates and selects the best one.
There is a thread on HN that discusses this topic.

GPT-4 Summary

OpenAI’s new o3 system has achieved a groundbreaking 75.7% on the ARC-AGI-Pub Semi-Private Evaluation within a $10k compute limit, surpassing previous models. A high-compute version scored 87.5%, marking a major step in AI’s adaptability to novel tasks. This success highlights a shift from scaling existing architectures to innovative mechanisms for generalization and test-time knowledge recombination.

Key achievements:

• ARC-AGI performance: o3 shows unprecedented adaptability, unlike previous GPT-family models, which struggled with novel tasks.
• Efficiency challenges: Low-compute costs $17–$20 per task but high-compute performance remains expensive and exploratory.
• Core innovation: o3 employs natural language program search, generating and executing task-specific solutions during runtime, guided by deep learning priors.

Despite its advancements, o3 is not AGI, as it still fails simple tasks. Upcoming benchmarks, such as ARC-AGI-2 in 2025, aim to further challenge AI systems while encouraging open-source progress.

This breakthrough underscores a qualitative leap in AI research, reshaping paths toward AGI and sparking broader scientific engagement.

I stumbled on HN over this very interesting article about a new kind of context memory system that, is able to remove information that is “unhelpful or redundant details”.

Thinking further, i think this would be super helpful for semantic search, that is currently not very performant due to the missing filters that extract importance. I have tried to counter this problem until now via summarization through small LLMs, but as one might guess turns out as not very precise and super expensive. There are other ideas one could post process text with LLMs but they are not very efficient either.

Paper https://arxiv.org/abs/2410.13166

TLDR by GPT-4o

(the article is very good, you might to prefer to read it over this TLDR, but here is it anyway)

Sakana AI introduces Neural Attention Memory Models (NAMMs), a novel memory system for transformers inspired by human selective memory. NAMMs optimize how transformers store and retrieve information, enabling them to “remember” important tokens and “forget” redundant ones, significantly improving efficiency and performance, particularly for long-context tasks.

Key Technical Highlights:

1. Evolutionary Optimization: NAMMs use evolutionary algorithms to train a neural classifier that decides which tokens to keep or discard, bypassing non-differentiable challenges.
2. Execution Steps:
   • Convert attention sequences into spectrograms.
   • Compress data using an exponential moving average (EMA).
   • Use a classifier to score and selectively prune tokens.
3. Generalization:
   • NAMMs can zero-shot transfer to other transformers (e.g., vision, reinforcement learning) without retraining.
   • They adapt to tasks differently—retaining global information in early layers and focusing on local details in later ones.
4. Performance: Tested on LongBench, InfiniteBench, and their Japanese benchmark ChouBun, NAMMs reduce memory usage and outperform prior hand-designed memory strategies (H₂O, L₂).

NAMMs demonstrate cross-domain mastery and efficiency gains across diverse tasks and input modalities, paving the way for future research into learning transformers directly atop evolved memory systems.