This is a super cool leaderboard for lossless text compression via NLP (and yes, that includes AI)! The top solution manages to compress the first GB of the English Wikipedia to a whopping 10.7% of its original size, including the compression program itself!

Hyper Text Compression: Shrinking Wikipedia to 10.7% of its Size!

Automatic TLDR by Gemini 2.5 Pro:

This page describes the Large Text Compression Benchmark, an open competition that ranks lossless data compression programs. The primary goal is to encourage research in artificial intelligence (AI) and natural language processing (NLP) by treating text compression as a language modeling problem.

Benchmark Overview

• Test Data: The benchmark uses the first $10^9$ bytes (1 GB) of an English Wikipedia XML dump from March 3, 2006, known as enwik9.
• Ranking Metric: Programs are ranked solely by the total size, which is the sum of the compressed enwik9 file size and the size of the zipped decompresser program. A smaller total size is better.
• Secondary Information: Data such as compression/decompression speed and memory usage are provided for informational purposes but do not influence the rankings.
• Goal: The benchmark’s main purpose is not to find the best general-purpose compressor but to push the boundaries of data modeling, a fundamental challenge in both AI and compression.

Key Findings and Algorithms

The results table shows a wide variety of compression programs, ranked from the best compression ratio to the worst. A clear trend emerges from the top-performing entries:

• Dominance of AI Models: The highest-ranking compressors, such as nncp and cmix, utilize sophisticated AI-based algorithms. These include neural network models like Transformers (Tr) and Long Short-Term Memory (LSTM), as well as advanced Context Mixing (CM) techniques. These methods excel at modeling the complex patterns in natural language text, resulting in superior compression ratios.
• Trade-offs: There is a significant trade-off between compression ratio, speed, and memory. The top-ranked AI-driven compressors are extremely slow and require vast amounts of memory (often many gigabytes) and, in some cases, specialized hardware like GPUs.
• Traditional Algorithms: More conventional algorithms like Lempel-Ziv (LZ), Burrows-Wheeler Transform (BWT), and Prediction by Partial Match (PPM) are found further down the list. While they are generally much faster and use less memory, they cannot achieve the same level of compression as the leading AI models on this specific text-based task.

Hutter Prize

The benchmark is closely related to the Hutter Prize, which offers prize money for open-source compression improvements on a smaller subset of the data (enwik8, the first $10^8$ bytes). This prize has specific hardware and time constraints, encouraging practical advancements in the field.

As i coded on OuroborosDB i noticed that i need a very fast queue for a rather unique architectural design decision.
I try to build the network module in such a way that i can test the behavior completely deterministic while “simulating” entire clusters in a single process.

So i build a test prototype of my global network queue with go’s channels and noticed that it was a major performance bottleneck, after writing 2 different ring buffer queue implementations it became clear that some queues behave completely different under different congestion levels and core counts - some so unpredictable that i just did not wanted to use them in my project.

This prompted me to take a relatively large chunk out of my free time and write a suite to benchmark different queue implementations i build under different conditions and score them based on their performance and predictability.

The result of this work is GoQueueBench

These are the results of the benchmark suite:

Please note that i build the package so that all queue adhere to the same interface and can be swapped out easily.

This is a article that explores the capabilities of OpenAI’s o3 model. HN

The impacts of AI reasoning are getting closer and closer to surpassing human capabilities.
But running it to solve a problem is extremely expensive.

Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of “programs” (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.

(programs is defined here as “My mental model for LLMs is that they work as a repository of vector programs”)

As far as i understand this and it’s implications i appears to me this process is more like a directed brute force with many candidates and selects the best one.
There is a thread on HN that discusses this topic.

GPT-4 Summary

OpenAI’s new o3 system has achieved a groundbreaking 75.7% on the ARC-AGI-Pub Semi-Private Evaluation within a $10k compute limit, surpassing previous models. A high-compute version scored 87.5%, marking a major step in AI’s adaptability to novel tasks. This success highlights a shift from scaling existing architectures to innovative mechanisms for generalization and test-time knowledge recombination.

Key achievements:

• ARC-AGI performance: o3 shows unprecedented adaptability, unlike previous GPT-family models, which struggled with novel tasks.
• Efficiency challenges: Low-compute costs $17–$20 per task but high-compute performance remains expensive and exploratory.
• Core innovation: o3 employs natural language program search, generating and executing task-specific solutions during runtime, guided by deep learning priors.

Despite its advancements, o3 is not AGI, as it still fails simple tasks. Upcoming benchmarks, such as ARC-AGI-2 in 2025, aim to further challenge AI systems while encouraging open-source progress.

This breakthrough underscores a qualitative leap in AI research, reshaping paths toward AGI and sparking broader scientific engagement.