AI and infrastructure news
Filter by type: All169
- 17 days ago x.com
URM combines recurrence and nonlinearity for reasoning
A post on Universal Transformer research attributes reasoning gains to recurrent inductive bias and strong nonlinearity, and describes design choices for local token mixing and training.
- 17 days ago x.com
AHN proposes continuous memory for long-context LLMs
On October 9, 2025, a ByteDance publication introduced Artificial Hippocampus Networks, an architecture that continuously compresses information outside a language model's context window.
- 17 days ago
arxiv.orgCompressed attention to reduce costs in long-context models
Published on October 7, 2025, the paper presents an approach that compresses queries, keys, and values to perform attention in a smaller latent space, targeting compute and KV-cache costs.
- 17 days ago x.com
Attention and kernel regression: a useful connection
A 10-slide summary published on October 4, 2025 examines the relationship between kernel regression and attention, presenting this perspective as an aid to understanding the mechanism.
- 17 days ago x.com
CCA: attention in a compressed latent space
A post dated October 7, 2025 describes an architecture that computes attention in a smaller latent space and reports lower memory, parameter, and operation counts in tests on H100.
- 17 days ago x.com
Model circuits can help predict reasoning errors
A post published on October 13, 2025 describes a method using attribution graphs to identify incorrect reasoning on arithmetic tasks, with improved detection metrics.
- 17 days ago x.com
Balancing adaptation and memory in model training
A study published on October 5, 2025, reports that a model trained on 12,000 small tasks combined new ideas and notes distinct effects from structured and mixed lessons.
- 17 days ago x.com
How to observe information flow in LLMs
Published on October 9, 2025, the publication proposes three stages for information flow in LLMs and links massive activations to attention sinks and compression valleys.
- 17 days ago arxiv.org
Continuous Thought Machines explore neuron-level dynamics
Published on October 6, 2025, the paper presents a model using neuron-level processing and synchronization as central representations, inspired by biological neural dynamics.
- 17 days ago x.com
Short reasoning hints for evaluating AI systems
A paper published on October 5, 2025 describes a hint generator and a solver trained with reinforcement learning. For mathematics, it reports 44% higher accuracy on AIME 2025 than RL with long chains.
- 17 days ago x.com
Dragon Hatchling explores local memory in language models
Published on October 3, 2025, the post describes an architecture with simple neurons and connections that store short-term memory through Hebbian learning.
- 17 days ago x.com
Dragon Hatchling proposes an interpretable LLM architecture
Published on October 1, 2025, the post describes an LLM architecture based on locally interacting neural particles and says it rivals GPT-2 performance.
- 17 days ago x.com
Study compares attention in 13 LLMs, with 17% performing best
Published on October 12, 2025, the study reports experiments training 13 LLMs with different proportions of conventional attention and DeltaNet linear attention. The configuration with 17% attention—two of 12 layers—achieved the best result.
- 17 days ago arxiv.org
Gradient descent viewed as attention over examples
Published on February 6, 2023, the article interprets gradient-trained linear layers as memories of training examples and describes predictions as unnormalized attention over training experience.
- 17 days ago x.com
Grokking in modular addition: three stages to generalization
A post dated October 4, 2025 describes a paper proposing three stages of grokking in modular addition and a data threshold near n log n, with weight decay and moderate width supporting feature learning.
- 17 days ago arxiv.org
MoE-CL proposes continuous LLM adaptation
Published on October 6, 2025, the paper presents a continuous tuning method using LoRA experts and reports evaluations on the MTL5 and Tencent3 benchmarks, plus an A/B test.
- 17 days ago x.com
What attention sinks reveal about LLMs
Experiments on models from 410 million to 120 billion parameters link extreme activation norms of the initial token to compression valleys and attention sinks.
- 17 days ago x.com
RLAD explores reasoning abstractions with reinforcement learning
Published on October 3, 2025, the paper presents RLAD, a framework using two-player reinforcement learning to help language models discover procedural hints in natural language.
- 17 days ago x.com
RLMs: an approach to very long prompts
Published on October 15, 2025, the publication describes language models that interact recursively with long prompts through a REPL and reports evaluations on the OOLONG and BrowseComp-Plus benchmarks.
- 17 days ago x.com
SwiReasoning switches reasoning modes to reduce tokens
A publication dated October 12, 2025 describes a method that switches between latent and explicit reasoning and reports token-efficiency gains on Qwen3-8B models.
Energy-based transformers refine token predictions
A paper published on September 27, 2025 reports that 44-million-parameter Energy-Based Transformers outperformed vanilla transformers on three of four benchmarks using RedPajama-Data-v2.
- 17 days ago x.com
A unified view of Attention-MoE and FFN-MoE
A paper published on September 24, 2025 proposes treating pre-mixed tokens and (V·Wo) as an FFN expert, unifying sparse attention and FFN approaches.
Agents optimize GPU attention kernels
A September 28, 2026 NVIDIA post reports attention kernels up to 2.96 times faster than FlashKDA on B300, with one tenth the state error.
Algorithms in transformer weights: practical applications
Published on June 24, 2026, the article about The Transformer Cookbook describes encoding addition, search, and branching algorithms in transformer weights, following work on RASP.
Continual learning in LLMs with iSDFT
A repository published on September 22, 2026 presents a self-distillation method for continual learning in LLMs, with training code, checkpoints, and evaluations.
Continuous learning on a notebook with 8 GB of VRAM
The mini-AGI repository describes a model trained from scratch on a notebook with 8 GB of VRAM, using batch size 1, disk-paged weights, and learning-rate adjustments to reduce forgetting.
- 18 days ago x.com
Hybrid architectures beyond standard attention
A post published on July 8, 2026, compares the architectures of Qwen 3.5 and Nemotron 3 and describes linear attention and state-space models as examples of convergent evolution.
- 18 days ago arxiv.org
Attention in long-context Transformers
Published on September 15, 2026, the paper analyzes attention rank collapse as context grows and studies rescaling scores by a polylogarithmic factor.
- 18 days ago arxiv.org
Natively trainable sparse attention for long contexts
An article published on September 26, 2026, presents NSA, a sparse attention approach combining algorithmic innovations and hardware-aligned optimizations for more efficient long-context modeling.
- 18 days ago x.com
Sparse attention can optimize KV block selection
A Tencent publication dated September 14, 2026 describes a method that uses language modeling loss to learn which KV blocks each query should access.
- 18 days ago x.com
Dynamic linear attention adapts memory to token information
A post published on June 15, 2026, describes Dynamic Linear Attention adapting memory states to token information while keeping a fixed cache, and reports gains in benchmarks, throughput, and memory use.
- 18 days ago x.com
Linear and sparse attention: evaluating long-context models
A post dated August 26, 2026, claims GLM-5.3 uses three linear-attention layers for each sparse-attention layer, highlighting compute-efficiency trade-offs for long contexts.
On-Demand Attention aims to balance long context and speed
A publication dated September 20, 2026 describes On-Demand Attention, which uses a lightweight head to trigger full context when needed, aiming to speed decoding while recovering much of the performance lost with local attention.
- 18 days ago arxiv.org
Attention sinks and rescaling in transformers
Published on October 5, 2026, the paper examines attention sinks and residual sinks in large language models and proposes how these outliers, softmax attention, and RMSNorm rescale other components.
BF16 in FlashAttention: monitor gradient stability
On September 29, 2026, a technical work reported that BF16 rounding in FlashAttention can break a softmax gradient symmetry and cause explosions near the end of training. The authors propose GProj as a correction.
- 18 days ago x.com
A 64-token cache for linear attention: what to evaluate
A post about HOLA, published on July 5, 2026, describes a parameter-free token-surprise metric and a 64-token cache to address catastrophic forgetting in linear-attention models.
- 18 days ago arxiv.org
Sparse layers and looping for scaling language models
An article published on September 26, 2026 compares standard Transformers and Mixture-of-Experts, with and without looping, and reports a scaling advantage for Looped-MoE models.
- 18 days ago arxiv.org
Evaluating efficient attention in hybrid models
Published on June 16, 2026, the paper examines sliding-window attention and recurrent sequence mixers in hybrid language models, focusing on scalability, mechanisms, and architecture decisions.
- 18 days ago arxiv.org
How to assess autoregressive inference bottlenecks
An ICLR 2026 paper addresses a decoding bottleneck in probabilistic Transformer models. Companies can measure inference performance and identify opportunities for optimization.
How to assess LLM access to knowledge in its parameters
In a June 28, 2026 publication, the Google Research blog examines how reasoning may help language models access knowledge stored in their parameters. The question can inform internal tests of models and prompts.
Let's assess your project
I reply with scope and a proposal.