AI and infrastructure news
Filter by type: Paper77
- 19 days ago
arxiv.orgBF16 gradients rise late in training
A paper reports a 1,000-fold increase in gradient norm after 25 billion tokens in a transformer trained with FlashAttention-3 in BF16.
- 19 days ago arxiv.org
HiLS learns chunk selection for sparse attention
Published on July 7, 2026, the paper presents HiLS, an approach that learns chunk selection for sparse attention using the language-modeling loss.
- 19 days ago arxiv.org
HOLA pairs linear attention with an exact, bounded KV cache
Published on July 3, 2026, the paper presents HOLA, an architecture combining recurrent state and a bounded key-value cache, with reported gains in perplexity and long-range recall.
- 19 days ago arxiv.org
HOLA combines recurrent state and a limited cache for long contexts
A post dated July 6, 2026 describes HOLA, which uses recurrent state and a small KV cache for exact memory. The method reduces perplexity by 16.1% and maintains robust retrieval at 32k tokens.
- 19 days ago arxiv.org
HySparse2 cuts long-context costs through KV reuse
Published on September 23, 2026, the post describes two levels of KV sharing to reduce prefill FLOPs and cache size, with better retrieval scores than MiMo-V2.6's Hybrid SWA at a context length of 1 million tokens.
- 19 days ago arxiv.org
iLLaDA: masked diffusion in an 8B model
A description published on June 25, 2026 presents iLLaDA, an 8B masked-diffusion language model trained from scratch with fully bidirectional attention.
- 19 days ago arxiv.org
Interpretability beyond human concepts
Published on September 22, 2026, the paper investigates whether language models use internal representations that do not map adequately to familiar human concepts.
- 19 days ago arxiv.org
Jamba combines Transformer, Mamba and MoE
The paper, published on September 20, 2026, describes a hybrid architecture that interleaves Transformer and Mamba blocks and uses mixture-of-experts in some layers.
- 19 days ago arxiv.org
KATA explores spherical packing for recall in linear attention
A paper published on September 16, 2026 presents KATA, a geometric approach that formulates associative recall in linear attention as a spherical packing problem.
- 19 days ago arxiv.org
KITE expands Transformers without increasing KV computation
Published on September 29, 2026, the material describes KITE, an approach that expands decoder components by reusing a KV cache generated by a smaller prefiller.
- 19 days ago arxiv.org
LoopCD uses previous loops in decoding
Published on October 2, 2026, the paper presents LoopCD, a training-free method that uses representations from previous loops to guide token selection in Looped Transformers.
- 19 days ago arxiv.org
Looped flows train recurrent updates with local denoising
A paper published on September 12, 2026 proposes local denoising objectives to train recurrent hidden-state updates, including when backpropagation covers few steps.
- 19 days ago arxiv.org
Mamba and Transformers: a connection behind the architectures
An article published on September 24, 2026 explores theoretical connections between state-space models such as Mamba and attention variants.
- 19 days ago arxiv.org
Megalodon and long-sequence evaluation
An article published on September 24, 2026 presents an architecture based on Mega, designed for efficient pretraining and inference with unlimited context.
- 19 days ago arxiv.org
Sparse memory expands RNN state without raising compute per token
Published on July 11, 2026, the post describes Sparse Delta Memory, which combines an RNN with explicit sparse-addressed memory to increase state capacity while keeping compute per token constant.
- 19 days ago arxiv.org
Memory Attention proposes token memory for calculating values
In a post published on September 25, 2026, Memory Attention combines layer-specific token memory with contextual keys to form attention values, with potential KV cache reduction and CPU offloading.
- 19 days ago arxiv.org
MiniCPM-SALA combines sparse and linear attention for long contexts
A paper presents a hybrid architecture with 9 billion parameters and reports good results on long-context benchmarks, with scalable inference up to 1 million tokens.
- 19 days ago arxiv.org
Mixture-of-Depths adjusts compute per token in Transformers
An article dated September 22, 2026 presents a method that allocates compute across sequence positions and limits token participation in each layer.
- 19 days ago arxiv.org
Native Hybrid Attention and the long-context trade-off
An article published on October 5, 2026 presents Native Hybrid Attention, an approach aimed at balancing speed in long contexts with detail retention. Companies can assess the proposal using benchmarks tied to their tasks.
PolaFormer++ proposes polarity-aware features for linear attention
The paper dated September 26, 2026 presents a theoretical criterion for feature-map spikiness and evaluates linear attention across six families of visual tasks.
- 19 days ago arxiv.org
Executable programs for studying Transformer attention heads
A paper published on June 29, 2026 proposes approximating deep-network components with executable programs, applying the method to attention heads in Transformer language models.
- 19 days ago arxiv.org
PTXBench measures performance of AI-generated GPU kernels
Published on September 29, 2026, PTXBench evaluates matrix multiplication and attention kernels in architecture-specific PTX on H100 and B200 GPUs, measuring correctness, speed, and whether requested instructions are actually executed.
- 19 days ago arxiv.org
Quiet-STaR: training models to reason before responding
A paper published on September 22, 2026 describes an approach for training language models to generate internal reasoning before visible text, based on the idea that text often contains implicit reasoning.
- 19 days ago arxiv.org
Tracing training data to LLM circuits
Published on July 7, 2026, the paper presents a method for linking training samples to interpretable units in language models and examines effects on attention heads and in-context learning.
RESA: recovering context with sparse attention
Published on September 29, 2026, the RESA paper proposes using residual estimation to recover context ignored by sparse attention, with no additional computational cost, according to the paper.
- 19 days ago arxiv.org
RetNet proposes recurrence as an alternative to attention
Published on September 24, 2026, the Retentive Network paper presents a language-model architecture that combines parallel training with recurrent inference.
- 19 days ago arxiv.org
RWKV combines parallel training with recurrent inference
An article published on September 20, 2026 describes an architecture combining Transformer-style parallel training with RNN-style recurrent inference, with memory and computation scaling linearly with sequence length.
- 19 days ago arxiv.org
Softmax Linear Attention seeks better selection at linear cost
Published on September 16, 2026, the paper proposes applying softmax at the head level to restore global competition in linear attention while retaining linear complexity.
- 19 days ago arxiv.org
Deep learning theory for analyzing model behavior
A book published on June 24, 2026, presents a theoretical framework linking deep neural network architecture and training to model outputs.
Transformer: from attention to implementation
Chapter 8 of Speech and Language Processing covers Transformer components from intuition to matrix formulation, providing a foundation for assessing how the architecture relates to business applications.
- 19 days ago arxiv.org
Transformers: attention and computational costs
An article published on August 2, 2026 explains the mathematical components of Transformers and discusses techniques to reduce attention processing and memory costs.
- 19 days ago arxiv.org
Looped Transformers: greater depth without more parameters
An article dated July 17, 2026 examines how looped Transformers reuse blocks to increase unfolded depth and how parameter sharing affects residual scaling.
- 19 days ago arxiv.org
Looped Transformers: reducing redundancy and cost
A paper published on September 26, 2026, identifies three types of computational redundancy across loops in looped Transformers. The authors report 1.65× lower latency and 6× lower memory use.
- 19 days ago arxiv.org
Recurrent Transformers as programmable computers
An article dated September 17, 2026 describes recurrent Transformer layers that emulate basic computing operations, including conditional branches and program counters.
- 19 days ago arxiv.org
Block-recurrent Transformers target linear scaling
An article published on September 17, 2026 presents a recurrent, block-based Transformer architecture with complexity linear in sequence length.
- 19 days ago arxiv.org
Tucker Attention frames attention variants through low-rank factorization
Published on September 16, 2026, the paper presents Tucker Attention as a tensor factorization approach that frames MHA, GQA, and MLA as specialized low-rank approximate attention methods.
- 19 days ago arxiv.org
Universal Transformers reuse one layer recurrently
A post dated September 17, 2026 describes an architecture that repeatedly applies the same Transformer layer instead of stacking distinct layers, and discusses sequential and parallel computation in sequence models.
Let's assess your project
I reply with scope and a proposal.