AI and infrastructure news
Filter by type: All169
LiFT expands iterative inference with a shared DiT core
Published on October 6, 2026, the research reports that LiFT outperforms a dense DiT baseline on ImageNet 256×256 with about 60% fewer parameters.
- 18 days ago
arxiv.orgLoopCD uses previous loops in decoding
Published on October 2, 2026, the paper presents LoopCD, a training-free method that uses representations from previous loops to guide token selection in Looped Transformers.
LoopCD refines Transformer decoding without additional training
The paper, published on October 2, 2026, presents a technique that compares predictions from early and late recurrent states to refine decoding. It reports higher accuracy on AIME 2024 with Ouro-2.6B-Thinking.
Looped-DiT reuses blocks to reduce computation
A post published on October 1, 2026, says Looped-DiT outperformed a model 6.5 times larger on image-generation benchmarks while using 4.9 times less inference computation.
- 18 days ago arxiv.org
Looped flows train recurrent updates with local denoising
A paper published on September 12, 2026 proposes local denoising objectives to train recurrent hidden-state updates, including when backpropagation covers few steps.
- 18 days ago arxiv.org
Mamba and Transformers: a connection behind the architectures
An article published on September 24, 2026 explores theoretical connections between state-space models such as Mamba and attention variants.
Matryoshka Attribution identifies weights linked to model behavior
In a post published on September 28, 2026, Matryoshka Attribution is presented as a technique for ranking neural representations and weight changes by their influence on behavior.
- 18 days ago arxiv.org
Megalodon and long-sequence evaluation
An article published on September 24, 2026 presents an architecture based on Mega, designed for efficient pretraining and inference with unlimited context.
Shared memory reduces memory use in recurrent Transformers
An article published on October 5, 2026 reports a 76–79% reduction in context memory and improved quality in models with 150 million to 1 billion parameters.
- 18 days ago arxiv.org
Sparse memory expands RNN state without raising compute per token
Published on July 11, 2026, the post describes Sparse Delta Memory, which combines an RNN with explicit sparse-addressed memory to increase state capacity while keeping compute per token constant.
Token-indexed memory changes attention in Transformers
A post published on September 25, 2026 describes an architecture that sums contextual keys and token-indexed, layer-specific memory vectors, and allows the memory to reside in CPU memory.
- 18 days ago arxiv.org
Memory Attention proposes token memory for calculating values
In a post published on September 25, 2026, Memory Attention combines layer-specific token memory with contextual keys to form attention values, with potential KV cache reduction and CPU offloading.
- 18 days ago x.com
Memory Attention: reusing keys in V=K+M
A publication dated September 27, 2026 reports preliminary ablation results suggesting V=K+M is an effective architectural choice; validation at larger scales remains necessary.
- 18 days ago arxiv.org
MiniCPM-SALA combines sparse and linear attention for long contexts
A paper presents a hybrid architecture with 9 billion parameters and reports good results on long-context benchmarks, with scalable inference up to 1 million tokens.
- 18 days ago arxiv.org
Mixture-of-Depths adjusts compute per token in Transformers
An article dated September 22, 2026 presents a method that allocates compute across sequence positions and limits token participation in each layer.
- 18 days ago x.com
Flow models refine reasoning in stages
A paper dated September 17, 2026 turns a flow model into a recurrent model: predictions are fed back for iterative refinement, with a local loss at each step and no backpropagation through time.
Diffusion language models: techniques and applications
Published on July 8, 2026, the Kuleshov Group post presents diffusion language models and discusses masked diffusion, iterative refinement, post-training, and variable-length generation.
- 18 days ago arxiv.org
Native Hybrid Attention and the long-context trade-off
An article published on October 5, 2026 presents Native Hybrid Attention, an approach aimed at balancing speed in long contexts with detail retention. Companies can assess the proposal using benchmarks tied to their tasks.
- 18 days ago x.com
NextLat explores predicting the next latent state
A June 16, 2026 publication describes a transformer training method and says it may enable compact world models and up to 3.3× faster inference.
The role of chain of thought in serial tasks
An article dated October 1, 2026, examines whether chain of thought contributes computation in Transformers and reports reduced accuracy when it is removed from serial tasks.
OPAL lowers attention costs in long contexts
A September 29, 2026 post describes OPAL, a method that trains linear attention models on their own outputs and reports full retrieval recovery and 83–93% of mathematical reasoning performance.
OpenMythos explores a recurrent-depth Transformer
Published on June 19, 2026, the repository presents an open-source theoretical reconstruction featuring recurrent depth, MoE routing, and adaptive halting per token.
PC-ALM trains residual MLPs up to 1,000 layers deep
Published on September 14, 2026, the paper presents PC-ALM, a local learning method for deep networks with performance described as nearly equivalent to backpropagation.
PolaFormer++ proposes polarity-aware features for linear attention
The paper dated September 26, 2026 presents a theoretical criterion for feature-map spikiness and evaluates linear attention across six families of visual tasks.
- 18 days ago x.com
Post points to efficient designs in frontier Chinese models
Published on August 26, 2026, the post says frontier Chinese models are adopting linear or sparse attention, specialized residual designs, and Muon. It also presents open-source collaboration as a strategy for efficiency.
- 18 days ago arxiv.org
Executable programs for studying Transformer attention heads
A paper published on June 29, 2026 proposes approximating deep-network components with executable programs, applying the method to attention heads in Transformer language models.
- 18 days ago x.com
Synthesized programs could replace some attention heads
A post published July 1, 2026 describes Python programs synthesized from attention maps, reporting 69–79% mean IoU and little QA decline when replacing up to 30–40% of heads.
- 18 days ago arxiv.org
PTXBench measures performance of AI-generated GPU kernels
Published on September 29, 2026, PTXBench evaluates matrix multiplication and attention kernels in architecture-specific PTX on H100 and B200 GPUs, measuring correctness, speed, and whether requested instructions are actually executed.
- 18 days ago arxiv.org
Quiet-STaR: training models to reason before responding
A paper published on September 22, 2026 describes an approach for training language models to generate internal reasoning before visible text, based on the idea that text often contains implicit reasoning.
- 18 days ago arxiv.org
Tracing training data to LLM circuits
Published on July 7, 2026, the paper presents a method for linking training samples to interpretable units in language models and examines effects on attention heads and in-context learning.
RESA: recovering context with sparse attention
Published on September 29, 2026, the RESA paper proposes using residual estimation to recover context ignored by sparse attention, with no additional computational cost, according to the paper.
- 18 days ago x.com
Token retention may reduce reliance on the KV cache
A publication dated September 13, 2026, claims that keeping four initial tokens and a 64-token window without training matches or exceeds distilled linear-attention models on nearly all benchmarks.
- 18 days ago arxiv.org
RetNet proposes recurrence as an alternative to attention
Published on September 24, 2026, the Retentive Network paper presents a language-model architecture that combines parallel training with recurrent inference.
- 18 days ago arxiv.org
RWKV combines parallel training with recurrent inference
An article published on September 20, 2026 describes an architecture combining Transformer-style parallel training with RNN-style recurrent inference, with memory and computation scaling linearly with sequence length.
- 18 days ago x.com
Safety tuning may affect models’ attribution of minds
A study reports that tuning to prevent consciousness claims also reduces the attribution of minds to other entities in three models, and that mechanistic interventions restore these tendencies.
Sparse attention block selection reduced to O(N log N)
A paper published on September 28, 2026 presents hierarchical block selection with LogSumExp, reducing the cost of this step from O(N²) to O(N log N) for long-context models.
- 18 days ago arxiv.org
Softmax Linear Attention seeks better selection at linear cost
Published on September 16, 2026, the paper proposes applying softmax at the head level to restore global competition in linear attention while retaining linear complexity.
- 18 days ago x.com
SwiLA proposes switching between linear attention maps
A paper presented as work at COLM 2026 and dated October 5, 2026 describes an approach that switches among multiple linear maps and highlights the trade-off between expressiveness and memory.
- 18 days ago arxiv.org
Deep learning theory for analyzing model behavior
A book published on June 24, 2026, presents a theoretical framework linking deep neural network architecture and training to model outputs.
Transformer: from attention to implementation
Chapter 8 of Speech and Language Processing covers Transformer components from intuition to matrix formulation, providing a foundation for assessing how the architecture relates to business applications.
Let's assess your project
I reply with scope and a proposal.