AI and infrastructure news
Filter by type: All169
Structuring AI self-improvement cycles
An article published on September 13, 2026 presents a framework for AI systems to identify limitations, validate changes, and retain improvements through persistent cycles.
- 18 days ago
arxiv.orgHow LLMs represent sums and arithmetic errors
An article published on June 28, 2026, analyzes internal representations during sums with multiple operands and proposes a geometric model to explain arithmetic errors.
- 18 days ago arxiv.org
Reducing Cost and Latency in LLM Applications
A July 10, 2026 article explores alternatives to long reasoning chains to balance accuracy, context size, compute cost, and latency.
- 18 days ago arxiv.org
How to test n-gram lookups in language models
A technical publication dated August 25, 2026 describes Engram, a module that retrieves n-gram embeddings and can move its lookup table to the CPU with little inference slowdown.
- 18 days ago arxiv.org
Sharing QKV projections in Transformers
A paper published on July 2, 2026 evaluates attention variants that share projections and reports that equating key and value was excessive for the author's small ternary LLMs.
- 18 days ago arxiv.org
CTM-AI explores an AI architecture inspired by consciousness
Published on August 25, 2026, the article presents the early CTM-AI project, which applies the Conscious Turing Machine to AI architecture without reporting benchmark results.
- 18 days ago arxiv.org
Depth-wise batching in recurrent models: how to assess it
A post published on September 2, 2026 describes how overlapping requests at different recurrent depths may improve utilization in models with adaptive compute.
- 18 days ago arxiv.org
DepthBench proposes evaluating computational depth in Transformers
Presented on September 29, 2026, DepthBench compares information-flow approaches and examines whether they help deeper Transformers use their layers effectively.
- 18 days ago arxiv.org
Distilling Transformers into Mamba models
A paper published September 29, 2026 studies transferring knowledge from pretrained Transformers to state space models such as Mamba.
- 18 days ago x.com
Diffusing Blame proposes training networks without backpropagation
A post published on July 17, 2026 describes an error-signal routing method and reports results in image recognition, locomotion, and Craftax.
Diffusion loops: reusing Transformer blocks
An article published on October 3, 2026 describes a text-to-image model that reuses Transformer blocks during denoising and reports gains from adding loops.
- 18 days ago x.com
Dust explores gradient-free transformer training
Published on October 5, 2026, the paper presents Dust, a zeroth-order method using activation perturbations, and reports scaling experiments and comparisons with backpropagation.
Recurrent state could extend Transformer memory
The description of the Recurrent Looped Transformer, dated September 13, 2026, presents a recurrent decoder that preserves state between tokens to extend the temporal information path.
- 18 days ago x.com
Attention factorization and gradient flow: implications for models
Published on September 30, 2026, the publication discusses how factoring the attention matrix as S = WWᵀ may change gradient flow and remove an information-exponent bottleneck in optimization.
- 18 days ago arxiv.org
Context filtering for attention on constrained hardware
A SAS article published on September 16, 2026 describes an end-to-end trained selector inside Softmax to filter context in attention mechanisms aimed at hardware with limited computing resources.
- 18 days ago arxiv.org
FlashAttention-3: optimizing attention on Hopper GPUs
Published on September 24, 2026, the paper presents asynchronous execution and low-precision techniques to accelerate attention on Hopper GPUs, focusing on memory traffic and hardware utilization.
- 18 days ago arxiv.org
FlowTracer tracks influential tokens in LLM reasoning
Published on July 4, 2026, the article presents a method that uses attention to trace information propagation and direct credit assignment in reinforcement learning.
- 18 days ago x.com
Recurrent flows refine reasoning on discrete tasks
A post published on September 3, 2026 describes models combining continuous flows and recurrent refinement to solve structured problems such as Sudoku.
- 18 days ago x.com
FuncAttn brings functional attention to continuous data
A June 25, 2026 post describes a method using structured linear operators for attention between function spaces, with results in PDE solving, 3D segmentation, and regression.
- 18 days ago x.com
Functional Attention proposes linear-cost attention
Published on July 6, 2026, the paper describes representing query and key/value fields with learned bases and a compact matrix, reporting linear cost in the number of points.
- 18 days ago arxiv.org
Gated Linear Attention: linear attention with gates
Published on September 20, 2026, the paper presents a transformer formulation with linear attention and data-dependent gates, also expressible as an RNN with matrix-valued hidden states.
- 18 days ago x.com
GLM-5.3-Flash: an architecture overview
Published on August 26, 2026, the overview describes a hybrid architecture with 3:1 attention, a smaller sparse MoE backbone, a four-stream mHC residual path, and a native vision encoder.
- 18 days ago arxiv.org
GOAT proposes trainable priors for attention
Published on October 3, 2026, the paper frames standard attention as an entropic optimal transport problem with an implicit uniform prior and presents GOAT, which uses a continuous, trainable prior.
- 18 days ago arxiv.org
BF16 gradients rise late in training
A paper reports a 1,000-fold increase in gradient norm after 25 billion tokens in a transformer trained with FlashAttention-3 in BF16.
- 18 days ago arxiv.org
HiLS learns chunk selection for sparse attention
Published on July 7, 2026, the paper presents HiLS, an approach that learns chunk selection for sparse attention using the language-modeling loss.
- 18 days ago arxiv.org
HOLA pairs linear attention with an exact, bounded KV cache
Published on July 3, 2026, the paper presents HOLA, an architecture combining recurrent state and a bounded key-value cache, with reported gains in perplexity and long-range recall.
- 18 days ago arxiv.org
HOLA combines recurrent state and a limited cache for long contexts
A post dated July 6, 2026 describes HOLA, which uses recurrent state and a small KV cache for exact memory. The method reduces perplexity by 16.1% and maintains robust retrieval at 32k tokens.
- 18 days ago x.com
HydraHead combines two attention types per head
Published on June 30, 2026, the post about HydraHead describes an architecture that combines Full Attention and Linear Attention at the head level to pursue greater efficiency in long-context models.
HySparse2 proposes two-level KV sharing
Dated September 27, 2026, the paper presents a hybrid sparse-attention method intended to reduce prefill demands and compact KV cache storage for agents with long contexts.
- 18 days ago arxiv.org
HySparse2 cuts long-context costs through KV reuse
Published on September 23, 2026, the post describes two levels of KV sharing to reduce prefill FLOPs and cache size, with better retrieval scores than MiMo-V2.6's Hybrid SWA at a context length of 1 million tokens.
Recurrent-depth AI: monitoring challenges
A September 2, 2026 publication says the Astra AI uses recurrent-depth reasoning, which may make monitoring harder, affect costs and performance, and complicate oversight.
- 18 days ago arxiv.org
iLLaDA: masked diffusion in an 8B model
A description published on June 25, 2026 presents iLLaDA, an 8B masked-diffusion language model trained from scratch with fully bidirectional attention.
- 18 days ago x.com
iLLaDA trains an 8-billion-parameter diffusion model
A paper published on June 27, 2026 describes training an 8-billion-parameter Transformer with bidirectional masked diffusion on 12 trillion tokens.
- 18 days ago arxiv.org
Interpretability beyond human concepts
Published on September 22, 2026, the paper investigates whether language models use internal representations that do not map adequately to familiar human concepts.
- 18 days ago x.com
Automated interpretability for attention heads
A paper published on June 29, 2026 describes a technique that explains attention heads with Python programs. Replacing about 40% of attention patterns in Llama-3B with program outputs had little effect on performance across tasks.
Jagged Flash Attention on NVIDIA Blackwell: lessons for kernel optimization
In an article published on October 1, 2026, the PyTorch blog describes an attention kernel developed with TLX for NVIDIA Blackwell. For the jagged shapes in Meta's Generative Ads Model, the article reports performance exceeding FlashAttention-4.
- 18 days ago arxiv.org
Jamba combines Transformer, Mamba and MoE
The paper, published on September 20, 2026, describes a hybrid architecture that interleaves Transformer and Mamba blocks and uses mixture-of-experts in some layers.
- 18 days ago arxiv.org
KATA explores spherical packing for recall in linear attention
A paper published on September 16, 2026 presents KATA, a geometric approach that formulates associative recall in linear attention as a spherical packing problem.
Triton paged-attention kernel for technical evaluation
A July 10, 2026 post shares a Triton paged-attention kernel attributed to an AI model and a solution file hosted on KernelBench.
- 18 days ago arxiv.org
KITE expands Transformers without increasing KV computation
Published on September 29, 2026, the material describes KITE, an approach that expands decoder components by reusing a KV cache generated by a smaller prefiller.
Let's assess your project
I reply with scope and a proposal.