AI and infrastructure news
Filter by type: Paper77
- 18 days ago
arxiv.orgSparse attention in transformers after post-training
A paper published on December 8, 2025 reports that constrained-loss sparsity regularization retained the pretraining loss of models up to 7 billion parameters while using about 0.4% of attention connections.
- 18 days ago arxiv.org
Exclusive attention: an alternative for language models
An article published on March 12, 2026 presents an attention variant that reportedly outperforms standard attention in models of up to 2.7 billion parameters.
- 18 days ago arxiv.org
BEAVER proposes probability bounds for LLM safety
Published on December 10, 2025, the paper presents a framework for calculating deterministic probability bounds on whether an LLM satisfies a safety property, contrasting them with sampling-based estimates.
- 18 days ago arxiv.org
CliffordNet tests Clifford algebra in neural networks
A post published on April 20, 2026 describes an architecture using the geometric product, without attention or mixer layers, and reports 77.82% accuracy on CIFAR-100 with 1.4 million parameters.
- 18 days ago arxiv.org
How to validate questions about knowledge graphs
A post published on December 10, 2025 describes SEAL, a framework that extracts the minimum core of a question and uses an agent to validate it against a database.
- 18 days ago arxiv.org
KV cache compression could cut memory use by up to 50×
A publication dated June 12, 2026 describes an Attention Matching method for KV cache compression, reporting reductions of up to 50× on some datasets with minimal quality loss.
- 18 days ago arxiv.org
DeepCrossAttention combines Transformer layers based on input
A description published on March 16, 2026 presents DeepCrossAttention, which uses learned, input-dependent weights to combine Transformer layer outputs.
- 18 days ago arxiv.org
GATs weigh the importance of graph neighbors
The 2018 ICLR paper presents networks that assign weights to connected neighbors. The February 12, 2026 publication highlights the approach as an option to compare for graph-data tasks.
- 18 days ago arxiv.org
HybridGen proposes hybrid attention for long-context inference
A paper published on April 21, 2026 presents HybridGen, a framework coordinating attention between CPU and GPU with CXL memory for long-context language-model inference.
- 18 days ago arxiv.org
Instability in low-precision Flash Attention
Published on March 15, 2026, the paper examines catastrophic loss spikes in low-precision training with Flash Attention, linking them to similar internal attention data patterns and accumulating biased rounding errors.
- 18 days ago arxiv.org
Interleaved Head Attention explores communication between heads
A post published on April 11, 2026, explores communication between attention heads within a layer, focusing on reasoning with Interleaved Head Attention.
- 18 days ago arxiv.org
KArAt proposes learnable attention without fixed softmax
Published on October 27, 2025, the paper presents KArAt, a learnable alternative to softmax attention. It reports mixed results in vision transformers and low-rank variants intended to reduce memory use.
- 18 days ago arxiv.org
Kascade reduces attention computation in long contexts
Published on February 18, 2026, the article describes a technique that reuses attention results across Transformer layers to reduce inference work without retraining the model or changing its weights.
- 18 days ago arxiv.org
Mamba-3 proposes fixed-state sequence modeling
Published on May 7, 2026, the post describes a non-Transformer language architecture that compresses context into a fixed-size state and claims decoding time constant with respect to sequence length.
- 18 days ago arxiv.org
MoDA connects attention across LLM layers
Published on April 19, 2026, the paper presents MoDA, a mechanism combining key-value pairs from the current layer with information from earlier layers.
- 18 days ago arxiv.org
Masked diffusion models and token order
An ICML 2025 paper examines why masked diffusion models underperform on generation benchmarks and identifies challenges in training with subsets of tokens.
- 18 days ago arxiv.org
Tensor Product Attention proposes reducing KV cache memory
Published on October 27, 2025, the article presents an attention technique using contextual low-rank components to reduce KV cache memory use during inference.
- 18 days ago arxiv.org
Compressed attention to reduce costs in long-context models
Published on October 7, 2025, the paper presents an approach that compresses queries, keys, and values to perform attention in a smaller latent space, targeting compute and KV-cache costs.
- 18 days ago arxiv.org
Continuous Thought Machines explore neuron-level dynamics
Published on October 6, 2025, the paper presents a model using neuron-level processing and synchronization as central representations, inspired by biological neural dynamics.
- 18 days ago arxiv.org
Gradient descent viewed as attention over examples
Published on February 6, 2023, the article interprets gradient-trained linear layers as memories of training examples and describes predictions as unnormalized attention over training experience.
- 18 days ago arxiv.org
MoE-CL proposes continuous LLM adaptation
Published on October 6, 2025, the paper presents a continuous tuning method using LoRA experts and reports evaluations on the MTL5 and Tencent3 benchmarks, plus an A/B test.
- 19 days ago arxiv.org
Attention in long-context Transformers
Published on September 15, 2026, the paper analyzes attention rank collapse as context grows and studies rescaling scores by a polylogarithmic factor.
- 19 days ago arxiv.org
Natively trainable sparse attention for long contexts
An article published on September 26, 2026, presents NSA, a sparse attention approach combining algorithmic innovations and hardware-aligned optimizations for more efficient long-context modeling.
- 19 days ago arxiv.org
Attention sinks and rescaling in transformers
Published on October 5, 2026, the paper examines attention sinks and residual sinks in large language models and proposes how these outliers, softmax attention, and RMSNorm rescale other components.
- 19 days ago arxiv.org
Sparse layers and looping for scaling language models
An article published on September 26, 2026 compares standard Transformers and Mixture-of-Experts, with and without looping, and reports a scaling advantage for Looped-MoE models.
- 19 days ago arxiv.org
Evaluating efficient attention in hybrid models
Published on June 16, 2026, the paper examines sliding-window attention and recurrent sequence mixers in hybrid language models, focusing on scalability, mechanisms, and architecture decisions.
- 19 days ago arxiv.org
How to assess autoregressive inference bottlenecks
An ICLR 2026 paper addresses a decoding bottleneck in probabilistic Transformer models. Companies can measure inference performance and identify opportunities for optimization.
- 19 days ago arxiv.org
How LLMs represent sums and arithmetic errors
An article published on June 28, 2026, analyzes internal representations during sums with multiple operands and proposes a geometric model to explain arithmetic errors.
- 19 days ago arxiv.org
Reducing Cost and Latency in LLM Applications
A July 10, 2026 article explores alternatives to long reasoning chains to balance accuracy, context size, compute cost, and latency.
- 19 days ago arxiv.org
How to test n-gram lookups in language models
A technical publication dated August 25, 2026 describes Engram, a module that retrieves n-gram embeddings and can move its lookup table to the CPU with little inference slowdown.
- 19 days ago arxiv.org
Sharing QKV projections in Transformers
A paper published on July 2, 2026 evaluates attention variants that share projections and reports that equating key and value was excessive for the author's small ternary LLMs.
- 19 days ago arxiv.org
CTM-AI explores an AI architecture inspired by consciousness
Published on August 25, 2026, the article presents the early CTM-AI project, which applies the Conscious Turing Machine to AI architecture without reporting benchmark results.
- 19 days ago arxiv.org
Depth-wise batching in recurrent models: how to assess it
A post published on September 2, 2026 describes how overlapping requests at different recurrent depths may improve utilization in models with adaptive compute.
- 19 days ago arxiv.org
DepthBench proposes evaluating computational depth in Transformers
Presented on September 29, 2026, DepthBench compares information-flow approaches and examines whether they help deeper Transformers use their layers effectively.
- 19 days ago arxiv.org
Distilling Transformers into Mamba models
A paper published September 29, 2026 studies transferring knowledge from pretrained Transformers to state space models such as Mamba.
- 19 days ago arxiv.org
Context filtering for attention on constrained hardware
A SAS article published on September 16, 2026 describes an end-to-end trained selector inside Softmax to filter context in attention mechanisms aimed at hardware with limited computing resources.
- 19 days ago arxiv.org
FlashAttention-3: optimizing attention on Hopper GPUs
Published on September 24, 2026, the paper presents asynchronous execution and low-precision techniques to accelerate attention on Hopper GPUs, focusing on memory traffic and hardware utilization.
- 19 days ago arxiv.org
FlowTracer tracks influential tokens in LLM reasoning
Published on July 4, 2026, the article presents a method that uses attention to trace information propagation and direct credit assignment in reinforcement learning.
- 19 days ago arxiv.org
Gated Linear Attention: linear attention with gates
Published on September 20, 2026, the paper presents a transformer formulation with linear attention and data-dependent gates, also expressible as an RNN with matrix-valued hidden states.
- 19 days ago arxiv.org
GOAT proposes trainable priors for attention
Published on October 3, 2026, the paper frames standard attention as an entropic optimal transport problem with an implicit uniform prior and presents GOAT, which uses a continuous, trainable prior.
Let's assess your project
I reply with scope and a proposal.