AI and infrastructure news
Filter by type: All169
- 2 days ago x.com
Meta paper proposes byte-level models without a tokenizer
Meta's paper 'Breaking the Token Ceiling' describes models trained on raw bytes with distillation. A post credits 1B byte models with performance equal to or above Llama 3.2-1B and Gemma-3-1B, claims still to be confirmed.
μLattice: an alternative to attention for memory scaling, according to its author
An open-source project and paper describe μLattice, a memory design that grows with input without quadratic cost. The performance claims come from the author and have not been independently verified.
LiFT: looped transformers for image generation with flow matching and adjustable inference cost
The paper presents LiFT, an image generator with flow matching that reuses a shared transformer core across loops. According to the author, a model trained with 2 loops improves when run with up to 16 loops at inference, without retraining.
Shared-KV attention: efficiency and compute trade-offs
A publication dated April 29, 2026 compares high-order MQA and MLA: sharing KV can ease cache pressure at the cost of more computation per token.
- 17 days ago
arxiv.orgSparse attention in transformers after post-training
A paper published on December 8, 2025 reports that constrained-loss sparsity regularization retained the pretraining loss of models up to 7 billion parameters while using about 0.4% of attention connections.
- 17 days ago arxiv.org
Exclusive attention: an alternative for language models
An article published on March 12, 2026 presents an attention variant that reportedly outperforms standard attention in models of up to 2.7 billion parameters.
- 17 days ago k-a.in
Evaluate compressed sparse attention in models
An implementation published on April 28, 2026 reproduces the Compressed Sparse Attention technique described in a recent paper, giving engineering teams a reference for examining this attention variant.
- 17 days ago arxiv.org
BEAVER proposes probability bounds for LLM safety
Published on December 10, 2025, the paper presents a framework for calculating deterministic probability bounds on whether an LLM satisfies a safety property, contrasting them with sampling-based estimates.
- 17 days ago x.com
CALM proposes predicting vector blocks, not tokens
A text published on January 24, 2026 describes CALM models that predict blocks of meaning, claiming 4× more information per step and 44% less training compute.
- 17 days ago arxiv.org
CliffordNet tests Clifford algebra in neural networks
A post published on April 20, 2026 describes an architecture using the geometric product, without attention or mixer layers, and reports 77.82% accuracy on CIFAR-100 with 1.4 million parameters.
- 17 days ago x.com
How attention and MLPs store knowledge
Published on April 22, 2026, the paper examines trade-offs in knowledge storage by attention matrices and MLPs, identifying limits linked to finite samples and compute, as well as noise in long contexts.
Comparing three on-policy self-distillation methods
A February 12, 2026 article examines OPSD, SDFT, and SDPO, methods using denser feedback, and discusses benchmarks and limitations.
Estimating training costs for loop models
A blog article published on April 19, 2026 analyzes seven loop-model variants and how gradient structure affects FLOPs and memory use.
- 17 days ago arxiv.org
How to validate questions about knowledge graphs
A post published on December 10, 2025 describes SEAL, a framework that extracts the minimum core of a question and uses an agent to validate it against a database.
- 17 days ago arxiv.org
KV cache compression could cut memory use by up to 50×
A publication dated June 12, 2026 describes an Attention Matching method for KV cache compression, reporting reductions of up to 50× on some datasets with minimal quality loss.
- 17 days ago arxiv.org
DeepCrossAttention combines Transformer layers based on input
A description published on March 16, 2026 presents DeepCrossAttention, which uses learned, input-dependent weights to combine Transformer layer outputs.
- 17 days ago x.com
FlashAttention-4 targets optimization on Blackwell GPUs
A publication dated March 11, 2026 describes new pipelines, reformulated calculations, and memory optimizations for attention workloads on Blackwell GPUs.
- 17 days ago x.com
Activation functions as a target for automated research
A February 7, 2026 publication reports using AlphaEvolve to explore activation functions as part of broader research into new model architectures.
- 17 days ago arxiv.org
GATs weigh the importance of graph neighbors
The 2018 ICLR paper presents networks that assign weights to connected neighbors. The February 12, 2026 publication highlights the approach as an option to compare for graph-data tasks.
- 17 days ago arxiv.org
HybridGen proposes hybrid attention for long-context inference
A paper published on April 21, 2026 presents HybridGen, a framework coordinating attention between CPU and GPU with CXL memory for long-context language-model inference.
- 17 days ago arxiv.org
Instability in low-precision Flash Attention
Published on March 15, 2026, the paper examines catastrophic loss spikes in low-precision training with Flash Attention, linking them to similar internal attention data patterns and accumulating biased rounding errors.
- 17 days ago arxiv.org
Interleaved Head Attention explores communication between heads
A post published on April 11, 2026, explores communication between attention heads within a layer, focusing on reasoning with Interleaved Head Attention.
- 17 days ago arxiv.org
KArAt proposes learnable attention without fixed softmax
Published on October 27, 2025, the paper presents KArAt, a learnable alternative to softmax attention. It reports mixed results in vision transformers and low-rank variants intended to reduce memory use.
- 17 days ago x.com
Kascade speeds inference with sparse attention
Published on December 24, 2025, the paper describes a training-free method that reuses key patterns across layers. It reports up to 4.1× faster generation and 2.2× faster prefill on H100 GPUs.
- 17 days ago arxiv.org
Kascade reduces attention computation in long contexts
Published on February 18, 2026, the article describes a technique that reuses attention results across Transformer layers to reduce inference work without retraining the model or changing its weights.
Triton kernels and PyTorch operators for AttnRes
Published on May 3, 2026, the repository provides Triton kernels and PyTorch operators for Block Attention Residuals (AttnRes).
- 17 days ago arxiv.org
Mamba-3 proposes fixed-state sequence modeling
Published on May 7, 2026, the post describes a non-Transformer language architecture that compresses context into a fixed-size state and claims decoding time constant with respect to sequence length.
Attention mechanisms: comparing Transformer architectures
An article published on April 19, 2026, surveys more than 13 attention mechanisms and explains how they shape context and memory processing in Transformers and LLMs.
microGPT in C explores low-level optimizations
Published on May 3, 2026, the repository presents a dependency-free implementation of microGPT in plain C, using AVX2 intrinsics, fixed-width dot-product kernels, and a fast approximation of the exponential function.
- 17 days ago arxiv.org
MoDA connects attention across LLM layers
Published on April 19, 2026, the paper presents MoDA, a mechanism combining key-value pairs from the current layer with information from earlier layers.
- 17 days ago arxiv.org
Masked diffusion models and token order
An ICML 2025 paper examines why masked diffusion models underperform on generation benchmarks and identifies challenges in training with subsets of tokens.
Nanochat outlines architecture and optimization choices to investigate
Published on February 1, 2026, the post about nanochat lists model architecture components and describes splitting optimization between Muon and AdamW.
- 17 days ago x.com
Head parallelism may speed up MoE systems
A post published on February 9, 2026 describes an architecture that distributes heads across GPUs and reports up to 1.61× speedup over standard MoE with expert parallelism.
Compact reasoning could lower inference costs
A post published on April 22, 2026 describes Latent Reasoning Tuning, a method that trains models to reason with compact internal representations and claims gains over other efficient reasoning methods on mathematics and general benchmarks.
- 17 days ago x.com
ROME: segment-level credit for training agents
A February 10, 2026 publication describes ROME, a 3-billion-parameter model trained with segment-level rewards and structured code tasks, SFT, and RL.
Scaled dot-product attention: from mechanism to practice
A post published on February 1, 2026 explains how queries, keys, and values guide token relevance and presents a notebook for implementing LLM and NLP systems from first principles.
Sequential Attention aims for leaner models
On February 9, 2026, Google Research presented Sequential Attention, an approach that selects features or layers during training to seek faster, leaner models without sacrificing accuracy.
Techniques for more efficient LLM serving
Class notes published on May 13, 2026 cover quantization, sparsity, attention, speculative decoding, and batching to improve LLM inference and serving.
- 17 days ago arxiv.org
Tensor Product Attention proposes reducing KV cache memory
Published on October 27, 2025, the article presents an attention technique using contextual low-rank components to reduce KV cache memory use during inference.
- 17 days ago x.com
Loop transformer reaches 100% on Sudoku-Extreme
A publication dated June 1, 2026 describes a compact, 800,000-parameter loop transformer that scored 100% on Sudoku-Extreme after 15 minutes of training.
Let's assess your project
I reply with scope and a proposal.