AI and infrastructure news
Filter by type: Article33
LiFT: looped transformers for image generation with flow matching and adjustable inference cost
The paper presents LiFT, an image generator with flow matching that reuses a shared transformer core across loops. According to the author, a model trained with 2 loops improves when run with up to 16 loops at inference, without retraining.
Shared-KV attention: efficiency and compute trade-offs
A publication dated April 29, 2026 compares high-order MQA and MLA: sharing KV can ease cache pressure at the cost of more computation per token.
- 17 days ago k-a.in
Evaluate compressed sparse attention in models
An implementation published on April 28, 2026 reproduces the Compressed Sparse Attention technique described in a recent paper, giving engineering teams a reference for examining this attention variant.
Comparing three on-policy self-distillation methods
A February 12, 2026 article examines OPSD, SDFT, and SDPO, methods using denser feedback, and discusses benchmarks and limitations.
Estimating training costs for loop models
A blog article published on April 19, 2026 analyzes seven loop-model variants and how gradient structure affects FLOPs and memory use.
Attention mechanisms: comparing Transformer architectures
An article published on April 19, 2026, surveys more than 13 attention mechanisms and explains how they shape context and memory processing in Transformers and LLMs.
Sequential Attention aims for leaner models
On February 9, 2026, Google Research presented Sequential Attention, an approach that selects features or layers during training to seek faster, leaner models without sacrificing accuracy.
Techniques for more efficient LLM serving
Class notes published on May 13, 2026 cover quantization, sparsity, attention, speculative decoding, and batching to improve LLM inference and serving.
Energy-based transformers refine token predictions
A paper published on September 27, 2025 reports that 44-million-parameter Energy-Based Transformers outperformed vanilla transformers on three of four benchmarks using RedPajama-Data-v2.
Agents optimize GPU attention kernels
A September 28, 2026 NVIDIA post reports attention kernels up to 2.96 times faster than FlashKDA on B300, with one tenth the state error.
Algorithms in transformer weights: practical applications
Published on June 24, 2026, the article about The Transformer Cookbook describes encoding addition, search, and branching algorithms in transformer weights, following work on RASP.
On-Demand Attention aims to balance long context and speed
A publication dated September 20, 2026 describes On-Demand Attention, which uses a lightweight head to trigger full context when needed, aiming to speed decoding while recovering much of the performance lost with local attention.
BF16 in FlashAttention: monitor gradient stability
On September 29, 2026, a technical work reported that BF16 rounding in FlashAttention can break a softmax gradient symmetry and cause explosions near the end of training. The authors propose GProj as a correction.
How to assess LLM access to knowledge in its parameters
In a June 28, 2026 publication, the Google Research blog examines how reasoning may help language models access knowledge stored in their parameters. The question can inform internal tests of models and prompts.
Structuring AI self-improvement cycles
An article published on September 13, 2026 presents a framework for AI systems to identify limitations, validate changes, and retain improvements through persistent cycles.
Diffusion loops: reusing Transformer blocks
An article published on October 3, 2026 describes a text-to-image model that reuses Transformer blocks during denoising and reports gains from adding loops.
HySparse2 proposes two-level KV sharing
Dated September 27, 2026, the paper presents a hybrid sparse-attention method intended to reduce prefill demands and compact KV cache storage for agents with long contexts.
Recurrent-depth AI: monitoring challenges
A September 2, 2026 publication says the Astra AI uses recurrent-depth reasoning, which may make monitoring harder, affect costs and performance, and complicate oversight.
Jagged Flash Attention on NVIDIA Blackwell: lessons for kernel optimization
In an article published on October 1, 2026, the PyTorch blog describes an attention kernel developed with TLX for NVIDIA Blackwell. For the jagged shapes in Meta's Generative Ads Model, the article reports performance exceeding FlashAttention-4.
Triton paged-attention kernel for technical evaluation
A July 10, 2026 post shares a Triton paged-attention kernel attributed to an AI model and a solution file hosted on KernelBench.
LiFT expands iterative inference with a shared DiT core
Published on October 6, 2026, the research reports that LiFT outperforms a dense DiT baseline on ImageNet 256×256 with about 60% fewer parameters.
LoopCD refines Transformer decoding without additional training
The paper, published on October 2, 2026, presents a technique that compares predictions from early and late recurrent states to refine decoding. It reports higher accuracy on AIME 2024 with Ouro-2.6B-Thinking.
Looped-DiT reuses blocks to reduce computation
A post published on October 1, 2026, says Looped-DiT outperformed a model 6.5 times larger on image-generation benchmarks while using 4.9 times less inference computation.
Matryoshka Attribution identifies weights linked to model behavior
In a post published on September 28, 2026, Matryoshka Attribution is presented as a technique for ranking neural representations and weight changes by their influence on behavior.
Shared memory reduces memory use in recurrent Transformers
An article published on October 5, 2026 reports a 76–79% reduction in context memory and improved quality in models with 150 million to 1 billion parameters.
Token-indexed memory changes attention in Transformers
A post published on September 25, 2026 describes an architecture that sums contextual keys and token-indexed, layer-specific memory vectors, and allows the memory to reside in CPU memory.
Diffusion language models: techniques and applications
Published on July 8, 2026, the Kuleshov Group post presents diffusion language models and discusses masked diffusion, iterative refinement, post-training, and variable-length generation.
The role of chain of thought in serial tasks
An article dated October 1, 2026, examines whether chain of thought contributes computation in Transformers and reports reduced accuracy when it is removed from serial tasks.
OPAL lowers attention costs in long contexts
A September 29, 2026 post describes OPAL, a method that trains linear attention models on their own outputs and reports full retrieval recovery and 83–93% of mathematical reasoning performance.
PC-ALM trains residual MLPs up to 1,000 layers deep
Published on September 14, 2026, the paper presents PC-ALM, a local learning method for deep networks with performance described as nearly equivalent to backpropagation.
Sparse attention block selection reduced to O(N log N)
A paper published on September 28, 2026 presents hierarchical block selection with LogSumExp, reducing the cost of this step from O(N²) to O(N log N) for long-context models.
Recurrent training improves reasoning benchmark results
A post published on September 16, 2026, reports that local denoising objectives improved recurrent models on six benchmarks, including 58.8% accuracy on ARC-AGI-1.
TWT reduces compute for Vision Transformers
Published on September 26, 2026, the post describes Transformer-Within-Transformer, which replaces groups of similar layers. For DINOv2, it reports about half the compute with a small drop in accuracy.
Let's assess your project
I reply with scope and a proposal.