AI and infrastructure news
Filter by type: Post49
- 19 days ago x.com
Automated interpretability for attention heads
A paper published on June 29, 2026 describes a technique that explains attention heads with Python programs. Replacing about 40% of attention patterns in Llama-3B with program outputs had little effect on performance across tasks.
- 19 days ago x.com
Memory Attention: reusing keys in V=K+M
A publication dated September 27, 2026 reports preliminary ablation results suggesting V=K+M is an effective architectural choice; validation at larger scales remains necessary.
- 19 days ago x.com
Flow models refine reasoning in stages
A paper dated September 17, 2026 turns a flow model into a recurrent model: predictions are fed back for iterative refinement, with a local loss at each step and no backpropagation through time.
- 19 days ago x.com
NextLat explores predicting the next latent state
A June 16, 2026 publication describes a transformer training method and says it may enable compact world models and up to 3.3× faster inference.
- 19 days ago x.com
Post points to efficient designs in frontier Chinese models
Published on August 26, 2026, the post says frontier Chinese models are adopting linear or sparse attention, specialized residual designs, and Muon. It also presents open-source collaboration as a strategy for efficiency.
- 19 days ago x.com
Synthesized programs could replace some attention heads
A post published July 1, 2026 describes Python programs synthesized from attention maps, reporting 69–79% mean IoU and little QA decline when replacing up to 30–40% of heads.
- 19 days ago x.com
Token retention may reduce reliance on the KV cache
A publication dated September 13, 2026, claims that keeping four initial tokens and a 64-token window without training matches or exceeds distilled linear-attention models on nearly all benchmarks.
- 19 days ago x.com
Safety tuning may affect models’ attribution of minds
A study reports that tuning to prevent consciousness claims also reduces the attribution of minds to other entities in three models, and that mechanistic interventions restore these tendencies.
- 19 days ago x.com
SwiLA proposes switching between linear attention maps
A paper presented as work at COLM 2026 and dated October 5, 2026 describes an approach that switches among multiple linear maps and highlights the trade-off between expressiveness and memory.
Let's assess your project
I reply with scope and a proposal.