AI and infrastructure news

Post49

Attention and model architecture

Attention and model architecture Post
10/23/263 days ago x.com
Meta paper proposes byte-level models without a tokenizer
Meta's paper 'Breaking the Token Ceiling' describes models trained on raw bytes with distillation. A post credits 1B byte models with performance equal to or above Llama 3.2-1B and Gemma-3-1B, claims still to be confirmed.
Attention and model architecture Post
10/08/2618 days ago x.com
CALM proposes predicting vector blocks, not tokens
A text published on January 24, 2026 describes CALM models that predict blocks of meaning, claiming 4× more information per step and 44% less training compute.
Attention and model architecture Post
10/08/2618 days ago x.com
How attention and MLPs store knowledge
Published on April 22, 2026, the paper examines trade-offs in knowledge storage by attention matrices and MLPs, identifying limits linked to finite samples and compute, as well as noise in long contexts.
Attention and model architecture Post
10/08/2618 days ago x.com
FlashAttention-4 targets optimization on Blackwell GPUs
A publication dated March 11, 2026 describes new pipelines, reformulated calculations, and memory optimizations for attention workloads on Blackwell GPUs.
Attention and model architecture Post
10/08/2618 days ago x.com
Activation functions as a target for automated research
A February 7, 2026 publication reports using AlphaEvolve to explore activation functions as part of broader research into new model architectures.
Attention and model architecture Post
10/08/2618 days ago x.com
Kascade speeds inference with sparse attention
Published on December 24, 2025, the paper describes a training-free method that reuses key patterns across layers. It reports up to 4.1× faster generation and 2.2× faster prefill on H100 GPUs.
Attention and model architecture Post
10/08/2618 days ago x.com
Head parallelism may speed up MoE systems
A post published on February 9, 2026 describes an architecture that distributes heads across GPUs and reports up to 1.61× speedup over standard MoE with expert parallelism.
Attention and model architecture Post
10/08/2618 days ago x.com
ROME: segment-level credit for training agents
A February 10, 2026 publication describes ROME, a 3-billion-parameter model trained with segment-level rewards and structured code tasks, SFT, and RL.
Attention and model architecture Post
10/08/2618 days ago x.com
Loop transformer reaches 100% on Sudoku-Extreme
A publication dated June 1, 2026 describes a compact, 800,000-parameter loop transformer that scored 100% on Sudoku-Extreme after 15 minutes of training.
Attention and model architecture Post
10/08/2618 days ago x.com
URM combines recurrence and nonlinearity for reasoning
A post on Universal Transformer research attributes reasoning gains to recurrent inductive bias and strong nonlinearity, and describes design choices for local token mixing and training.
Attention and model architecture Post
10/08/2618 days ago x.com
AHN proposes continuous memory for long-context LLMs
On October 9, 2025, a ByteDance publication introduced Artificial Hippocampus Networks, an architecture that continuously compresses information outside a language model's context window.
Attention and model architecture Post
10/08/2618 days ago x.com
Attention and kernel regression: a useful connection
A 10-slide summary published on October 4, 2025 examines the relationship between kernel regression and attention, presenting this perspective as an aid to understanding the mechanism.
Attention and model architecture Post
10/08/2618 days ago x.com
CCA: attention in a compressed latent space
A post dated October 7, 2025 describes an architecture that computes attention in a smaller latent space and reports lower memory, parameter, and operation counts in tests on H100.
Attention and model architecture Post
10/08/2618 days ago x.com
Model circuits can help predict reasoning errors
A post published on October 13, 2025 describes a method using attribution graphs to identify incorrect reasoning on arithmetic tasks, with improved detection metrics.
Attention and model architecture Post
10/08/2618 days ago x.com
Balancing adaptation and memory in model training
A study published on October 5, 2025, reports that a model trained on 12,000 small tasks combined new ideas and notes distinct effects from structured and mixed lessons.
Attention and model architecture Post
10/08/2618 days ago x.com
How to observe information flow in LLMs
Published on October 9, 2025, the publication proposes three stages for information flow in LLMs and links massive activations to attention sinks and compression valleys.
Attention and model architecture Post
10/08/2618 days ago x.com
Short reasoning hints for evaluating AI systems
A paper published on October 5, 2025 describes a hint generator and a solver trained with reinforcement learning. For mathematics, it reports 44% higher accuracy on AIME 2025 than RL with long chains.
Attention and model architecture Post
10/08/2618 days ago x.com
Dragon Hatchling explores local memory in language models
Published on October 3, 2025, the post describes an architecture with simple neurons and connections that store short-term memory through Hebbian learning.
Attention and model architecture Post
10/08/2618 days ago x.com
Dragon Hatchling proposes an interpretable LLM architecture
Published on October 1, 2025, the post describes an LLM architecture based on locally interacting neural particles and says it rivals GPT-2 performance.
Attention and model architecture Post
10/08/2618 days ago x.com
Study compares attention in 13 LLMs, with 17% performing best
Published on October 12, 2025, the study reports experiments training 13 LLMs with different proportions of conventional attention and DeltaNet linear attention. The configuration with 17% attention—two of 12 layers—achieved the best result.
Attention and model architecture Post
10/08/2618 days ago x.com
Grokking in modular addition: three stages to generalization
A post dated October 4, 2025 describes a paper proposing three stages of grokking in modular addition and a data threshold near n log n, with weight decay and moderate width supporting feature learning.
Attention and model architecture Post
10/08/2618 days ago x.com
What attention sinks reveal about LLMs
Experiments on models from 410 million to 120 billion parameters link extreme activation norms of the initial token to compression valleys and attention sinks.
Attention and model architecture Post
10/08/2618 days ago x.com
RLAD explores reasoning abstractions with reinforcement learning
Published on October 3, 2025, the paper presents RLAD, a framework using two-player reinforcement learning to help language models discover procedural hints in natural language.
Attention and model architecture Post
10/08/2618 days ago x.com
RLMs: an approach to very long prompts
Published on October 15, 2025, the publication describes language models that interact recursively with long prompts through a REPL and reports evaluations on the OOLONG and BrowseComp-Plus benchmarks.
Attention and model architecture Post
10/08/2618 days ago x.com
SwiReasoning switches reasoning modes to reduce tokens
A publication dated October 12, 2025 describes a method that switches between latent and explicit reasoning and reports token-efficiency gains on Qwen3-8B models.
Attention and model architecture Post
10/08/2618 days ago x.com
A unified view of Attention-MoE and FFN-MoE
A paper published on September 24, 2025 proposes treating pre-mixed tokens and (V·Wo) as an FFN expert, unifying sparse attention and FFN approaches.
Attention and model architecture Post
10/07/2619 days ago x.com
Hybrid architectures beyond standard attention
A post published on July 8, 2026, compares the architectures of Qwen 3.5 and Nemotron 3 and describes linear attention and state-space models as examples of convergent evolution.
Attention and model architecture Post
10/07/2619 days ago x.com
Sparse attention can optimize KV block selection
A Tencent publication dated September 14, 2026 describes a method that uses language modeling loss to learn which KV blocks each query should access.
Attention and model architecture Post
10/07/2619 days ago x.com
Dynamic linear attention adapts memory to token information
A post published on June 15, 2026, describes Dynamic Linear Attention adapting memory states to token information while keeping a fixed cache, and reports gains in benchmarks, throughput, and memory use.
Attention and model architecture Post
10/07/2619 days ago x.com
Linear and sparse attention: evaluating long-context models
A post dated August 26, 2026, claims GLM-5.3 uses three linear-attention layers for each sparse-attention layer, highlighting compute-efficiency trade-offs for long contexts.
Attention and model architecture Post
10/07/2619 days ago x.com
A 64-token cache for linear attention: what to evaluate
A post about HOLA, published on July 5, 2026, describes a parameter-free token-surprise metric and a 64-token cache to address catastrophic forgetting in linear-attention models.
Attention and model architecture Post
10/07/2619 days ago x.com
Diffusing Blame proposes training networks without backpropagation
A post published on July 17, 2026 describes an error-signal routing method and reports results in image recognition, locomotion, and Craftax.
Attention and model architecture Post
10/07/2619 days ago x.com
Dust explores gradient-free transformer training
Published on October 5, 2026, the paper presents Dust, a zeroth-order method using activation perturbations, and reports scaling experiments and comparisons with backpropagation.
Attention and model architecture Post
10/07/2619 days ago x.com
Attention factorization and gradient flow: implications for models
Published on September 30, 2026, the publication discusses how factoring the attention matrix as S = WWᵀ may change gradient flow and remove an information-exponent bottleneck in optimization.
Attention and model architecture Post
10/07/2619 days ago x.com
Recurrent flows refine reasoning on discrete tasks
A post published on September 3, 2026 describes models combining continuous flows and recurrent refinement to solve structured problems such as Sudoku.
Attention and model architecture Post
10/07/2619 days ago x.com
FuncAttn brings functional attention to continuous data
A June 25, 2026 post describes a method using structured linear operators for attention between function spaces, with results in PDE solving, 3D segmentation, and regression.
Attention and model architecture Post
10/07/2619 days ago x.com
Functional Attention proposes linear-cost attention
Published on July 6, 2026, the paper describes representing query and key/value fields with learned bases and a compact matrix, reporting linear cost in the number of points.
Attention and model architecture Post
10/07/2619 days ago x.com
GLM-5.3-Flash: an architecture overview
Published on August 26, 2026, the overview describes a hybrid architecture with 3:1 attention, a smaller sparse MoE backbone, a four-stream mHC residual path, and a native vision encoder.
Attention and model architecture Post
10/07/2619 days ago x.com
HydraHead combines two attention types per head
Published on June 30, 2026, the post about HydraHead describes an architecture that combines Full Attention and Linear Attention at the head level to pursue greater efficiency in long-context models.
Attention and model architecture Post
10/07/2619 days ago x.com
iLLaDA trains an 8-billion-parameter diffusion model
A paper published on June 27, 2026 describes training an 8-billion-parameter Transformer with bidirectional masked diffusion on 12 trillion tokens.

Let's assess your project

I reply with scope and a proposal.