Qwen3.8-27B: converting full attention to Gated DeltaNet via per-layer distillation

What happened and what to do

The post announces Qwen3.8-27B, with all 64 layers converted to Gated DeltaNet, a form of linear attention. According to the text, the 16 original full attention layers were recovered through per-layer distillation, with LoRA adaptation at rank 32. The model covers English and Japanese, and the post presents the recipe as a concrete way to convert a pretrained full attention transformer into linear attention.

In practice, a company can evaluate this technique as an alternative for workloads where attention cost weighs on the service. Before any adoption, the client's real tasks would need to be measured, comparing the original model with the converted variant, and the cost of the distillation process estimated. Wendelmaques can diagnose whether this conversion fits the client's case, set up a comparative evaluation pipeline, and implement inference on the client's own infrastructure with continuous monitoring.

How the consultancy can help

Feasibility diagnosis for the client's workload, with a comparative evaluation between the original model and a linear attention variant. Then implementation of a distillation and evaluation pipeline, and private inference on the client's infrastructure, with operation and monitoring.

Next step

Send a short description of your use case and the tasks that depend on language models. Wendelmaques replies with a scoped proposal for diagnosis and implementation.