AI and infrastructure news

Post29

LLM inference

LLM inference Post
10/20/265 days ago x.com
Diffusion text models generate the full output at once: what changes for inference
Post describes diffusion text models that produce the whole response simultaneously, instead of token by token like autoregressive LLMs. According to the author, generation is fast, with a latency and cost profile distinct from standard inference.
LLM inference Post
10/20/265 days ago x.com
Intel GPU D270 (Crescent Island): inference accelerator with high-capacity LPDDR5X memory
A post describes Intel's D270 GPU (Crescent Island), a PCIe 5.0 card for inference with 32 Xe cores, XMX matrix engines and about 160 GB of LPDDR5X, expandable to 480 GB.
LLM inference Post
10/19/266 days ago x.com
Pruning 6 layers of Qwen3-8B-Base with LaCo cuts the model by 16.7%
The post reports removing up to 10 layers from Qwen3-8B-Base using LaCo. With 6 layers removed, the model is 16.7% smaller and, according to the author, keeps 78-94% of its reasoning scores, with a perplexity ratio of 1.68x.
LLM inference Post
10/09/2616 days ago x.com
Samsung opens LittleBit, a method that compresses 13B LLMs to under 1 GB
According to the post, Samsung's LittleBit uses latent factorization and sub-1-bit levels to shrink the weights of 13B LLMs to under 1 GB. Worth evaluating for local deployment.
LLM inference Post
10/08/2617 days ago x.com
Benchmark reports 2.89× faster TTFT at 64K context
Published on May 9, 2026, the benchmark reports that Luce PFlash achieved 2.89× faster time to first token than Ollama with a 64K context.
LLM inference Post
10/08/2617 days ago x.com
CLI estimates VRAM before loading models
The do-i-have-the-vram tool estimates the GPU memory needed to load a model without loading it, helping assess whether it fits in available VRAM.
LLM inference Post
10/08/2617 days ago x.com
How to compare MTP inference on older GPUs
A May 16, 2026 publication reports llama.cpp throughput benchmarks with MTP for Qwen 3.6 models on three GTX 1080 Ti GPUs and describes a setup that can inform internal testing.
LLM inference Post
10/08/2617 days ago x.com
DEER proposes diffusion drafts to accelerate LLM inference
A publication dated December 18, 2025 describes DEER, which drafts tokens with diffusion models and verifies them with autoregressive models. The text claims up to 5.54× faster inference without quality loss.
LLM inference Post
10/08/2617 days ago x.com
DFlash and DDTree run Qwen3.6-27B on one RTX 3090
A post dated April 23, 2026 reports 73 tokens per second with speculative decoding on a single RTX 3090 and identifies architectural compatibility as a factor in running the model.
LLM inference Post
10/08/2617 days ago x.com
DS4 fork enables tensor-parallel inference across multiple machines
Published on June 9, 2026, a fork of the antirez/ds4 repository adds support for combining tensor parallelism with node roles to distribute inference across multiple machines and GPUs.
LLM inference Post
10/08/2617 days ago x.com
hf-mem estimates memory for new models
A publication dated February 17, 2026 presents a command-line tool for estimating memory requirements as model architectures change.
LLM inference Post
10/08/2617 days ago x.com
Local inference: performance and power on an RTX 3090
Tests of eight local LLMs on one RTX 3090 compared throughput and energy efficiency at power limits ranging from 100 W to 450 W.
LLM inference Post
10/08/2617 days ago x.com
LMCache reuses KV cache across GPU, CPU and disk
On January 30, 2026, the description of the open-source LMCache project presented an LLM-serving extension that reuses KV states across GPUs, CPUs and local disks, including repeated text chunks beyond prefixes.
LLM inference Post
10/08/2617 days ago x.com
Post assesses inference on two RTX Pro 6000 GPUs
Published on April 29, 2026, the post claims that a language model's experts are already quantized to 4 bits and could fit on two RTX Pro 6000 GPUs; it points to missing SM120 support in two serving options.
LLM inference Post
10/08/2617 days ago x.com
RTX 3090: efficiency and throughput in local inference
Tests reported on May 9, 2026 compared a 27-billion-parameter model on an RTX 3090, varying concurrency and power limits.
LLM inference Post
10/08/2617 days ago x.com
Serving MoE models with AWS EFA
On November 6, 2025, a Perplexity article described kernels for serving MoE models across AWS GPUs using EFA, without GPUDirect Async.
LLM inference Post
10/08/2617 days ago x.com
SWAA proposes more efficient inference for long contexts
A post dated December 15, 2025 describes training-free recipes to adapt full-attention language models to linear scaling and recover performance.
LLM inference Post
10/08/2617 days ago x.com
Concurrent Llama 3.1 8B inference on an RTX 3090
A Backprop post published on August 16, 2024 reports reasonable performance in a benchmark with more than 100 concurrent requests on a single GPU.
LLM inference Post
10/08/2617 days ago x.com
Layered inference on memory-limited GPUs
On September 25, 2024, the AirLLM publication described inference that loads one Transformer layer at a time from disk, as well as block quantization and compression support.
LLM inference Post
10/08/2617 days ago x.com
Jamba 1.5: 256K context and inference evaluation
On August 22, 2024, AI21 Labs announced the Jamba 1.5 Mini and Large models, combining Mamba and Joint Attention, with a 256K-token context window and features including ExpertInt8 and JSON output.
LLM inference Post
10/08/2617 days ago x.com
PHLoRA extracts LoRA adapters from fine-tuned models
Published on September 17, 2025, the post describes a method for extracting low-rank adapters from fine-tuned checkpoints without training data or gradients.
LLM inference Post
10/08/2617 days ago x.com
Inference throughput: SGLang and vLLM on A6000 GPUs
A post published on August 4, 2024 reports higher throughput with SGLang than with vLLM on four A6000 GPUs, but similar performance with smaller batches.
LLM inference Post
10/07/2618 days ago x.com
BITCOS reduces storage for ternary LLMs
In a post published on September 20, 2026, Intel describes BITCOS, a method that stores exact ternary LLM weights at 1.485 bits per weight and reports decode throughput gains.
LLM inference Post
10/07/2618 days ago x.com
How to assess KV cache reductions
A post reports an alleged reduction in KV cache use from 389,000 to 890 bytes per token over three years. The claim points to an opportunity to measure memory and cost in long-context systems.
LLM inference Post
10/07/2618 days ago x.com
How vLLM schedules requests and allocates the KV cache
The September 26, 2026 article traces a request through vLLM and explains per-iteration scheduling and KV cache block allocation.
LLM inference Post
10/07/2618 days ago x.com
KV Cache: Why Queries Are Not Stored
A post dated August 30, 2026 distinguishes the temporary use of queries from the storage of keys and values in the KV cache during inference.
LLM inference Post
10/07/2618 days ago x.com
Post describes calibration-guided low-bit compression of Hy4
A post published on August 29, 2026, says Tencent reduced the Hy4 preview from 1.5 TB to 200 GB by setting precision per layer. It reports results on two benchmarks.
LLM inference Post
10/07/2618 days ago x.com
Unsloth reports 1-bit quantization reducing model to 217 GB
In a post published on August 28, 2026, Unsloth reports dynamically quantizing GLM-5.3 to 1 bit: 217 GB versus 1.5 TB in BF16, with about 76% top-1 accuracy.
LLM inference Post
10/07/2618 days ago x.com
Serving models across GPU, RAM and NVMe
A September 12, 2026 post reports running a model on an RTX 5090 with 125.7 GiB of RAM and part of the model file on NVMe.

Let's assess your project

I reply with scope and a proposal.