AI and infrastructure news | Wendelmaques

Skip to content

Project23

LLM inference Project
10/08/2617 days ago github.com
dflash-mlx v0.1.5 expands runtime and serving features
On May 3, 2026, version 0.1.5 added a unified CLI, RAM and SSD caching, prefix matching, and experimental support for long contexts.
LLM inference Project
10/08/2617 days ago github.com
Running Llama 70B on an RTX 3090 with ntransformer
On February 22, 2026, the ntransformer repository page says its C++/CUDA inference engine runs Llama 70B on an RTX 3090, transferring data from NVMe to the GPU without passing through the CPU.
LLM inference Project
10/08/2617 days ago github.com
llama.cpp adds Multi-Token Prediction support
On May 17, 2026, a pull request to the llama.cpp repository added support for MTP heads. The report cites 65 tokens per second for Qwen3.6-27B on an RTX 3090, versus 38 without MTP.
LLM inference Project
10/08/2617 days ago github.com
llmfit helps assess models for local hardware
The llmfit repository, published on GitHub on February 21, 2026, analyzes local hardware to identify compatible models and estimate inference speed.
LLM inference Project
10/08/2617 days ago github.com
Lucebox explores speculative inference on consumer GPUs
Published on May 9, 2026, the post presents Lucebox, a speculative inference server for heterogeneous hardware, and reports that Qwen3.6-27B reaches 120–200 tokens per second on an RTX 3090.
LLM inference Project
10/08/2617 days ago github.com
Community recipes for serving LLMs on RTX GPUs
Published on May 24, 2026, the repository collects model-independent configurations for serving LLMs on RTX 3090, 4090, and 5090 GPUs, using vLLM, llama.cpp, and ik_llama.
LLM inference Project
10/08/2617 days ago github.com
Tencent presents HPC-Ops operators for LLM inference
On January 27, 2026, Tencent presented HPC-Ops, an open-source library of operators for LLM inference, with GPU operations and multinode communication support.
LLM inference Project
10/08/2617 days ago github.com
Training neural networks on the Neural Engine: evaluating the ANE repository
Published on March 2, 2026, the ANE repository describes neural network training using private APIs reverse-engineered from Apple's Neural Engine. Learn how to evaluate this approach before adopting it.
LLM inference Project
10/08/2617 days ago github.com
Voice agents: parallel models may reduce silence
A contribution to Gradbot, published on June 5, 2026, describes sending a brief response to TTS while another model reasons and calls tools in the background.
LLM inference Project
10/08/2617 days ago github.com
NVIDIA Model Optimizer gathers model compression techniques for inference
NVIDIA published a post on Model Optimizer, a library with quantization, distillation, pruning, neural architecture search and speculative decoding, with exportable checkpoints for vLLM, SGLang, TensorRT-LLM and TensorRT.
LLM inference Project
10/08/2617 days ago github.com
LLM Compressor brings model compression techniques to vLLM
In a post published on August 16, 2024, Neural Magic introduced a library of compression techniques intended to reduce inference latency while preserving accuracy.
LLM inference Project
10/08/2617 days ago github.com
Mirage compiles LLMs into persistent megakernels
Published on June 19, 2025, the publication describes a compiler that transforms LLMs into persistent megakernels and claims latency reductions of 1.2 to 6.7 times.
LLM inference Project
10/07/2618 days ago github.com
Colibri streams MoE experts from disk for local inference
A pure-C inference engine loads experts from mixture-of-experts models from disk, offering an approach to evaluate local execution on hardware with limited memory.
LLM inference Project
10/07/2618 days ago github.com
Reproducing multi-GPU serving and thermal profiling
A GitHub repository published on July 30, 2026 presents a reproducible GLM-5.2 EXL3 MTP3 serving configuration and thermal profile for four RTX PRO 6000 Blackwell GPUs.
LLM inference Project
10/07/2618 days ago github.com
Model communication on Huawei Ascend NPUs
The publication, dated September 30, 2026, describes a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs.
LLM inference Project
10/07/2618 days ago github.com
Inference configuration across two DGX Sparks
A pull request dated July 31, 2026 updates the default checkpoint and installs the encoder on both ranks in a two-DGX-Spark configuration.
LLM inference Project
10/07/2618 days ago github.com
Silent failures in LLM serving: a practical diagnosis
Published on July 28, 2026, the GitHub repository catalogs LLM serving failures, their symptoms, mechanisms, checks, and fixes.
LLM inference Project
10/07/2618 days ago github.com
LLM inference with limited VRAM
A post dated July 17, 2026 describes a framework for heterogeneous inference and fine-tuning, distributing experts across GPU and CPU.
LLM inference Project
10/07/2618 days ago github.com
Open book covers 12 chapters on AI infrastructure
Published on September 13, 2026, the book covers model architectures, accelerators, inference, and training. Its scope can help companies identify concrete infrastructure challenges.
LLM inference Project
10/07/2618 days ago github.com
Ridgeline: profiling LLM inference with the roofline model
Published on July 8, 2026, the text describes Ridgeline, a PyTorch-based profiler that applies the roofline model to analyze LLM inference workloads.
LLM inference Project
10/07/2618 days ago github.com
Strata brings local MoE inference to GPUs with 8 GB
On October 1, 2026, the Strata repository page described running the 125-billion-parameter Qwen3.8-Flash-Next MoE model on NVIDIA GPUs with at least 8 GB of VRAM.
LLM inference Project
10/07/2618 days ago github.com
TinyRouter explores routing across open models
A publication dated July 4, 2026 describes a router with about 10,000 parameters and reports benchmark results, while noting that more experiments are needed.
LLM inference Project
10/07/2618 days ago github.com
Zlaya: Laya inference on CPU with Zig
Published on September 23, 2026, the GitHub repository presents Zlaya, a compact CPU inference engine for Laya, implemented in Zig.

Let's assess your project

Wendelmaques Pereira

I reply with scope and a proposal.

Looking for something else?Frequently asked questions Guides Contact