Pithos: C inference engine with hybrid expert offloading for Liquid AI LFM models

What happened and what to do

The Pithos repository describes a C inference engine for Liquid AI's LFM models. According to the developer, the engine keeps frequently used experts in VRAM, runs the others on the CPU, and reads weights from disk when needed. The reported results were 17.5 to 21.9 tokens per second, against 11.2 to 11.3 for a reference engine with CUDA acceleration, on a GTX 1650.

For a company, the case shows a concrete trade-off in local inference with limited GPU memory: splitting experts between GPU and CPU can raise throughput without changing hardware. A fitting project would start with a diagnosis of the machine fleet, measuring memory, latency and throughput with real workloads, before choosing between offloading, quantization or another model. The implementation could then include private inference service on the client's infrastructure, with an API, latency and memory monitoring on a dashboard, and quality regression tests.

How the consultancy can help

Diagnosis of local inference with limited VRAM: throughput and memory measurement on real workloads, comparison of offloading strategies, and architecture proposal on the client's infrastructure, with scoped implementation and operation.

Next step

If your team wants to run models with limited GPU memory on its own hardware, send a short description of the case, the available hardware and the latency requirements. We reply with a scoped proposal.