Qwen3.8-27B batch serving on an RTX 3090 capped at 250W: self-reported figures

What happened and what to do

The post describes running the Qwen3.8-27B model on an RTX 3090 limited to 250W, serving 64 simultaneous chats at about 1035 tok/s combined and 127 tok/s for a single chat. The setup supports context up to 150K and was assembled with two Docker Compose commands. The figures are self-reported by the author and show batch throughput for a 27B model on consumer GPU hardware.

For a company, the case suggests that private inference with mid-sized models can run on accessible hardware, provided sizing is measured rather than assumed. A possible implementation starts with a load test using the client's own workflow, measuring tokens per second, latency per request, power draw and memory use at long context. The infrastructure can then be packaged in versioned containers, with a configured power limit, continuous monitoring and request routing that keeps sensitive data inside the client's network.

How the consultancy can help

Wendelmaques can diagnose whether a model of similar size meets your load, with a benchmark on your own traffic, and implement private inference on your infrastructure, with containers, power limits, throughput monitoring and ongoing operation.

Next step

If your team wants to evaluate private inference on your own GPUs, send a short description of your case, the chat volume and the context requirements. We will reply with a scoped proposal for diagnosis and implementation.