#Vllm

Latent Reasoning Escapes the Lab, Bolted Onto DeepSeek-V4

A 36M-parameter head makes V4-Flash think silently in latent space — and the serving hacks matter more than the benchmarks.

Article · 2w ago2

Your Agents Are Waiting on the CPU, Not the GPU

Agentic workloads moved the inference bottleneck off the accelerator, and hardware ratios are quietly following.

Article · 3w ago5

DeepSeek V4 Flash on One AMD GPU Took Nine Patches

A single 192 GB MI300X now serves the full checkpoint — proof of AMD's hardware edge and its lingering software gap.

Article · 3w ago1

The Real Cost of 'Just Use vLLM,' According to Netflix

Netflix's Model Runtime team details the version pinning, GIL workarounds, and preemption bugs behind production LLM inference.

Article · 1mo ago2

Autoscale GPU Inference on EKS with Karpenter and Spot Instances

Let Karpenter provision and bin-pack spot GPU nodes under a vLLM server, then delete them when traffic drops.

Tutorial · 1mo ago0

The Real Cost of Running SOTA LLMs Locally

Why brute-forcing frontier models onto local hardware fails, and how clever pipeline decomposition offers a sustainable path forward.

Article · 1mo ago4

Ornith-1.0: Coding Models That Train Their Own Agent Scaffolds

By optimizing both the reasoning loop and the code output, these MIT-licensed models bring native agentic capabilities to local hardware.

Article · 2mos ago0

Serve an Open-Source LLM at Scale with vLLM on a Rented GPU Instance

Go from a bare cloud VM to a production-ready, OpenAI-compatible inference server in under an hour, using vLLM's continuous batching to hit thousands of output tokens per second on a single GPU.

Tutorial · 2mos ago0