What happened

Red Hat released OpenShift 5 on 9 November. For AI, three changes stand out against OpenShift 4. The MCP server, which lets AI agents read and act on a cluster, moves from Technology Preview in OpenShift 4.22 to full support. The MCP gateway also reaches general availability. OpenShift Lightspeed changes from a chat assistant to an agent that diagnoses cluster problems and assesses update risk. OpenShift 5 is also the base of Red Hat AI Factory with NVIDIA.

Why it matters

Agents need fast model answers. On OpenShift AI, Red Hat runs model inference with vLLM and its llm-d router. In Red Hat's published test, llm-d sent each follow-up turn to the server that already held the conversation in its cache. The KV cache hit rate rose from about 62% to about 90%. The mean time to first token fell from 208.96 to 112.54 milliseconds.

The data

MetricvLLM, round-robinllm-d, prefix-awareUnit
Mean time to first token208.96112.54milliseconds
Later-turn time to first token (average)195.2386.89milliseconds
95th percentile time to first token727.26237.96milliseconds
KV cache hit rateabout 62about 90percent
Throughput0.520.49requests per second