Broadcast inference stack: vLLM vs SGLang vs TensorRT-LLM

The demonstration finishes before anyone has time to complain. Then your broadcast inference stack receives work from every feed, an archive job joins the queue, and the next useful result arrives after its deadline.

That is the test a production serving system has to pass. A fast response to one prompt establishes a starting point. Production capacity depends on whether the serving system keeps completing the right work as new work continues to arrive.

vLLM, SGLang and TensorRT-LLM all deserve evaluation. Choose between them using the model, hardware, arrival pattern and operational deadline you actually have. A generic tokens-per-second ranking cannot make that decision for a master-control room or a production team.


AI-generated editorial illustration. This is a conceptual workload, not an Amira product interface or a real installation.

What changes when inference becomes a continuous broadcast workload?

The source keeps producing work even when the server falls behind. That makes arrival rate, queue age and result deadlines essential parts of the specification.

This article concerns large language model (LLM) serving and supported multimodal analysis for jobs such as transcript classification and metadata generation. It does not imply that any text-generation endpoint is a complete live speech-recognition or captioning system. Likewise, streaming output tokens over an API does not establish continuous audio ingestion, timestamp handling or caption delivery.

The same distinction applies to prototypes. Ollama supports parallel requests when memory permits and exposes queue controls; its documentation describes overload responses when that queue fills. Calling it single-user-only would be wrong. The weak evidence is the single-user demonstration, regardless of the software used. Ollama concurrency and queue documentation.

Translate feeds into an arrival rate

Consider an illustrative metadata service receiving prepared transcript windows from 24 feeds. Each feed supplies one job every five seconds. Assume each prompt contains 1,200 tokens, including instructions and context, and each completed response contains 60 generated tokens.

The offered load is:

• 24 ÷ 5 = 4.8 requests per second.
• 4.8 × 1,200 = 5,760 input tokens per second, before accounting for prefix-cache reuse.
• 4.8 × 60 = 288 output tokens per second.

These are chosen workload assumptions, not a GPU benchmark. Input processing and output generation have different costs, so adding those token rates into one capacity figure would be misleading.


Original Amira Labs calculation. The values describe an illustrative offered workload, not measured engine performance or a recommended captioning delay.

Now specify when the result is useful. Suppose this metadata application requires a complete, validated response within three seconds of the transcript window closing. That deadline includes application queuing, transfer, inference and output validation. The five-second collection window is earlier in the timeline; it has not disappeared from the age of the underlying material.

Also test the arrival pattern. Twenty-four jobs arriving together every five seconds have the same average rate as evenly spaced jobs, but create a different queue. Use both patterns when they are plausible. A long-run average can hide the burst that misses the deadline.

This is separate from choosing a location. Our edge, on-prem and cloud placement guide covers where the service runs. Here, the question is how it behaves once the work arrives.

How should you compare vLLM, SGLang and TensorRT-LLM?

Shortlist engines that support your exact model and hardware, then compare deadline performance under the same workload. The table gives starting points for evaluation, not a universal ranking.

Documented capabilities: vLLM project overview, SGLang project overview, and TensorRT-LLM scheduling documentation. The proposed tests are editorial recommendations.

vLLM: examine prefill, decode and memory pressure

For an autoregressive language model, prefill processes the input prompt; decode generates subsequent output tokens. Continuous batching allows requests to join and leave ongoing execution instead of requiring every request in a fixed batch to finish together.

vLLM's V1 tuning guide describes chunked prefill, which splits long prompts and schedules that work alongside decoding. Its settings trade input progress against output-token latency. The same guide warns that insufficient key-value, or KV, cache space can cause requests to be preempted and recomputed, increasing end-to-end latency. vLLM optimisation and tuning.

The KV cache holds attention state associated with token context. Fitting model weights into memory therefore does not establish how many active requests will fit. Use the longest permitted prompts and outputs in a stress test, as well as the typical cases. The broadcast GPU sizing guide covers the accompanying hardware decision.

vLLM is a useful baseline candidate when its support matrix matches the workload. That is a testing recommendation, not a claim that it wins every comparison.

SGLang: test the reuse your application actually has

SGLang's RadixAttention reuses common prompt prefixes, and its runtime also provides continuous batching and structured outputs. A broadcast application that repeatedly supplies the same instructions or reference material is a reasonable candidate to test. Prefix reuse is not exclusive to SGLang. SGLang runtime capabilities.

Use the real prompt structure. Repeating an identical synthetic prompt throughout a benchmark can produce a cache-hit pattern that changing transcripts never achieve. Include requests with new context and changed instructions.

Be especially careful when copying tuning recommendations. SGLang's guidance explicitly labels its large-batch throughput advice as offline batch inference. It separately explains how weights, KV cache, graph buffers and activations compete for memory. A setting intended to keep an offline queue full is not evidence that it meets a live result deadline. SGLang hyperparameter tuning.

TensorRT-LLM: identify the release before describing the workflow

TensorRT-LLM warrants evaluation when the target deployment is built around supported NVIDIA hardware. Its documented in-flight batching interleaves requests to use the GPU efficiently; that capability still needs testing against the application's deadline. NVIDIA's scheduling guide.

Avoid repeating an outdated blanket requirement to convert every model and build a TensorRT engine first. NVIDIA's latest migration guide, updated August 27, 2026, states that the TensorRT engine backend has been removed and PyTorch is the sole execution backend for the documented workflow. AutoDeploy remains built on that backend. Older pinned releases can have different workflows. TensorRT-LLM backend migration guide.

Record the actual release or commit and use its matching instructions. Check the feature combination as well as the individual features: NVIDIA publishes combinations marked supported, unsupported, untested or with known issues. A feature list alone is insufficient. TensorRT-LLM feature combination matrix.

Account for maintenance status

The lifecycle of the serving project belongs in the decision. Hugging Face's TGI maintenance-mode change was merged on December 11, 2025; GitHub shows the repository was archived on March 21, 2026. These are different events. TGI maintenance-mode change and archive notice.

An existing deployment does not stop working because a repository is archived. It does need a deliberate plan for support, security fixes and migration testing. For a new deployment, assess who will maintain the selected runtime throughout the service's expected life.

Project documentation was checked on August 30, 2026. Moving documentation branches and supported-feature matrices are not promises about every released container.

How do you benchmark useful capacity instead of a fast demonstration?

Measure how much valid work completes before the application's deadline at the intended offered load. Call this application goodput, and define exactly what qualifies.

A server can complete requests successfully after their outputs have become too late to use. It can also generate a syntactically valid response containing the wrong classification. Track timeliness and task quality separately, then count only results that satisfy the defined acceptance criteria as useful completions.

Keep the clocks and counters separate

Record time to first token, inter-token latency and complete-response latency. For a producer reading a draft as it appears, first-token responsiveness matters. For a downstream system waiting for a complete metadata object, it does not establish completion.

Add timestamps outside the model server: source-window close, application admission, dispatch and acceptance of the validated result. Server telemetry may omit an upstream queue or downstream processing.

vLLM exposes separate waiting-time, prefill, decode, first-token and cache-related metrics. Use those to investigate delays rather than treating GPU utilisation as the service-health indicator. vLLM production metrics.

Report offered, admitted, completed, rejected, expired and accepted-on-time counts with a consistent accounting convention. Include queue age and the slow end of the latency distribution. State how unfinished requests at the end of a test are counted. Silently discarding them makes an overloaded service look healthier.

Do not let the load generator hide overload

A fixed number of clients that wait for responses before submitting more work will slow their own arrival rate when the server slows. That can be appropriate for interactive users. It does not reproduce media jobs that continue to arrive on schedule.

Replay the planned arrival timing and record any work waiting at the generator itself. vLLM's benchmark documentation explicitly warns that combining a maximum-concurrency cap with a requested arrival rate can reduce the actual request rate. It also supports latency-based goodput objectives. Those tool metrics still need the application's external timing and quality checks. vLLM serving benchmark controls.

Use a workload sequence that exposes the failure modes

Begin with the target steady load and run long enough to observe sustained queue and memory behaviour. Then exercise a synchronised arrival burst, long prompts, cold caches, a controlled restart and the intended failover path. Do this in an authorised test environment.

Test the representative mix too. A background archive analysis may have a very different output length from a short live metadata request. If they share capacity, show that the live workload remains within its deadline. If they cannot coexist reliably, separate their queues or compute allocation and test that arrangement.

Preserve the model and tokenizer revisions, prompt template, precision, input/output distributions, runtime version, GPU configuration and cache state with every result. Without those, a comparison is difficult to repeat and easy to misinterpret.

Where do serving optimisations stop helping?

They stop being a sufficient answer when the work exceeds sustainable capacity, the cache assumptions fail or the operational path is incomplete.

Prefix caching is useful when requests share reusable input. vLLM's documentation explicitly distinguishes saving prefill work from generating new output tokens; caching does not remove the decode work. Test both warm and cold behaviour, including after a restart. vLLM prefix-caching limits.

A larger concurrency setting also does not create memory. SGLang documents the trade-off between KV-cache capacity and memory needed for other execution components. vLLM documents the latency cost of memory-driven preemption. Treat increasing concurrency as an experiment with measurable consequences. SGLang memory tuning, vLLM preemption guidance.

Nor should every prototype become a distributed serving project. If a small, measured workload meets its deadline on a simpler supported deployment, additional routing and disaggregation may create operating work without a demonstrated benefit. Keep the simpler option in the comparison.

The article does not present an Amira-run head-to-head benchmark. There is no defensible universal throughput-collapse percentage or winner here. Aggregate token throughput can improve with batching while individual deadlines are missed. Identify that boundary on your workload rather than importing a number from a different model or GPU.

Finally, serving software is one component. It does not by itself provide media ingest, programme identity, caption QC, editorial approval or a tested recovery process. A generated classification should not silently become an authorised action. Review runtime and dependency licences separately from model rights and content permissions; installing an engine does not settle those questions.

What should you test before approving a production inference stack?

Approve a reproducible operating envelope: a documented workload, deadline, quality target and recovery behaviour that the team can verify.

1. Write the workload contract. Specify feeds, job cadence, input/output sizes and burst patterns. Choose the deadline's starting and ending events.
2. Run a matched comparison. Keep the model and task constant across compatible engines. Record unsupported configurations rather than silently substituting a different model or precision.
3. Test the overload policy. Decide which work waits, which expires and which follows an approved fallback. Verify that cancellation actually releases work where supported. Never let missing analysis masquerade as a successful check.
4. Rehearse recovery and handover. Demonstrate restart, rollback and the first useful result after recovery. Give operators the queue-age alert, affected programme identity and escalation contact they need.

Ask any supplier to show the same evidence: what load was offered, what fraction produced acceptable results on time, what failed, and how the service recovered. Include operating effort in the decision alongside hardware capacity.

A good serving-stack choice leaves the engineering team with a repeatable test and the shift with a clear failure policy. The next additional feed should enter a measured operating envelope, not restart the experiment.

Sources

• Ollama: FAQ, concurrent processing and queue controls.
• vLLM: Project capabilities and supported serving features.
• vLLM: Optimisation and tuning, including preemption and chunked prefill.
• vLLM: Automatic prefix caching and its limits.
• vLLM: Production metrics.
• vLLM: Serving benchmark controls, arrival rates and goodput.
• SGLang: Project capabilities.
• SGLang: Hyperparameter tuning and memory allocation.
• NVIDIA: TensorRT-LLM in-flight batching and request scheduling.
• NVIDIA: TensorRT-LLM backend removal migration guide, updated August 27, 2026.
• NVIDIA: TensorRT-LLM feature combination matrix.
• Hugging Face: TGI maintenance-mode pull request, merged December 11, 2025, with repository archive notice dated March 21, 2026.