GPU servers & LLM inference serving
Running a large language model on your own hardware is a systems problem: memory, bandwidth, scheduling and security. NVIDIA HGX is the eight-GPU baseboard platform behind many AI servers: the HGX B200 carries eight Blackwell SXM GPUs with 1.4 TB of total GPU memory and fifth-generation NVLink at 1.8 TB/s per GPU, so a big model can be sharded across GPUs that talk to each other fast.
What we do
- On-premises LLM architecture and GPU sizing
- vLLM and SGLang deployment with tuned batching
- Quantization and model bake-offs on your data
- Rack, power and cooling plans for HGX servers
- API gateway, authentication and audit logging
- Retrieval pipelines over private documents
- Air-gapped updates and model integrity checks
- Workstation-class serving with llama.cpp and Ollama
The serving software decides how much of that hardware you use. vLLM brings PagedAttention for efficient key-value memory, continuous batching, prefix caching, quantization (FP8, INT8, INT4, GPTQ, AWQ, GGUF) and an OpenAI-compatible server. SGLang, hosted under LMSYS, offers RadixAttention, prefix caching and multi-GPU parallelism for large language and multimodal models. llama.cpp and Ollama cover the other end: one workstation or laptop, CPU or GPU, quantized models in GGUF form.
We design the whole stack for regulated businesses: model choice and licence review, GPU and rack sizing, power and cooling, network isolation, authentication in front of the API, logging that does not leak prompts, evaluation against your own documents and a rollback plan. We do not resell hardware; we advise on and operate what you buy. See our AI and super-intelligence services.
Leading platforms we work with
- NVIDIA HGX
- vLLM
- SGLang
- llama.cpp
- Ollama
- TensorRT-LLM
- Triton Inference Server
- Hugging Face TGI
- LM Studio
Platforms and practices
- NVIDIA HGX B200, H200 and H100 servers
- Tensor and pipeline parallelism
- FP8 and INT4 quantization
- Continuous batching and prefix caching
- OpenAI-compatible inference APIs
- Mixture-of-experts models
- Model evaluation harnesses
- GPU monitoring and capacity planning
- Air-gapped model delivery
References
Sources last checked .
Product names are trademarks of their owners and are listed to describe our experience; they do not imply endorsement or partnership.
A private AI rack with a spare for every job
Two GPU servers run the large model as active-active replicas, so either can be patched or fail without an outage. A third handles smaller models, embeddings and OCR. The inference tier has no route to the internet, and every prompt and response is written to write-once storage.
671 billion parameters, 37 billion at work
A mixture-of-experts model keeps every expert in GPU memory but computes only a few per token. In a DeepSeek-V3‑class model each layer holds 256 routed experts plus one shared expert; a small router picks the best 8 for each token. All 671B parameters stay resident, about 37B are computed per token, and the experts are spread across the node's 8 GPUs so every GPU contributes to every token. Public architecture figures; the picked cells are an example.
Fast for one user, fast enough for everyone
Generation speed each user sees as more people share one inference node, in output tokens per second. A single user gets about 118 tok/s; at the busiest hour, with around 30 to 40 streams at once, each user still sees about 60, twice the 30 tok/s target and ten times the 5 to 8 tok/s at which people read. One node then produces roughly 1,950 tok/s in total. Modeled figures for an 8‑GPU B200‑class server, FP8 weights, speculative decoding; illustrative.
How we use GPU servers & LLM inference serving for clients
Where this matters most
Frequently asked questions
Direct answers to the questions we hear most. Something else on your mind?
Send us a messageWhy run a language model on our own hardware?
Typical reasons are privacy, regulation, predictable cost and offline operation. Documents never leave your network, and you control retention and logging. The trade-off is that you operate the stack. We design it for regulated businesses and say plainly when a hosted service fits better.
What is NVIDIA HGX?
The eight-GPU baseboard platform behind many AI servers. The HGX B200 carries eight Blackwell SXM GPUs with 1.4 TB of total GPU memory and fifth-generation NVLink at 1.8 TB/s per GPU, so a big model can be sharded across GPUs that talk to each other fast.
What do vLLM and SGLang do?
They are serving engines that decide how much of the hardware you use. vLLM brings PagedAttention, continuous batching, prefix caching, quantization and an OpenAI-compatible server. SGLang, hosted under LMSYS, offers RadixAttention, prefix caching and multi-GPU parallelism. We evaluate both against your workload.
Can a smaller business run a model without a GPU server?
Yes, for smaller models. llama.cpp and Ollama run quantized models in GGUF form on one workstation or laptop, on CPU or GPU. Quality and speed are lower than a large model on a GPU server, so we test against your own documents before recommending a size.
What security work is involved?
Network isolation, authentication in front of the API, and logging that does not leak prompts. We also review the licence of each model, run an evaluation against your documents, deliver models into air-gapped networks when needed, and prepare a rollback plan. The hardware needs power and cooling sized for it.
Do you sell the servers?
No. We do not resell hardware. We advise on sizing, power, cooling and rack layout, and we operate what you buy. NVIDIA, vLLM and the other names on this page describe experience with the products, not a partnership.