LLM Server Requirements: GPU VRAM, RAM and Storage by Model Size
LLM server requirements begin with the exact inference workload: parameter count, weight precision, quantization format, context length, batch behavior, concurrent requests and latency target. Model weights are only the first VRAM consumer. KV cache, runtime workspaces, kernels and fragmentation can determine whether a model fits and how many requests it can serve.
The guide is for teams choosing self-hosted LLM hardware for development, an internal assistant or a production API. It focuses on stable model classes and formulas instead of one short-lived product version.
Use the LLM server requirements as a planning workflow: define the workload, collect a normal and peak baseline, identify the first limiting resource, shortlist two viable designs and test them with production-like data. Keep assumptions visible so a future review can update the model without repeating discovery.
Commercial options referenced in this guide: AI servers (https://unihost.com/dedicated/ai-servers/); GPU servers (https://unihost.com/dedicated/gpu/); dedicated servers (https://unihost.com/dedicated/)
Related Unihost reading: AI server hardware guide (https://unihost.com/blog/ai-servers-2025-hardware/); CPU vs GPU vs RAM for AI (https://unihost.com/blog/choosing-server-specs-ai/); GPU servers for AI and machine learning (https://unihost.com/blog/gpu-dedicated-servers-for-ai-machine-learning/)
LLM server requirements for inference vs training
Training stores weights, gradients, optimizer state and activations, so its memory and communication demands are much larger than inference. Inference primarily holds weights and runtime state, but long contexts and high concurrency can make KV cache substantial. A server sized for one interactive request may fail as a shared API.
Collect model format, tokens per request, context length, batch size, concurrent sequences, time to first token, tokens per second and GPU utilization during ordinary traffic and during a representative peak. Align infrastructure timestamps with application traces so the team can connect a slow request or failed job to the resource that was constrained. Averages are useful for cost planning, but p95, p99 and queue growth show whether short bursts are already damaging the service.
Specify inference, fine-tuning or training explicitly and size each environment from its own memory and throughput profile. The main planning risk is using a training recommendation for inference or testing one short prompt and assuming production concurrency will fit. Validate the choice with a workload trace with realistic prompt and output lengths at the target concurrency. Keep the test conditions and acceptance threshold in the runbook so later changes can be checked against the same baseline.
Capacity should also cover maintenance and failure behavior. Reserve enough room for monitoring, log rotation, security scanning, backup activity and the temporary loss of a node when the architecture promises continuity. This reserve is not a fixed percentage: derive it from the failure scenario and show it explicitly in the worksheet.
How parameters, precision and quantization affect VRAM for LLM
Weight memory is approximately parameter count multiplied by bytes per parameter. FP16 or BF16 uses about two bytes per parameter, 8-bit about one, and 4-bit about half a byte before format overhead. Quantization reduces weight memory, but quality, kernel support and speed must be tested with the chosen runtime.
Build the baseline from parameter count, storage format, loaded weight footprint, runtime workspace, KV cache per sequence and fragmentation. Record the same signals before and after every tuning or infrastructure change. If throughput rises while tail latency and errors remain controlled, the change created usable capacity. If queues grow or latency bends upward, the system has reached a limit even when one headline utilization number still looks comfortable.
Calculate the theoretical weight floor, add measured runtime overhead and then reserve VRAM for the target context and concurrency. Watch for treating the model file size as the total VRAM requirement or assuming every 4-bit format has identical overhead. Before ordering or migrating, use loading the exact quantized artifact in the production engine and running the longest expected requests. A documented rejection criterion is as important as a success criterion because it tells the team when to stop the rollout or move to the next capacity tier.
A useful decision has a scale path. State what can be expanded in place, what requires a restart or migration and which threshold starts that work. Procurement lead time, data-copy duration and change windows belong in capacity planning because a resource that can be added next month may not help during next week’s peak.
AI server sizing table by model class
The table below uses broad parameter classes so it stays useful as model names change. Ranges assume inference and include planning room beyond raw weights, but not unlimited context or concurrency. Multi-GPU placement may be required when one accelerator cannot hold the model and runtime state.
Use actual VRAM allocation at idle and under load, KV cache growth, batch occupancy and out-of-memory events as a small capacity model rather than a dashboard snapshot. Separate steady demand, scheduled work and exceptional peaks. The model should explain which resource saturates first, how long the saturation lasts and which customer or operational outcome changes at that point.
Select the smallest GPU class that meets latency and concurrency after validation, then keep a tested path to a larger or multi-GPU profile. The design can still fail through buying only enough VRAM to load the model with no space for production requests. Prove the intended behavior with a concurrency ramp with maximum context and output limits. Include monitoring, backups and security controls in the test because production overhead should not appear for the first time after launch.
Cost should be attached to a unit of successful work, such as an order, request, completed job, restored terabyte or accepted model response. That view prevents a cheap configuration from winning when it misses the latency or recovery target, and it prevents unused headroom from being treated as free.
Planning ranges for quantized LLM VRAM
| Model class | 4-bit weight floor | Practical single-node planning range | Typical direction |
|---|---|---|---|
| 7-8B parameters | About 4-5 GB before overhead | 8-16 GB VRAM | Development and light API |
| 13-14B | About 7-9 GB before overhead | 16-24 GB VRAM | Higher quality with moderate concurrency |
| 30-34B | About 16-20 GB before overhead | 24-48 GB VRAM | Large single GPU or carefully tuned runtime |
| 65-70B | About 35-40 GB before overhead | 48-80 GB VRAM or multiple GPUs | Production server with strict context planning |
| 100B and above | More than 50 GB before overhead | Multi-GPU, often 80 GB-class devices | Architecture and interconnect become first-order choices |
CPU, system RAM, NVMe and network requirements
The GPU does not remove the need for a balanced host. CPU handles tokenization, request routing and preprocessing. System RAM holds model files, page cache, CPU offload and multiple workers. NVMe for AI shortens model loading and supports vector indexes or datasets. Network matters for API traffic, model distribution and multi-node communication.
Measure CPU saturation during tokenization, host RAM peak, model load time, NVMe throughput and latency, network transfer and GPU idle gaps with enough resolution to capture bursts and enough duration to expose leaks, cache effects and background jobs. Keep the workload mix visible. A test dominated by easy requests can report healthy averages while the expensive path is already queueing.
Prevent the host from starving the LLM inference GPU by matching CPU lanes, RAM channels, PCIe topology and storage to the accelerator count. Do not ignore pairing an expensive GPU with too little system RAM, slow storage or insufficient PCIe connectivity. Confirm the recommendation through cold start, model reload, concurrent request and data ingestion tests. Record which assumption has the lowest confidence and retest that assumption first when traffic, data or software changes.
Keep software efficiency in the model. Query plans, cache policy, compression, batching and concurrency limits can change resource demand more than one hardware tier. Re-run the same evidence set after tuning so the final purchase reflects the improved system rather than an avoidable defect.
Single GPU vs multi-GPU and concurrency
A single GPU is simpler and often gives the best latency when the model fits. Multi-GPU is necessary when weights and runtime state exceed one device or when throughput requires more replicas. Tensor parallelism, pipeline parallelism and independent replicas have different interconnect and scheduling costs.
Create a repeatable evidence set from per-GPU memory, inter-GPU transfer, synchronization time, batch efficiency, queue time and tokens per second per request. Store workload inputs beside the results, including software version, data size, cache state and concurrency. This turns the next capacity review into a comparison instead of another estimate from memory.
Prefer independent replicas for throughput when the model fits one GPU; use model parallelism when memory capacity requires it. The most expensive mistake would be adding GPUs without an engine that scales efficiently or without enough CPU and network capacity to feed them. Reduce that uncertainty with one-GPU and multi-GPU benchmarks at the same quality, context and concurrency. Keep an upgrade or rollback path that does not depend on the already constrained component.
Capacity should also cover maintenance and failure behavior. Reserve enough room for monitoring, log rotation, security scanning, backup activity and the temporary loss of a node when the architecture promises continuity. This reserve is not a fixed percentage: derive it from the failure scenario and show it explicitly in the worksheet.
Example profiles for development, production API and high concurrency
A development profile favors flexibility and enough VRAM for the chosen quantized model. A production API adds ECC where available, headroom, monitoring and a defined replica strategy. High-concurrency inference prioritizes aggregate VRAM, memory bandwidth, efficient batching, queue control and failure isolation. A GPU server for inference should be selected against measured tokens per second, time to first token and concurrency at the intended model format.
Track accepted requests per second, time to first token, output tokens per second, queue delay, GPU memory and error rate at the component and service levels. Resource headroom is valuable only when it preserves the latency, correctness and recovery objectives that matter to the business. Use the first consistently constrained metric to guide the next test.
Size against the service-level objective and cost per accepted response, not a synthetic maximum token rate. A common failure mode is mixing interactive and batch traffic without admission control, causing long prompts to block short requests. Use a trace-driven test with production request distribution before treating the configuration as production-ready. Review the result with the application owner as well as the infrastructure team.
A useful decision has a scale path. State what can be expanded in place, what requires a restart or migration and which threshold starts that work. Procurement lead time, data-copy duration and change windows belong in capacity planning because a resource that can be added next month may not help during next week’s peak.
Reference inference profiles
| Profile | GPU direction | Host direction | Operational priority |
|---|---|---|---|
| Development | One GPU sized for the target quantized model | 64-128 GB RAM, fast NVMe | Fast iteration and reproducibility |
| Production API | One or more replicas with VRAM headroom | 128-256 GB RAM, redundant NVMe where needed | Latency, monitoring and rollback |
| High concurrency | Multiple high-memory GPUs or replicas | High-core CPU, 256 GB RAM and above, high-speed network | Batching, queue limits and fault isolation |
Compare AI servers (https://unihost.com/dedicated/ai-servers/) and GPU servers (https://unihost.com/dedicated/gpu/).
Cost, data privacy and management checklist
Compare hardware cost with utilization, request volume and staff effort. Self-hosting can improve data control and predictable capacity, but the team owns model updates, access policy, patching, monitoring, abuse controls, backups and incident response. Sensitive prompts and outputs need retention and audit decisions.
Collect cost per accepted response, GPU utilization, idle hours, incident load, patch age, access events and data retention during ordinary traffic and during a representative peak. Align infrastructure timestamps with application traces so the team can connect a slow request or failed job to the resource that was constrained. Averages are useful for cost planning, but p95, p99 and queue growth show whether short bursts are already damaging the service.
Choose an AI dedicated server when steady use, privacy or control justifies the operational responsibility, and add management when internal coverage is incomplete. The main planning risk is treating self-hosting as a hardware purchase while ignoring runtime security and ongoing operations. Validate the choice with a production-readiness review covering performance, security, recovery and ownership. Keep the test conditions and acceptance threshold in the runbook so later changes can be checked against the same baseline.
Cost should be attached to a unit of successful work, such as an order, request, completed job, restored terabyte or accepted model response. That view prevents a cheap configuration from winning when it misses the latency or recovery target, and it prevents unused headroom from being treated as free.
Request an AI workload configuration
Frequently Asked Questions
How much VRAM does an LLM need?
Start with parameter count multiplied by bytes per parameter, then add runtime workspace and KV cache for context and concurrency. A 4-bit model uses roughly half a byte per parameter before format overhead. Test the exact model and engine because practical allocation differs.
Can an LLM run on CPU only?
Yes, especially smaller quantized models and low-throughput tasks. CPU inference is usually slower and may need substantial system RAM, but it can suit development, batch work or privacy-sensitive services without strict latency. Benchmark the target model on the exact CPU.
How much RAM is required for self-hosted inference?
System RAM must cover the OS, model files, page cache, CPU offload, preprocessing and workers. Many single-GPU projects start around 64-128 GB, while larger or multi-GPU systems may need 256 GB or more. Measure peak host allocation during realistic concurrency.
When is a multi-GPU server necessary?
Use multiple GPUs when the model and runtime state cannot fit one device or when independent replicas are needed for throughput and availability. Confirm that the inference engine supports the chosen parallelism and that PCIe or GPU interconnect will not erase the gain.
