Running an open-source LLM on your own server: the honest requirements

TL;DR: The constraint that determines everything is VRAM. A model must fit in your GPU's memory to run at usable speed, and the memory required scales with parameter count and precision. Roughly: an 8-billion-parameter model at 4-bit quantisation fits in 8GB, a 14B needs about 16GB, and a 70B needs 48GB or more.

CPU-only inference works and is slow — usable for batch jobs, painful for anything interactive. And the economics rarely favour self-hosting on cost alone: GPU instances are expensive enough that API pricing wins for most workloads until volume is very high. Self-host for data control, predictable cost, or offline operation, not because you expect it to be cheaper.

Why VRAM is the whole question

A language model's weights must be loaded into memory for inference. On a GPU that means video memory, and if the model does not fit, you either offload part of it to system RAM, which is dramatically slower, or you cannot run it.

The rough arithmetic: memory required is approximately the parameter count multiplied by the bytes per parameter, plus overhead for the context and activations.

At full 16-bit precision, each parameter takes two bytes. So an 8-billion-parameter model needs roughly 16GB for weights alone before any context.

At 4-bit quantisation, each parameter takes about half a byte, so the same model needs roughly 4 to 5GB for weights, leaving room for context in an 8GB card.

This is why quantisation is not optional for anyone running on consumer or modest cloud hardware. It is the difference between a model fitting and not fitting.

Overhead beyond weights: the key-value cache for context grows with sequence length and batch size, and it is substantial at long contexts. The serving framework itself uses memory. Plan for weights plus meaningful headroom rather than weights exactly.

System RAM is not a substitute. Partial offloading to system RAM works and is slow enough that most people abandon it for interactive use. If a model does not fit in VRAM, the practical answer is a smaller model or better quantisation, not offloading.

The practical implication: decide your VRAM budget first, then choose the largest model that fits comfortably with context headroom. Working the other way round — picking a model and hoping — is how people end up with something that technically runs and is unusable.

Quantisation, and what it costs you

Quantisation reduces the numerical precision of model weights, trading a small amount of output quality for a large reduction in memory.

The common levels:

16-bit is the reference precision most models are released at. Maximum quality, maximum memory.

8-bit roughly halves memory with quality loss that is difficult to detect in most tasks.

4-bit roughly quarters memory against 16-bit, with a modest and generally acceptable quality reduction. This is where most self-hosted deployments sit, and the common formats in this range are the practical default.

Below 4-bit saves further memory with degradation that becomes noticeable, particularly on reasoning-heavy tasks.

What quality loss actually looks like: not obvious breakage but subtle degradation — slightly worse instruction following, more errors on multi-step reasoning, occasional incoherence at longer outputs. On simple classification or extraction tasks the difference is often negligible. On complex reasoning it is more apparent.

The practical rule: a larger model at 4-bit generally outperforms a smaller model at 16-bit for the same memory budget. If you have 16GB, a 14B model quantised to 4-bit is usually a better choice than an 8B at 8-bit.

Test on your actual task. Quantisation impact varies by workload, and the only reliable evaluation is running your own representative prompts through both and comparing. Benchmarks will not tell you whether it is adequate for your specific use.

Model size against hardware

Approximate guidance. Actual requirements vary by model architecture, quantisation format and context length.

VRAM	Comfortable at 4-bit	Tight at 4-bit	Typical use
6–8GB	7–8B	13B with short context	Classification, extraction, simple chat
12GB	13–14B	20B	General assistant tasks
16GB	14B with long context	30B short context	Solid general use
24GB	30–32B	70B heavily quantised	Strong general capability
48GB	70B	Larger with tight context	Near-frontier open models
80GB+	70B at higher precision	Very large models	Production serving at quality
CPU only, 16GB RAM	7–8B, slowly	13B, very slowly	Batch jobs, no interactivity
CPU only, 32GB RAM	13B, slowly	30B, impractically	Batch jobs

Reading the table: the difference between comfortable and tight is context length and concurrency. A model that fits with a 2,000-token context may not fit with 32,000, and serving two concurrent requests roughly doubles the cache requirement.

What model size buys you: capability, particularly on reasoning, instruction following and less common knowledge. Small models are surprisingly capable at focused tasks — classification, extraction, summarisation, structured output — and noticeably weaker at open-ended reasoning.

The most common mistake: assuming you need a large model. Many production uses of language models are extraction, classification or templated generation, where a well-prompted 8B model performs adequately at a fraction of the hardware cost. Test the small model first.

CPU inference: when it is acceptable

Running on CPU without a GPU is entirely possible and the speed is the problem.

What to expect: on a modern server CPU, a 7 to 8 billion parameter model at 4-bit produces output at a rate that feels slow for conversation and is perfectly acceptable for a background job. Larger models are proportionally slower.

Where it works:

Batch processing where latency does not matter: classifying a queue of documents overnight, summarising records, generating content asynchronously.

Low-volume internal tools where an occasional wait is acceptable.

Development and testing before committing to GPU hardware.

Where it does not:

Anything interactive. Users will not wait.

Anything with concurrency, since CPU inference does not parallelise across requests the way GPU serving does.

Anything latency-sensitive in a user-facing path.

What helps: more memory bandwidth matters more than core count for inference, so newer platforms with faster memory outperform older ones with more cores. Smaller models help disproportionately. Aggressive quantisation helps.

The honest assessment: CPU inference on a standard VPS is a legitimate option for asynchronous work and a poor one for anything a person is waiting on. Test with your actual workload before designing around it, because the difference between "slow but fine" and "unusable" depends entirely on whether anything is waiting.

Choosing a serving stack

Three broad options, suited to different situations.

Simple local runners. Tools built around straightforward model management, exposing an API and handling quantised formats well. Easy to install, good documentation, sensible defaults, and they work on CPU and GPU alike.

Best for: getting started, single-user or low-concurrency use, development, and CPU inference.

Limitation: throughput under concurrent load is not their focus.

High-throughput serving frameworks. Built for production serving with continuous batching, efficient memory management for the key-value cache, and substantially better throughput when handling many simultaneous requests.

Best for: production serving with real concurrency.

Limitation: more complex to set up, GPU-focused, and generally expects the model to fit fully in VRAM.

Low-level inference libraries. Maximum control over quantisation formats and hardware use, excellent CPU support, and the foundation several higher-level tools are built on.

Best for: constrained hardware, unusual quantisation requirements, embedded use.

Limitation: more work to operate.

The practical path: start with a simple runner to validate that a given model at a given quantisation actually does your task well. Only move to a throughput-oriented framework once you have concurrency to justify the added operational complexity.

Expose an OpenAI-compatible API if your tooling expects one, which most serving options support. It makes switching between self-hosted and API-backed inference a configuration change rather than a rewrite, which is valuable while you are still deciding.

The cost comparison, honestly

This is where enthusiasm usually outruns arithmetic.

GPU instance pricing is substantial. A cloud GPU instance capable of serving a mid-sized model costs meaningfully more per month than a general-purpose VPS, and it costs that whether you run one request or a million.

API pricing is per token and has fallen considerably. For most small business workloads — thousands rather than millions of requests a month — the API cost is a fraction of a dedicated GPU instance.

The breakeven is high. You need sustained, high-volume usage for a dedicated GPU to cost less than API calls, and many organisations that assume they are past that point are not. Calculate your actual monthly token volume before assuming.

Costs people omit:

Idle time. A GPU instance costs the same at 3am with no traffic. API pricing is zero when you are not using it.

Your time. Model management, updates, monitoring, troubleshooting, evaluating quality after a model change.

Capacity planning. A GPU instance has a fixed ceiling, so a traffic spike degrades or fails rather than scaling.

Owned hardware changes the arithmetic if you already have suitable GPUs, since the marginal cost is electricity. This is genuinely cheaper for organisations with existing hardware, and it comes with the responsibility of running it.

The honest conclusion: self-host for reasons other than cost, then check whether the cost is acceptable. Self-hosting to save money works only at high sustained volume, and the crossover is further away than most people assume.

When self-hosting is the right call

The genuine reasons, which are not primarily financial.

Data cannot leave your infrastructure. Where a contractual obligation, a client requirement or a sector expectation means content must not be sent to a third-party API. Note that this is usually a contractual or policy requirement rather than a legal one under India's data protection framework, which permits cross-border transfer to jurisdictions not specifically restricted. But the requirement is real when a client imposes it.

Predictable cost at high volume. Fixed monthly cost regardless of usage is genuinely valuable when volume is high and variable, since it removes billing surprises.

Offline or air-gapped operation. Where there is no reliable internet path to an API, or where the environment is deliberately isolated.

Latency control. An inference server in the same region as your application removes API round-trip time. For Indian applications calling APIs hosted elsewhere, this can be a meaningful improvement, though it must be weighed against slower inference on modest hardware.

Model stability. APIs change: models are deprecated, behaviour shifts between versions, and prompts that worked stop working. A self-hosted model is frozen until you change it, which matters for applications where consistency is more important than capability.

Fine-tuning on proprietary data, where you want a model adapted to your domain and do not want that data used elsewhere.

Experimentation without per-token cost, which genuinely encourages more exploration.

What is not a good reason: assuming self-hosting is more private by default. It is more controlled, which is different. A self-hosted model on a poorly secured server is not more private than a reputable API with a proper data processing agreement, as covered in the DPA checklist.

Context length, the hidden memory cost

The requirement people plan for last and get surprised by.

Model weights are a fixed cost. The key-value cache, which holds the attention state for the tokens in context, grows with sequence length and with the number of concurrent requests.

Practical consequences:

A model that fits comfortably with a short context may not fit with a long one. Loading a model successfully does not mean it will handle your longest prompts.

Concurrency multiplies the cache requirement. Serving four simultaneous requests at long context can require several times the cache memory of a single request.

Long-context use cases — document analysis, large retrieval contexts, extended conversations — are considerably more memory-hungry than short prompts at the same model size.

Mitigations: cap the maximum context length your server accepts rather than allowing arbitrary input. Limit concurrency explicitly. Choose a smaller model if long context is essential to your task. Some serving frameworks implement cache quantisation and paged attention, which help substantially.

Test at your realistic maximum, not with short prompts. A deployment validated with 500-token prompts and then given 20,000-token documents will fail, and it will fail under exactly the load you deployed it for.

Operating an inference server

Treat model files as large artifacts. They are several gigabytes each, and downloading them consumes bandwidth and disk. Plan storage for the models you keep, and prune ones you have stopped using.

Pin model versions. Model releases are updated, and behaviour changes. Record exactly which model and quantisation you deployed, so you can reproduce and roll back.

Evaluate before switching. A newer or larger model is not automatically better for your task. Keep a small evaluation set of representative prompts with expected outputs, and run it whenever you change model or quantisation.

Monitor GPU memory and utilisation, not just whether the process is running. Approaching the VRAM ceiling is actionable; hitting it fails requests.

Set request timeouts and concurrency limits. An unbounded queue of long-context requests will exhaust memory.

Do not expose the inference API publicly without authentication. An open inference endpoint is compute someone else will happily use, and the bill or the resource exhaustion is yours.

Log requests and latency, so you can see whether performance is degrading and whether your context assumptions hold.

Back up configuration, not the models themselves, which are re-downloadable. What matters is your serving configuration, prompts, and evaluation set.

Reassess periodically. API pricing continues to fall and open models continue to improve. A decision that was correct a year ago may not be now, in either direction.

FAQs

How much VRAM do I need to run an LLM?

Roughly the parameter count multiplied by bytes per parameter, plus context overhead. At 4-bit quantisation, an 8-billion-parameter model needs about 8GB of VRAM comfortably, a 14B needs around 16GB, a 30B needs 24GB, and a 70B needs 48GB or more. Leave headroom for the context cache rather than sizing to the weights exactly.

Can I run an LLM without a GPU?

Yes, on CPU, and it is slow. A 7 to 8 billion parameter model at 4-bit produces output at a rate acceptable for background batch jobs and frustrating for anything interactive. Memory bandwidth matters more than core count. Use it for asynchronous work, not for anything a person is waiting on.

What is quantisation and does it hurt quality?

Quantisation reduces the numerical precision of model weights to save memory. Four-bit quantisation roughly quarters memory against 16-bit with modest quality loss that most tasks tolerate well. The degradation shows most on multi-step reasoning and least on classification and extraction. Test with your own representative prompts rather than trusting benchmarks.

Is a bigger model at 4-bit better than a smaller model at 16-bit?

Usually yes, for the same memory budget. A 14B model quantised to 4-bit generally outperforms an 8B at 8-bit on most tasks. The exception is workloads particularly sensitive to precision, which is why you should evaluate on your own task rather than applying the rule blindly.

Is self-hosting an LLM cheaper than using an API?

Rarely, until volume is high and sustained. A dedicated GPU instance costs the same whether idle or busy, while API pricing is per token and has fallen considerably. Calculate your actual monthly token volume before assuming you are past the breakeven, because many organisations that believe they are, are not.

What is the best way to serve a self-hosted model?

Start with a simple local runner to validate that your chosen model and quantisation actually perform your task well, since that is the question that matters most. Move to a throughput-oriented serving framework only once you have real concurrency justifying the operational complexity. Expose an OpenAI-compatible API so switching is a configuration change.

Why does my model fit when loading but fail on long prompts?

Because the key-value cache holding attention state grows with context length and concurrency, and it is separate from the weights. A model that loads comfortably at a short context can exhaust VRAM at a long one. Cap maximum context and concurrency explicitly, and test at your realistic maximum rather than with short prompts.

Do I need to self-host for data privacy?

Not usually as a legal matter, since India's framework permits cross-border transfer to jurisdictions not specifically restricted. Self-hosting matters when a client contract or sector policy requires it, or where you want to remove a processor from your data flow. Note that a poorly secured self-hosted server is not more private than a reputable API with a proper processing agreement.

What size model do I actually need?

Probably smaller than you assume. Many production uses are extraction, classification, summarisation or structured output, where a well-prompted 8B model performs adequately. Larger models matter most for open-ended reasoning and less common knowledge. Test the small model first, because the hardware cost difference is substantial.

Can I run an LLM on a normal VPS?

On CPU, yes, for small quantised models and asynchronous workloads. Interactive use effectively requires a GPU, which means a GPU instance rather than a standard VPS. If your agent workload calls a model API rather than running one locally, a standard VPS is fine, which is a different situation entirely.

How do I know if quantisation has degraded my results?

Keep an evaluation set of representative prompts with expected outputs and run it whenever you change model or quantisation. Automated benchmarks will not tell you whether quality is adequate for your specific task, and the degradation from quantisation tends to be subtle rather than obvious, showing as slightly worse instruction following rather than clear breakage.

Should I expose my inference API to the internet?

Not without authentication. An open inference endpoint is free compute that others will find and use, and the resource exhaustion or the bill is yours. Bind it to an internal network or a VPN interface, and require authentication if it must be reachable more broadly.

Conclusion

VRAM decides what you can run, and everything else follows from it. Pick your memory budget first, then the largest model that fits with genuine context headroom, then verify it actually does your task well before building anything on it.

Quantisation is not a compromise to avoid; it is the thing that makes self-hosting viable at all on affordable hardware. Four-bit is where most deployments sensibly sit, and a larger model quantised harder generally beats a smaller model at higher precision.

Two assumptions are worth checking before you commit. That you need a large model — most production uses are extraction and classification where an 8B performs adequately. And that self-hosting saves money — a GPU instance costs the same when idle, and the crossover against API pricing sits at higher volume than people expect.

Self-host for control, predictability, offline operation or model stability. Those are good reasons. Cost is usually not one until volume is high and sustained, and being honest about that upfront saves an expensive detour.

And test at your real context length with real concurrency. A deployment validated on short prompts will fail on long documents, and it will fail under exactly the workload you built it for.

HostCloud runs Linux VPS plans from ₹999 a month on NVMe with full root access and Indian data centres, well suited to agent orchestration and CPU-based batch inference. Details at https://hostcloud.in.