Module 15: Scaling up: from one laptop to a team server · Lesson 15.2 · about 25 min
Serving engines built for many users
Ollama, LM Studio, llama.cpp's server, vLLM and SGLang compared, with what each is good at and a starting command for each.
Module 15 · about 25 minutes · read first: 15.1 and 7.5. You’ll be able to choose a serving engine for your group and start it with one command.
1. Why this matters
The engine is the program that loads the model and answers requests. On a laptop, any engine will do. With many users, the engine decides how many people you can serve on the same hardware, sometimes by a factor of five or more.
Think of a restaurant. The model is the kitchen. The engine is the front-of-house team: how well they seat guests, share tables and keep orders moving decides how many people eat each evening, with the same kitchen.
2. The five engines, side by side
| Engine | Best for | Runs on | Model files | Batching |
|---|---|---|---|---|
| LM Studio | One person, or a few, with a friendly app | Windows, Mac, Linux | GGUF, MLX on Mac | Basic |
| Ollama | Simple shared server, small teams | Windows, Mac, Linux | GGUF (its own library) | A few parallel requests |
llama.cpp llama-server |
Full control, any hardware, small to medium teams | Almost anything, including CPU only | GGUF | Continuous, set slots with -np |
| vLLM | Many users on NVIDIA (and some AMD) cards | Linux servers | Hugging Face folders, AWQ, GPTQ, FP8 | Continuous, PagedAttention |
| SGLang | Many users, agents and chat with shared prompts | Linux servers | Hugging Face folders | Continuous, shares repeated prompt starts |
All five offer the OpenAI-style API from Lesson 11.1, so you can start simple and move up without rewriting anything.
3. Starting each one
The model names below are examples; use the ones you picked in Module 7.
# Ollama: allow 4 requests at once, listen on the network
OLLAMA_NUM_PARALLEL=4 OLLAMA_HOST=0.0.0.0 ollama serve
# llama.cpp: 4 slots sharing a 32,768-token context, metrics on
llama-server -m gemma-4-12b-Q4_K_M.gguf -c 32768 -np 4 --host 0.0.0.0 --port 8080 --metrics
# vLLM (Linux, NVIDIA): the model's name exactly as on its Hugging Face page, or a local folder
vllm serve <model-name-or-folder> --max-model-len 16384
# SGLang (Linux, NVIDIA)
python -m sglang.launch_server --model-path <model-name-or-folder> --port 30000
Each prints the address it is listening on. Point Open WebUI (Lesson 9.5) or your scripts at it.
4. Choosing
- Up to about 10 light users, or a Mac: Ollama or LM Studio. Least setup, good enough.
- 10 to 50 users, or agents that send many requests, on any hardware:
llama-serverwith 4 to 16 slots. - 50 or more users, or heavy agents, on NVIDIA Linux servers: vLLM or SGLang. They are harder to set up (drivers, Python environments, matching CUDA versions) but serve several times more people from the same card.
The memory rule from Module 4 still holds: the model plus every user’s KV cache must fit. vLLM grabs most of the card’s memory at start (about 90% by default) and hands it out as KV cache pages; that is normal.
How many conversations fit on one card? Walk through it with the numbers from the box below:
The card. 32 GB of graphics memory.
Take off the model. The weights take 8 GB and the engine needs about 2 GB of working space: 32 − 8 − 2 = 22 GB left for conversations.
One conversation's memory. This model's KV cache uses 0.125 MB per token. A full 8,192-token chat needs 0.125 × 8,192 = 1,024 MB, which is 1 GB.
Divide. 22 GB ÷ 1 GB = 22 conversations at full length at the same moment.
Why paging helps. Most chats are short. vLLM's PagedAttention hands out memory in small pages only as a chat grows, so in practice many more than 22 people can be in a batch at once.
Maths and principle
Principle: memory left after the weights decides how many conversations fit at once.
Equation: conversations at once ≈ (card memory − weights − working space) ÷ (KV cache per token × tokens per conversation).
Analogy: a restaurant: the kitchen is the model, the tables are KV cache. A bigger kitchen does not seat more guests; more floor space does.
Try it: 32 GB card, 8 GB weights, 2 GB working space, 0.125 MB per token (Lesson 8.4’s 8B example), 8,192 tokens per chat? (22 GB ÷ (0.125 MB × 8,192 = 1 GB) = 22 conversations at full length.)
- card memory
- the graphics card's total memory in GB
- weights
- the model file loaded into that memory
- working space
- room the engine needs for its own sums, about 1 to 2 GB
- KV cache per token
- memory each conversation uses for every token it holds (Lesson 8.4)
- tokens per conversation
- how long each chat may grow (its context)
- Physical meaning
- The kitchen (weights) takes a fixed part of the building; the rest is tables (KV cache). A bigger kitchen does not seat more guests, more floor space does. Shorter chats, or a smaller quant, leave room for more tables.
“vLLM is the fastest, so it is always the right choice.” vLLM shines with many users on NVIDIA Linux servers. On a Mac, or for a handful of people, it is the wrong tool: harder to set up, and it does not target Apple chips. Start with Ollama or LM Studio, and move up only when the queue says so; your scripts will not need to change.
5. Worked example
Made-up numbers. A college computer lab has a Linux server with two 32 GB cards and 60 students. They first try Ollama: at the start of class, answers take a minute to begin. They switch to vLLM with the same 12B model and see the first token in under 2 seconds for most students, because vLLM keeps 30 or more conversations in one batch.
6. Practice
- Which engine would you pick for a 3-person office with a Mac Studio, and why?
- What does
-np 4with-c 16384give each user inllama-server? - Why can you change engines without changing your Python scripts?
Answers
- Ollama or LM Studio: few users, a Mac (vLLM targets Linux GPU servers), least setup.
- Four slots of 4,096 tokens each.
- They all serve the same OpenAI-style API; only the base address and the model name change (Lesson 11.4).
New words: serving engine, slot, PagedAttention, safetensors, AWQ and GPTQ (quantised formats used by vLLM).
- LM Studio and Ollama suit one person or a small team; llama-server suits 10 to 50 users on any hardware; vLLM and SGLang suit many users on NVIDIA Linux servers.
- All of them speak the OpenAI-style API, so you can switch engines without rewriting scripts.
- Conversations at once ≈ (card memory − weights − working space) ÷ (KV cache per token × tokens per conversation).
- vLLM loads Hugging Face folders (and AWQ, GPTQ, FP8); llama.cpp and Ollama load GGUF.
- vLLM taking about 90% of the card at start is normal: it is reserving room for KV cache pages.
Check yourself
Sources
- vLLM documentation: OpenAI-compatible server and supported hardware (official page), https://docs.vllm.ai
- SGLang documentation (official page), https://docs.sglang.ai
- llama.cpp server README: --parallel, --cont-batching, --metrics (official page), https://github.com/ggml-org/llama.cpp/tree/master/tools/server
- Ollama FAQ: concurrency settings (official page), https://docs.ollama.com/faq
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.