Offline AI cheat sheet
The commands from the course on one page. Model names and paths are examples: use your own. On Windows PowerShell, set a variable with $env:OLLAMA_HOST="0.0.0.0"; on Mac and Linux, export OLLAMA_HOST=0.0.0.0.
ollama pull gemma4:12b | Download a model |
ollama run gemma4:12b | Chat in the terminal (type /bye to leave) |
ollama list | Models on disk |
ollama ps | Models loaded now, their size, and GPU or CPU |
ollama stop gemma4:12b | Unload a model from memory |
ollama rm gemma4:12b | Delete a model from disk |
ollama show gemma4:12b | Context length, quantisation and licence |
ollama create mybot -f Modelfile | Save a model with your own system prompt and settings |
OLLAMA_HOST=0.0.0.0 | Listen on the network, not only this computer (inside a trusted network only) |
OLLAMA_NUM_PARALLEL=4 | Answer up to 4 requests at once |
OLLAMA_CONTEXT_LENGTH=8192 | Default context length |
OLLAMA_KV_CACHE_TYPE=q8_0 | 8-bit KV cache to save memory (needs flash attention on) |
OLLAMA_KEEP_ALIVE=30m | How long a model stays loaded after use |
OLLAMA_MODELS=D:\models | Where model files are stored |
LM Studio command line Lesson 9.1
lms ls | Models on disk |
lms load <model> | Load a model |
lms server start | Start the local API server (port 1234) |
lms ps | Models loaded now |
llama-cli -m model.gguf -p "Hello" | One prompt in the terminal |
llama-server -m model.gguf -c 8192 -ngl 99 --port 8080 | API server: 8,192-token context, all layers on the GPU |
llama-server -m model.gguf -c 32768 -np 4 | 4 parallel slots sharing the context |
--cache-type-k q8_0 --cache-type-v q8_0 | 8-bit KV cache |
--split-mode layer --tensor-split 3,2 | Split a model across two cards |
llama-bench -m model.gguf | Measure prefill and writing speed |
pip install mlx-lm | Install |
mlx_lm.generate --model <model> --prompt "Hello" | One prompt |
mlx_lm.server --model <model> | API server (port 8080) |
mlx_lm.lora --model <model> --train --data ./data | Train a LoRA adapter |
vllm serve <model> --max-model-len 16384 | vLLM API server (Linux, NVIDIA) |
vllm serve <model> --tensor-parallel-size 2 | Split across 2 cards |
python -m sglang.launch_server --model-path <model> | SGLang API server |
curl http://localhost:11434/api/tags | Is Ollama running? Lists models |
curl http://localhost:11434/v1/models | The same, OpenAI-style |
curl http://localhost:8080/health | Is llama-server ready? |
nvidia-smi -l 2 | NVIDIA memory, load, heat and power every 2 seconds |
curl http://localhost:8080/metrics | llama-server counters (start it with --metrics) |
Activity Monitor > Memory | Mac memory pressure (keep it green) |
Task Manager > Performance > GPU | Windows graphics memory and load |
sha256sum model.gguf | Linux |
shasum -a 256 model.gguf | Mac |
certutil -hashfile model.gguf SHA256 | Windows |
Python: ask a local model Lesson 11.2
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")
reply = client.chat.completions.create(
model="gemma4:12b",
messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(reply.choices[0].message.content)
Install the library once with pip install openai. To use another engine, change the address: LM Studio http://localhost:1234/v1, llama-server and MLX http://localhost:8080/v1, vLLM http://localhost:8000/v1.
- Load your model and front end.
- Turn off Wi-Fi and unplug the cable.
- Ask a question and upload a document.
- Everything still works: you are offline.
See also the glossary. Commands checked against each tool's documentation in November 2026; flags change between versions, so run a tool with --help if one is refused.