Local LLM
Download or scan model files, prepare a runtime engine, then start your own model service.
Understand model files and engines first
Local LLM manages model files, inference engines, and service processes. It is included in the full edition and controlled by the optional-page switch in Settings.
Model files contain the weights. The runtime engine loads them and provides a service. Finishing a download does not mean the service has started, and installing an engine does not mean a model has been downloaded. GGUF is a common format for local model files.
Start with your first local model
- In Store, review the models and hardware guidance, and choose a quantized version suited to your machine. If you already have files, add a local model directory and scan it.
- Choose a runtime, detect the corresponding engine, and install it. Use the available controls to select a version or upgrade when needed.
- Select the model, port, compute resources, and context length. Start with conservative settings, then adjust model size and context once the basic setup works.
- Click Start, wait for output, and check the service status, address, and access key. If it fails, read the logs first rather than repeatedly starting it.
- Once the service works, use it through AI Assistant or another supported model entry.
- Click Stop when you are done to release resources used by the service. Leaving the page does not stop the service.
Choose a runtime for your system
| Runtime path | When it applies |
|---|---|
| llama.cpp | The basic runtime path on all three platforms; performance depends on the hardware and build |
| macOS Metal | Can use acceleration on supported Apple hardware |
| NVIDIA CUDA | Requires a supported NVIDIA environment and a corresponding build |
| CPU | A basic option without a suitable GPU; speed depends on the model and hardware |
| vLLM / SGLang | The current interface offers these options only on Linux when an NVIDIA or AMD GPU is detected |
Larger models and longer contexts generally need more memory. Allow room for your operating system and other work applications rather than judging requirements by the model file size alone.
Directories, models, and custom commands
Removing a scan directory only removes it from the directory list; it does not delete model files from disk. You can scan again after a download finishes.
Advanced users can set a custom engine path and launch arguments. When starting out, keep a working default setup. Do not change unfamiliar arguments, ports, and model formats all at once, or it becomes difficult to identify the cause of a failure.
Use it with AI Assistant
Confirm the local service works first, then choose an available model source in the assistant. Verify model serving and task execution separately: if a model is missing, check its files and scanning; if a connection fails, check the process, port, and key; if memory is insufficient, reduce the model size or context.
Running inference locally does not make every task offline. The assistant may still access the network when downloading dependencies, visiting web pages, or calling external services.