By You can run a capable AI assistant on hardware you own, and the install is the easy part. The usual arrangement is two pieces of software, a model runner that loads an open model and a web interface your staff log into, both free to download and both working within an afternoon. What takes longer is the graphics card, the licence terms on the model you chose, and the document settings that decide whether the answers are any good. Treat it as a small server you now operate, not a one-time install.
What the stack actually is
The model runner
The runner downloads an open model and answers requests from it. Ollama is the common choice. Its own documentation describes the starting point as "download Ollama and run a model on your computer" (Ollama docs).
The interface
On its own, a runner has no login screen, no user accounts and no document handling. That is the second piece. Open WebUI describes itself as "a self-hosted AI platform that's extensible, feature-rich, user-friendly, and built to run entirely offline," with "support for Ollama and OpenAI-compatible APIs" and install options across "Docker, Python, Kubernetes and desktop app" (Open WebUI docs).
What it needs from your hardware
This is where plans usually change. Ollama's hardware page says it "supports Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer," with driver 570 or newer required for compute capabilities 5.0 through 6.2. AMD cards on Linux need "the AMD ROCm v7 driver," and Vulkan covers some further GPUs on Windows and Linux (Hardware support). You can force it onto the processor with an invalid GPU ID, which works and is slow.
That page publishes no minimum memory figure, because model sizes vary too much for one. The model you want decides the card you need, so pick the model first and size the hardware to it rather than the reverse.
Three things self-hosting does not solve
The model still has a licence
Running a model on your own server does not make it yours. Google's Gemma Terms of Use require anyone distributing the model to include its use restrictions "as an enforceable provision" in downstream agreements, to give recipients a copy of the agreement, and to mark modified files with "prominent notices." Google may terminate the agreement for breach, at which point you delete all copies in your possession (Gemma Terms of Use). Those terms cover the models in their own appendix, and Gemma 4 sits under a separate licence. Read the terms for the exact model and version you install, and keep a record of what you accepted.
The software has terms as well. Open WebUI's licence says you "may NOT alter, remove, or obscure any 'Open WebUI' branding," with exceptions that include having "50 or fewer users in a 30-day period," being a substantive contributor with written permission, or holding an enterprise licence (Open WebUI licence). If part of the appeal was putting your own logo in front of customers, check that clause first.
Local is a setting, not a guarantee
Most businesses look at this to keep information on their own equipment. That holds only if the assistant is configured that way. Open WebUI connects to OpenAI-compatible APIs as readily as to a local runner, and Ollama's own documentation now offers cloud models to "run models without a local GPU" and can call "cloud models with Ollama, OpenAI, or Anthropic APIs" (Ollama docs). A staff member switching the model in a dropdown can change where the text goes. Decide which providers are configured, write it down, and check it after updates.
Your documents still need work
Pointing the assistant at your files is a separate job from installing it. Open WebUI supports document uploads and retrieval, but the model still needs enough context to use what is retrieved (Retrieval in Open WebUI). Ollama's current defaults depend on available graphics memory: 4,096 tokens below 24 GiB, 32,768 from 24 to under 48 GiB, and 262,144 at 48 GiB and above (Ollama context length). Check the actual setting for your installed model and increase it where needed, within the model's limits and available memory. A context window that is too small can leave retrieved information unused or only partly processed; increasing it does not by itself fix poor extraction or retrieval. Connecting Google Drive is more involved again, requiring a Google Cloud project with the Drive and Picker APIs enabled and credentials set as environment variables.
A hypothetical example
Take an Edmonton accounting firm that wants staff to query its own procedure manuals without sending client material outside. The install goes fine. Two weeks in, answers keep missing relevant details from long procedures, because retrieval and context settings were never tested against those files. One partner has been using a cloud model from the same dropdown without realizing. Nobody has read the model licence. None of that is a hardware problem, and all of it needed deciding before go-live.
What to check before you commit
- Name the one workflow this is for, and write down what a good answer looks like. If you cannot, you are buying a server to host a demo.
- Pick the model first, then size the graphics card to it. Confirm the card and driver are on the supported list.
- Read the licence for that specific model, and keep a copy of the version you accepted.
- Read the licence for the interface software, particularly if you intend to rebrand it or put it in front of customers.
- List every provider the interface is allowed to connect to, and remove the ones you have not approved.
- Check the context length and raise it if needed, then test retrieval with your longest real document.
- Decide who holds the administrator account, who approves new users, where backups go, and who applies updates.
- Test the failure cases. Ask something your documents do not cover and confirm the assistant says it does not know.
- Compare the full cost honestly: hardware, electricity, maintenance hours and downtime, against the subscription you were avoiding.
Where Lumin fits
Lumin implements AI for small and midsize service businesses in Edmonton by connecting approved information, existing software and a defined team workflow. Tools including OpenAI, Claude, Gemini and local models are assessed case by case rather than chosen in advance.
- We map one workflow and record a baseline before recommending self-hosted or hosted.
- We test with your team against representative tasks, missing-information cases and failure cases.
- Handover includes an operating guide, training, a named maintenance owner and confirmed subscription and usage costs, with one-time fees separated from recurring ones.
See AI integration or book a discovery call.