How to Run Open-Source AI Models on Your Own Computer: A Step-by-Step Guide for Kenyan Offices

TL;DR:

• You can run capable AI models on an office laptop or a small office server, with no internet needed once the model is downloaded and no per-user subscription.
• The easiest route is LM Studio (a free desktop app, no code). The most flexible is Ollama, and Ollama plus Open WebUI gives a whole office a private, browser-based AI chat on the local network.
• An 8 GB laptop can handle small models of roughly 1 to 4 billion parameters. Plan on 16 GB of RAM for 7 to 12 billion parameter models, and 32 GB or a good graphics card for larger ones.
• Local models are weaker than the best cloud tools on hard tasks, and someone must maintain them. They make most sense where data must stay in the building, internet is unreliable, or subscription costs block access.
• Running locally helps with Kenya's Data Protection Act, but it does not make you compliant on its own.

Why this matters for Kenyan offices

Many public offices want to use AI and hit the same three walls. The work involves citizen records, staff files or case notes that should not be pasted into a foreign company's website. The internet drops, especially outside Nairobi. And a monthly subscription for every officer is out of reach for most departments.

Running a model on your own machine addresses all three. Your documents and questions stay on your computer or server. Once set up, it works with the cable unplugged, and the software covered here costs nothing to download.

This post is part of our series on AI in government; for the bigger picture, start with our guide to AI in the public sector in Kenya. The first half is for managers deciding whether this is worth doing. The second half is for the ICT officer who will set it up.

The plain-English basics

What is an open-source or open-weight model?

An AI model is a very large file of numbers, called weights, that a program uses to produce text. Some companies keep their weights private, so you can only reach their models through their website or paid service. Others publish their weights for anyone to download: Meta (Llama), Google (Gemma), Alibaba (Qwen), Mistral AI, Microsoft (Phi) and IBM (Granite) all do this.

People call these "open-source models", but the label is loose. The Open Source Initiative's definition of open-source AI requires the weights, the full training code and enough information about the training data for someone else to rebuild the system, all under terms that let anyone use, study, change and share it. Most popular models only publish the weights, so the accurate term is open-weight.

The licence matters too. Gemma 4, Qwen 3.5 and Mistral 7B use the permissive Apache 2.0 licence. Microsoft's Phi-4-mini uses the MIT licence. Meta's Llama 3.2 has its own community licence with an acceptable use policy. These terms are usually workable for a government office, but someone should read the licence before a model goes into official use.

What does "running locally" mean?

The model runs on hardware you control: a laptop, a desktop or a server in your building. You download it once. After that, your questions and documents are processed on that machine and do not travel to an outside company.

What is quantisation?

Quantisation shrinks a model by storing its numbers with less precision, a bit like saving a photo at lower quality to make the file smaller. The model loses a little accuracy but becomes far lighter. Ollama's model pages show the effect: Google's Gemma 4 E4B is about 16 GB at full precision and about 6.6 GB in a common 4-bit version (tagged q4_K_M). The default downloads in LM Studio and Ollama are usually already quantised.

The decision: is local AI right for your office?

Factor	Cloud AI tools (ChatGPT, Copilot, Gemini and similar)	Local open-weight models
Privacy	Text goes to the provider's servers, usually outside Kenya	Text stays on your machine or office server
Cost	Free tiers have limits; paid plans are per user, per month	Software is free; you pay for hardware and staff time
Quality on hard tasks	Strongest available	Good for everyday drafting and summaries; weaker on complex analysis
Internet	Needed for every question	Needed only to download software and models
Maintenance	The provider handles it	Someone in-house must install, update and support it
Speed	Fast on any device	Depends on your hardware; can be slow on older laptops

When local makes sense

• You handle personal or sensitive data: citizen records, health information, staff files, procurement documents, investigation notes.
• Your connection is slow, costly or unreliable.
• You want many staff to use AI without paying for each person.
• At least one ICT officer is willing to own the setup.

When it does not

• The work needs the best possible reasoning, such as a complex policy analysis from scratch. A leading cloud model will do it better.
• Your computers are old, with 4 GB of RAM. Staff will be frustrated.
• Nobody can maintain it. An unpatched server holding sensitive chats is a new risk.

Many offices use both: a local model for personal data, and an approved cloud tool for work with public information.

The data protection angle

Kenya's Data Protection Act, 2019 and the Data Protection (General) Regulations, 2021 set rules on where personal data can go. Regulation 26 says personal data processed for certain strategic interests of the state must be processed through a server and data centre in Kenya, or at least one serving copy must be stored in a data centre in Kenya. The listed purposes include civil registration, elections, public finance systems, basic education and primary or secondary health care.

With a local model, the AI processing of that data never leaves your premises, which removes one cross-border question. Your other duties remain: lawful purpose, data minimisation, access control and security. Our guide to data protection and AI in the Kenyan public sector covers this in detail. This is general information, not legal advice; check specific cases with your data protection officer.

Hardware: what can your computer run?

Model size is described in parameters, written as 1B, 4B, 9B and so on (B means billion). More parameters usually means better answers and more memory.

A rough rule: a model's download size tells you roughly how much memory it uses once loaded, and you need extra room for the conversation and the operating system. The table is a rough guide based on download sizes listed on ollama.com in October 2026. Test on your own machines.

Your computer	What it can realistically run	Example models (download size)
Office laptop, 8 GB RAM, no graphics card	Small models, about 1B to 4B. Usable, not fast.	Llama 3.2 3B (2.0 GB), Phi-4-mini (2.5 GB), Granite 4 3B (2.1 GB), Qwen 3.5 4B (3.3–4.0 GB)
Laptop or desktop, 16 GB RAM	Mid-size models, about 7B to 12B	Mistral 7B (4.4 GB), Gemma 4 E4B (6.6–9.5 GB), Qwen 3.5 9B (6.6–7.6 GB)
Workstation or server, 32 GB RAM or more, ideally a graphics card with plenty of video memory	Larger models, about 26B to 31B	Gemma 4 26B (16–19 GB), Qwen 3.5 27B (17–20 GB)

Worth knowing:

• Graphics cards speed things up a lot. A model that fits in the graphics card's memory (VRAM) answers much faster than one running on the main processor. LM Studio recommends at least 4 GB of dedicated VRAM on Windows.
• Apple Silicon Macs work well. On M-series Macs the processor and graphics share one pool of memory, so a 16 GB Mac can run mid-size models. Ollama needs macOS 14 (Sonoma) or newer; LM Studio needs an Apple Silicon Mac on macOS 14 or newer and recommends 16 GB or more.
• Older Windows PCs may not qualify for LM Studio. It needs a processor with AVX2 support and recommends at least 16 GB of RAM.

Option A: LM Studio (easiest, no code)

LM Studio is a desktop app for Windows, macOS and Linux that looks much like ChatGPT. Since July 2025 it has been free to use at home and at work, with no form to fill; a paid Enterprise plan adds features like single sign-on. The app itself is closed source, while its command-line tool and developer libraries are open source. Check the current terms at lmstudio.ai before rolling it out across a department.

Step by step

1. Check the requirements on lmstudio.ai (see the hardware section above).
2. Download and install from lmstudio.ai/download, like any other program.
3. Find a model. Open the Discover tab and pick a suggested model or search by name, for example "Gemma", "Qwen" or "Llama". On an 8 GB laptop, start with a 3B or 4B model.
4. Download it. This is the only step that needs a good connection. Models are several gigabytes, so use office Wi-Fi, not mobile data.
5. Load the model. In the Chat tab, open the model loader at the top and select your model. This places it in your computer's memory.
6. Chat. Type a question and press Enter.

Using it offline

LM Studio's documentation says it can operate entirely offline once you have model files, and that nothing you type into a chat leaves your device. Searching for models, downloading them and updating the app need internet. A simple test: download your model, switch off Wi-Fi and ask a question.

You can also attach a .docx, .pdf or .txt file to a chat. Short documents are given to the model in full; long ones are searched for relevant passages (explained in the RAG section below).

Option B: Ollama (more flexible, uses the command line)

Ollama is a free, open-source tool that downloads and runs models through simple typed commands. It also runs a small local service that other programs can talk to, which makes it the base for an office-wide setup.

Install it

Windows (Windows 10 22H2 or newer): download the installer from ollama.com/download, or run this in PowerShell. It does not need administrator rights.

irm https://ollama.com/install.ps1 | iex

macOS (14 Sonoma or newer): download the app from ollama.com/download and drag it into Applications, or run this in Terminal.

curl -fsSL https://ollama.com/install.sh | sh

Linux (for example an Ubuntu server) uses the same command. Ollama then runs as a background service, which you can check with:

sudo systemctl status ollama

Download and run a model

All model names below were checked on ollama.com/library in October 2026. pull downloads a model; run starts a chat (and downloads first if needed):

ollama pull llama3.2
ollama run llama3.2

Type your question at the prompt, and type /bye to leave. To choose a specific model size, add a tag after a colon:

ollama run qwen3.5:4b
ollama run phi4-mini
ollama run gemma4:e2b

ollama run gemma4 on its own downloads the E4B version, which is 6.6 GB or more. On an 8 GB laptop, use gemma4:e2b or one of the smaller models.

Manage your models

ollama ls             # list downloaded models
ollama ps             # show loaded models, and whether they run on GPU or CPU
ollama stop llama3.2  # unload a model from memory
ollama rm llama3.2    # delete a model to free disk space

Models are stored in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows. On Windows you can move them to a bigger drive by setting a user environment variable called OLLAMA_MODELS to the new folder and restarting Ollama.

The local API

Ollama listens on port 11434 of your own computer, so internal tools your ICT team builds can send it questions. A quick test:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Summarise the purpose of a county integrated development plan in three sentences."}],
  "stream": false
}'

Ollama also offers an OpenAI-compatible address at http://localhost:11434/v1, so many tools written for OpenAI's API can point to your local model instead. By default Ollama only accepts connections from the same computer (127.0.0.1). Leave it that way unless you have a specific reason to change it.

Option C: a shared AI assistant for the whole office

A laptop setup helps one person. For a department, the better design is one capable machine running the model, with staff using it through their web browser. Open WebUI is a popular free interface for this: it looks like a familiar chat app, supports separate user accounts and works with Ollama.

What you need

• One server machine: ideally 32 GB of RAM and an NVIDIA graphics card, running Linux or Windows.
• Docker installed on it.
• The office local network, so staff can reach it from their own computers.

Set it up

The simplest route is Open WebUI's official image with Ollama already inside it. This command is from the Open WebUI quick-start guide. Replace your-secret-key with a long random value and keep it safe:

docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

If the server has an NVIDIA graphics card (with NVIDIA's container toolkit installed), add --gpus=all after -p 3000:8080. If Ollama is already installed directly on the machine, the quick-start guide gives a separate command for that case.

Then:

1. On the server, open http://localhost:3000 in a browser.
2. Create the first account. It becomes the administrator.
3. Download a model: go to Settings > Admin > Connections, select Manage on the Ollama connection and use Pull a model from Ollama.com.
4. Staff open http://<server-address>:3000 from their own computers on the office network.

Never run the container without the -v open-webui:/app/backend/data part: it stores all chats, users and settings.

Basic security

• Keep it off the public internet. Do not forward port 3000 on your router. For remote access, use the organisation's existing VPN.
• Control accounts. Open WebUI turns sign-up off once the administrator exists; if you enable sign-ups, new accounts wait for approval. Give each officer their own login.
• Back up the open-webui Docker volume, which holds every chat and account, and treat those backups as sensitive.
• Keep it updated. Name one person to update Docker, Open WebUI and the models on a schedule.
• Write a short usage rule on what data staff may enter and who can see what.

Open WebUI's licence asks you to keep its branding visible unless you have 50 or fewer users in 30 days or an enterprise licence. For most departments, that just means leaving the name and logo as they are.

Chatting with your own documents (RAG)

Staff will soon ask whether it can read the office circulars and answer questions about them. It can, through retrieval-augmented generation (RAG). The system splits your documents into passages, finds the ones most related to a question and gives only those to the model.

In Open WebUI, go to Workspace > Knowledge, create a knowledge base and upload text-based PDFs or text files. In a chat, type # to pick it. Open WebUI's default embedding model (the part that matches questions to passages) runs locally, so documents stay on your server.

Be careful with accuracy:

• Scanned PDFs are images, and the default reader may extract no text. Preview what was extracted, or turn on text recognition (OCR).
• Context length limits how much text the model sees at once. Ollama defaults to 4,096 tokens on machines with under 24 GB of graphics memory; Open WebUI's troubleshooting guide suggests 8,192 or more for complex documents.
• Small models make mistakes. Ask the model to quote the section it used, and check it against the source before anything goes into an official letter or report.

Choosing a model

Six reputable models available in both Ollama and LM Studio. Sizes are Ollama download sizes in October 2026; ranges cover different versions.

Model	Ollama name	Download size	Licence	Good for
Llama 3.2 3B (Meta)	llama3.2	2.0 GB	Llama 3.2 Community Licence	Quick drafting and summaries on modest laptops
Phi-4-mini 3.8B (Microsoft)	phi4-mini	2.5 GB	MIT	Reasoning for its size; long inputs (128K context)
Granite 4 3B (IBM)	granite4:3b	2.1 GB	Apache 2.0	Business tasks on low-spec machines
Qwen 3.5 4B and 9B (Alibaba)	qwen3.5:4b, qwen3.5:9b	3.3–4.0 GB; 6.6–7.6 GB	Apache 2.0	Broad multilingual use; reads images
Gemma 4 E2B and E4B (Google)	gemma4:e2b, gemma4:e4b	4.6–7.5 GB; 6.6–9.5 GB	Apache 2.0	Strong all-rounder built for laptops; reads images
Mistral 7B v0.3 (Mistral AI)	mistral	4.4 GB	Apache 2.0	General English writing; has no built-in moderation

What about Swahili? None of the model cards we checked lists Swahili as officially supported. Llama 3.2 officially supports eight languages and Phi-4-mini lists 23, without Swahili in either. Gemma 4 was pre-trained on more than 140 languages and Qwen 3.5 claims 201 languages and dialects, but neither names Swahili. Test two or three models on your own Swahili documents and have a fluent speaker judge the results.

A sensible start: llama3.2 and qwen3.5:4b on 8 GB laptops; gemma4:e4b and qwen3.5:9b on 16 GB machines.

Practical uses in a public office

Example prompts for clear, bounded tasks:

• Summaries: "Summarise this circular in five bullet points for heads of section. List any deadlines separately."
• Routine letters: "Draft a polite letter acknowledging a complaint about a delayed water connection. Promise a response within 14 days. Keep it under 200 words."
• Plain language: "Rewrite this notice in plain English for the public. Keep every date and amount unchanged."
• Minutes: "Turn these rough notes into minutes under three headings: decisions, action points with owners, and issues carried forward."
• Checking drafts: "List anything unclear, inconsistent or missing in this memo. Do not rewrite it."

Always review the output. A local model can sound confident and still be wrong.

Troubleshooting

Problem	Likely cause	What to try
Very slow answers	Model too big for memory, or running on the processor	Run ollama ps; switch to a smaller model; close heavy programs
Out of memory, or crashes on loading	Model needs more RAM or VRAM than you have	Use a smaller tag, for example qwen3.5:4b instead of 9b
Download keeps failing	Unstable connection	Use a wired or stronger connection; files are several gigabytes
Document answers ignore most of the file	Context too small, or a scanned PDF	Raise the context length; check extracted text; enable OCR
Open WebUI shows no models	No model downloaded, or no connection to Ollama	Check Settings > Admin > Connections and pull a model

Frequently asked questions

Is running AI locally really free?

The software here (LM Studio, Ollama, Open WebUI) is free to download, and the models listed have free licences. Your real costs are hardware, electricity and staff time.

Does it work without internet?

Yes, once the software and models are downloaded. You only need internet for downloads and updates.

Is a local model as good as ChatGPT or Copilot?

For everyday drafting, summaries and rewriting, small local models are often good enough. On complex reasoning and demanding writing, the leading cloud models are clearly better.

Does running locally make us compliant with the Data Protection Act?

It helps, because data does not leave your premises for AI processing. You still need a lawful purpose, access controls, security, backups and clear staff rules.

Can government offices use these models?

The licences shown allow use by organisations, including commercial use. Llama 3.2 adds an acceptable use policy. Have your legal or ICT governance team read the licence of any model before official use.

Where FuKazee fits in

FuKazee trains teams in practical AI. We assume nobody in the room knows AI yet, show the free version of every tool and never push one product. Since 2022 we have trained more than 3,000 people.

We can run a hands-on session for your ICT team on setting up and securing local models, or a session for a whole department on using AI in daily work, with clear rules on what data goes where. Our AI training for government in Kenya page explains how we work with public institutions.

To discuss a session for your team, email lead@fukazee.com

Sources

• Ollama README (GitHub)
• Ollama CLI reference
• Ollama FAQ
• Ollama on Windows
• Ollama on macOS
• Ollama on Linux
• Ollama API
• Ollama context length
• Ollama model library
• Ollama Gemma 4 tags
• Ollama Qwen 3.5
• Ollama Llama 3.2
• Ollama Phi-4-mini
• Ollama Mistral
• Ollama Granite 4
• LM Studio system requirements
• LM Studio getting started
• LM Studio offline operation
• LM Studio chat with documents
• LM Studio is free for use at work
• LM Studio CLI
• Open WebUI quick start
• Open WebUI with Ollama
• Open WebUI Knowledge
• Open WebUI RAG troubleshooting
• Open WebUI licence
• Gemma 4 E4B model card
• Gemma 4 licence
• Llama 3.2 3B Instruct model card
• Qwen3.5 9B model card
• Phi-4-mini-instruct model card
• Mistral 7B Instruct v0.3 model card
• Open Source Initiative, Open Source AI Definition 1.0
• Data Protection (General) Regulations, 2021 (ODPC)