Running a large language model on your own hardware is a great way to learn, experiment, and keep your data private. In this guide, I install Ollama and Open WebUI on an Ubuntu 24.04 VM, pull the Llama 3.2 model, and get a ChatGPT-style web interface running with Docker Compose.
What you will build: Ollama running as a systemd service on port 11434, and Open WebUI running in a Docker container on port 3010, talking to Ollama over your LAN.
Prerequisites: an Ubuntu 24.04 VM with internet access, root or sudo privileges, and at least 24 GB RAM (more is better for larger models). Commands below are shown as root; prefix with sudo if you are a regular user.
PART 1: INSTALL AND CONFIGURE OLLAMA
Step 1: Download and Install Ollama
# curl -fsSL https://ollama.com/install.sh | sh
Step 2: Verify the Ollama Service Status
# systemctl status ollama
The service should show active (running).
Step 3: Download the Llama 3.2 Model
# ollama pull llama3.2
Step 4: Verify the Downloaded AI Model
# ollama list
Step 5: Test the Llama 3.2 Model
# ollama run llama3.2
Example prompt:
>>> what is uname -r ?
To exit the interactive session, type /bye or press Ctrl + D.
Step 6: Verify the Ollama API Listening Port
# ss -tulpn | grep 11434
By default, Ollama listens on 127.0.0.1:11434, so the API is reachable only from the local machine.
Step 7: Install the Vim Text Editor
# apt install vim
Step 8: Configure the Ollama Service to Listen on All Interfaces
Create a systemd override directory:
# mkdir -p /etc/systemd/system/ollama.service.d/
Create or edit the override configuration:
# vi /etc/systemd/system/ollama.service.d/override.conf
Add the following configuration:
Environment="OLLAMA_HOST=0.0.0.0:11434"
Note: Listening on 0.0.0.0 exposes the Ollama API on all network interfaces. Restrict access using a firewall and trusted network rules.
Step 9: Reload the Systemd Configuration and Restart Ollama
# systemctl restart ollama
Step 10: Verify the Ollama API Is Listening on All Interfaces
# ss -tulpn | grep 11434
The output should now show the service listening on port 11434 across all interfaces (*:11434).
PART 2: INSTALL DOCKER AND DOCKER COMPOSE
Step 11: Install Docker Prerequisite Packages
# apt install apt-transport-https ca-certificates curl software-properties-common
Step 12: Add the Docker Repository and GPG Key
The old apt-key method is deprecated. Use the keyring method from the official Docker Engine installation guide (https://docs.docker.com/engine/install/ubuntu/):
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
# chmod a+r /etc/apt/keyrings/docker.asc
# echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
Step 13: Update the APT Package Index and Check Docker Availability
# apt update
# apt-cache policy docker-ce
The candidate version should now come from download.docker.com.
Step 14: Install Docker Engine
# apt install docker-ce
Step 15: Download and Install Docker Compose
# chmod +x /usr/local/bin/docker-compose
Tip: Alternatively, install the Compose v2 plugin with apt install docker-compose-plugin and use docker compose (with a space) everywhere. The standalone binary above is invoked as docker-compose (with a hyphen).
Step 16: Verify the Docker Compose Version
# docker-compose --version
PART 3: DEPLOY OPEN WEBUI
Step 17: Create the Docker Compose Configuration for Open WebUI
Create the docker-compose.yml file:
# vi docker-compose.yml
Add the following configuration:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "3010:8080"
environment:
- OLLAMA_BASE_URL=http://192.168.2.230:11434
volumes:
- open-webui-data:/app/backend/data
restart: unless-stopped
volumes:
open-webui-data:
Important: Replace 192.168.2.230 with the actual IP address of the Ubuntu VM running Ollama, if different. Make sure port 11434 is reachable from the Open WebUI container.
Step 18: Start the Open WebUI Container
Run this from the directory containing docker-compose.yml:
# docker compose up -d
(If you installed the standalone binary only, use docker-compose up -d.)
Step 19: Verify the Open WebUI Container Status
# docker ps
You should see the open-webui container with status Up, mapping port 3010.
Step 20: Access the Open WebUI Dashboard
Open your web browser and navigate to:
http://192.168.2.230:3010
Replace the IP address with your Ubuntu VM's address if necessary. On first launch, create the admin account.
Step 21: Verify Ollama Model Integration with Open WebUI
1. Open the Open WebUI dashboard.
2. Create your local account or sign in.
3. Go to Admin Settings → Connections and confirm the Ollama API connection points to http://192.168.2.230:11434 and is enabled.
Figure 1: Open WebUI Connections page with the Ollama API connection enabled.
4. Open a new chat.
5. Check that llama3.2 appears in the model selector.
6. Select the model and submit a test prompt.
Figure 2: Open WebUI new chat screen with llama3.2:latest selected.
If the model does not appear, verify the OLLAMA_BASE_URL setting and confirm the container can reach the Ollama API, for example with curl http://192.168.2.230:11434/api/tags.
PART 4: EXPLORING THE OPEN WEBUI SETTINGS
Once the chat is working, open your profile menu (bottom-left) and go to Settings. Open WebUI groups its options into user settings (personal to your account) and Admin settings (server-wide). Here is a quick tour of what each section does.
Note: Menu names and options vary slightly between Open WebUI versions, and Admin sections are visible only to administrator accounts.
User Settings
Personalization: Customize how the AI responds to you, such as your own system prompt and memory-style preferences.
Audio: Personal speech-to-text and text-to-speech options for voice input and read-aloud.
Data Controls: Import, export, archive, or delete your chats.
Usage: View usage information for your account.
Archived Chats: Browse and restore chats you archived from the sidebar.
Account: Manage your name, profile picture, password, and API keys.
About: Shows the installed Open WebUI version and update and license information.
Admin Settings
[System]
General: Global options such as new-user sign-ups, default user role, and the WebUI URL.
Authentication: Controls how users sign in, including login methods and SSO or LDAP options.
Interface: Server-wide UI defaults, such as banners, default prompt suggestions, and task model settings.
Connections: Where Open WebUI connects to model backends. This setup uses the Ollama API at http://192.168.2.230:11434, and you can also enable an OpenAI-compatible API.
Models: Manage available models, including downloading new Ollama models, setting visibility, and customizing model settings.
Sub-agents: Settings for delegating work to sub-agents (available in newer versions).
Integrations: Connect external tools and services that models can call.
Documents: Configure document handling for RAG, such as the embedding model and text chunking.
Audio: Choose the server-wide speech-to-text and text-to-speech engines.
Images: Configure an image generation backend.
Web Search: Pick a search engine so chats can pull in live web results.
Code Execution: Configure code execution and the code interpreter.
Pipelines: Manage Pipelines, a plugin framework for custom filters and workflows.
Analytics: Usage statistics across users and models.
Evaluations: Review model feedback and compare model performance.
Database: Database-level options such as exporting or backing up the data.
For a home lab, I recommend starting with Connections and Models, then trying Documents (RAG) and Web Search once the basics are stable.
SECURITY RECOMMENDATIONS
Avoid exposing port 11434 to untrusted networks. Ollama's API has no built-in authentication by default, so use firewall rules (for example ufw allowing only the Open WebUI host) or a secured reverse proxy when remote access is required.
CONCLUSION
You now have a private, self-hosted AI chat environment: Ollama serving Llama 3.2 on port 11434 and Open WebUI giving you a clean browser interface on port 3010. From here you can pull more models with ollama pull, connect additional APIs, or build RAG workflows with your own documents.
No comments:
Post a Comment