Using Local Models with Rossoctl

Rossoctl supports any OpenAI-compatible model backend. This guide covers using Ollama to run LLM models locally, eliminating the need for an external API key.

Overview
How It Works
Kind (Local Development)
OpenShift
Tested Models
Troubleshooting

Overview

Rossoctl agents use three environment variables to configure their LLM backend:

Variable	Description	Example (Ollama)	Example (OpenAI)
`LLM_API_BASE`	API endpoint URL	`http://host.docker.internal:11434/v1`	`https://api.openai.com/v1`
`LLM_API_KEY`	API key	`dummy` (Ollama ignores this)	Your OpenAI API key
`LLM_MODEL`	Model identifier	`qwen2.5:3b`	`gpt-4o-mini-2024-07-18`

When deploying agents through the Rossoctl UI or TUI, you select an LLM preset (ollama or openai) that populates these values automatically. No code changes are needed to switch between backends.

How It Works

Rossoctl provides two ways to configure LLM environment variables when deploying agents:

UI (ui-v2): Import a .env file (e.g. .env.openai or .env.ollama) from GitHub, or manually add env vars in the deploy form.
TUI: Select the openai or ollama preset, which injects the appropriate env vars directly into the deployment spec.

Kind (Local Development)

On Kind clusters, Ollama runs on the host machine and agents access it through Docker networking.

Prerequisites

Install Ollama: https://ollama.com/download
Pull a model:
```
ollama pull qwen2.5:3b
```
Start Ollama (listening on all interfaces):
```
OLLAMA_HOST=0.0.0.0 ollama serve
```

Agent Configuration

The default ollama environment set points to http://host.docker.internal:11434/v1, which resolves to the host machine from inside Docker/Kind containers. No additional configuration is needed.

When importing an agent in the UI, select the ollama environment variable set.

Automated Setup

The Kind full-test script handles Ollama installation and model pulling automatically:

./.github/scripts/local-setup/kind-full-test.sh --skip-cluster-destroy

This runs the 50-install-ollama.sh and 60-pull-ollama-model.sh scripts, which install Ollama and pull qwen2.5:3b.

OpenShift

On OpenShift, Ollama cannot run on the host machine since agents run in a remote cluster. There are two approaches:

Option 1: Deploy Ollama as a Pod (Recommended)

Deploy Ollama as a Kubernetes Deployment within the cluster so agents can reach it over the cluster network.

Create the Ollama Deployment:

kubectl apply -n rossoctl-system -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  labels:
    app: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        resources:
          requests:
            cpu: "2"
            memory: "8Gi"
          limits:
            cpu: "4"
            memory: "16Gi"
        volumeMounts:
        - name: ollama-data
          mountPath: /root/.ollama
      volumes:
      - name: ollama-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  selector:
    app: ollama
  ports:
  - port: 11434
    targetPort: 11434
EOF

Pull a model inside the pod:

kubectl exec -n rossoctl-system deploy/ollama -- ollama pull qwen2.5:3b

Configure the agent's LLM_API_BASE to point to the in-cluster service. When deploying an agent via the UI or TUI, set LLM_API_BASE to:
```
http://ollama.rossoctl-system.svc.cluster.local:11434/v1
```

Option 2: Use an External Ollama Server

If you have Ollama running on an accessible machine (e.g., a GPU workstation), you can point agents to its external address:

Start Ollama on the external machine:
```
OLLAMA_HOST=0.0.0.0 ollama serve
```
Ensure the OpenShift cluster can reach the machine over the network.
Update the ollama environment set LLM_API_BASE to http://<external-ip>:11434/v1.

Resource Considerations for OpenShift

Model Size	Minimum RAM	Recommended CPU	Notes
3B params (e.g., `qwen2.5:3b`)	8 Gi	2 cores	Good for testing and demos
8B params (e.g., `granite3.3:8b`)	16 Gi	4 cores	Better quality, tested on Apple M3 with 64 GB RAM
70B+ params	64+ Gi	8+ cores	Requires GPU for reasonable performance

For production OpenShift deployments, consider:

GPU nodes: Use nvidia.com/gpu resource requests for significantly better inference speed
Persistent storage: Replace emptyDir with a PVC to avoid re-downloading models on pod restart
Node affinity: Schedule the Ollama pod on nodes with adequate memory/GPU resources

Customizing the Ollama Endpoint

The default ollama preset points to http://host.docker.internal:11434/v1, which works for Kind but not for OpenShift. When deploying agents on OpenShift, set LLM_API_BASE to the in-cluster Ollama service URL:

http://ollama.rossoctl-system.svc.cluster.local:11434/v1

You can set this via the UI (add it as an env var when deploying the agent) or the TUI (use --env LLM_API_BASE=http://ollama.rossoctl-system.svc.cluster.local:11434/v1).

Tested Models

The following models have been tested with Rossoctl agents:

Ollama Models

Model	Size	Tested With	Notes
`qwen2.5:3b`	3B	Kind (CI), Apple Silicon	Default model in CI pipeline
`llama3.2:3b-instruct-fp16`	3B	Kind, Apple Silicon	Default in `ollama` environment set
`granite3.3:8b`	8B	Kind, Apple M3 (64 GB)	Tested with Slack Research Agent
`ibm/granite4:latest`	varies	Kind	Tested with GitHub Issue Agent

OpenAI Models

Model	Notes
`gpt-4o-mini-2024-07-18`	Default in `openai` environment set
`gpt-4.1-nano`	Lightweight, tested with Slack Research Agent
`gpt-4.1-mini`	Tested with Slack Research Agent
`gpt-4.1`	Tested with Slack Research Agent
`gpt-4o`	Tested with Slack Research Agent

Using Other Models

Any model that exposes an OpenAI-compatible API endpoint can be used. This includes:

Set LLM_API_BASE to the /v1 endpoint of your model server and LLM_MODEL to the model name it serves.

Troubleshooting

Ollama not reachable from Kind pods

Symptom: Agent pods fail to connect to the LLM.

Cause: host.docker.internal may not resolve on Linux without Docker Desktop.

Fix: On Linux with plain Docker, start Ollama on all interfaces and use the host IP:

OLLAMA_HOST=0.0.0.0 ollama serve

Then check the host gateway IP:

docker network inspect kind | grep Gateway

Set LLM_API_BASE to use that gateway IP when deploying the agent.

Model too slow or out of memory

Symptom: Agent responses are very slow or the Ollama pod gets OOMKilled.

Fix: Use a smaller model (e.g., qwen2.5:3b instead of an 8B model) or increase the pod's memory limits.

Agent uses wrong LLM backend

Symptom: Agent tries to call OpenAI instead of Ollama (or vice versa).

Fix: When deploying an agent, verify you selected the correct LLM preset (ollama or openai) or imported the correct .env file. You can check the running pod's environment:

kubectl exec -n team1 <agent-pod> -- env | grep LLM_

Table of Contents​

Overview​

How It Works​

Kind (Local Development)​

Prerequisites​

Agent Configuration​

Automated Setup​

OpenShift​

Option 1: Deploy Ollama as a Pod (Recommended)​

Option 2: Use an External Ollama Server​

Resource Considerations for OpenShift​

Customizing the Ollama Endpoint​

Tested Models​

Ollama Models​

OpenAI Models​

Using Other Models​

Troubleshooting​

Ollama not reachable from Kind pods​

Model too slow or out of memory​

Agent uses wrong LLM backend​

Table of Contents