# oikb Guide A complete guide to syncing content into Open WebUI Knowledge Bases. --- ## Table of Contents - [Installation](#installation) - [Getting Started](#getting-started) - [Your First Sync](#your-first-sync) - [Watch Mode](#watch-mode) - [Configuration File](#configuration-file) - [Generating with oikb init](#generating-with-oikb-init) - [Manual Setup](#manual-setup) - [Global Defaults](#global-defaults) - [Environment Variable Interpolation](#environment-variable-interpolation) - [Sources](#sources) - [Local Directories](#local-directories) - [GitHub](#github) - [GitLab / Bitbucket](#gitlab--bitbucket) - [Confluence](#confluence) - [BookStack](#bookstack) - [Cloud Storage (S3 / GCS / Azure)](#cloud-storage-s3--gcs--azure) - [SharePoint](#sharepoint) - [Nextcloud](#nextcloud) - [All Connectors](#all-connectors) - [Filtering](#filtering) - [Include / Exclude Globs](#include--exclude-globs) - [Max File Size](#max-file-size) - [Splitting Sources Across KBs](#splitting-sources-across-kbs) - [Daemon Mode](#daemon-mode) - [Starting the Daemon](#starting-the-daemon) - [Scheduling](#scheduling) - [API Endpoints](#api-endpoints) - [Authentication](#authentication) - [Webhooks](#webhooks) - [Enterprise Features](#enterprise-features) - [Prometheus Metrics](#prometheus-metrics) - [Structured JSON Logging](#structured-json-logging) - [Failure Notifications](#failure-notifications) - [Sync Locking](#sync-locking) - [Dry-Run via API](#dry-run-via-api) - [Deployment](#deployment) - [Docker](#docker) - [Docker Compose](#docker-compose) - [GitHub Actions](#github-actions) - [Kubernetes](#kubernetes) - [CLI Reference](#cli-reference) - [Troubleshooting](#troubleshooting) --- ## Installation ```bash pip install oikb ``` For specific connectors with extra dependencies: ```bash pip install oikb[gdrive] # Google Drive pip install oikb[s3] # AWS S3 pip install oikb[confluence] # Confluence pip install oikb[all] # Everything ``` Requires **Open WebUI 0.9.6+** and Python 3.10+. --- ## Getting Started ### Your First Sync Set your Open WebUI credentials: ```bash export OPEN_WEBUI_URL=http://localhost:3000 export OPEN_WEBUI_API_KEY=sk-your-api-key ``` Or save them permanently: ```bash oikb config set url http://localhost:3000 oikb config set token sk-your-api-key ``` Find your Knowledge Base ID in Open WebUI → Knowledge → click a KB → copy the ID from the URL. Sync a local directory: ```bash oikb sync ./docs --kb-id your-kb-id ``` That's it. Only new and modified files are uploaded (SHA-256 diffing). Run it again and nothing happens — it's fully incremental. ### Preview Before Syncing Always safe to preview first: ```bash oikb sync ./docs --kb-id your-kb-id --dry-run ``` This shows exactly what would be added, modified, or deleted without touching the KB. ### Watch Mode Auto-sync when files change: ```bash oikb watch ./docs --kb-id your-kb-id ``` Uses filesystem events (not polling) so changes are picked up instantly. --- ## Configuration File For multi-source setups or daemon mode, use a `.oikb.yaml` config file. ### Generating with oikb init The fastest way to create one: ```bash oikb init ``` This walks you through each source interactively and writes the file. ### Manual Setup Create `.oikb.yaml` in your project root: ```yaml sources: - name: docs source: ./docs kb-id: your-kb-id - name: wiki source: github:myorg/wiki kb-id: another-kb-id interval: 1h ``` Then sync all sources at once: ```bash oikb sync ``` ### Multiple sources in one Knowledge Base Entries with the same `kb-id` are scanned together and sent as one manifest. This prevents a sync of one source from deleting files belonging to another. `oikb sync --name NAME`, daemon schedules, and webhooks all sync the entire configured group. Each entry keeps its schedule; whenever one is due, the whole group runs under the existing KB lock. Use the same `url` and `token` settings for every entry in a group. ```yaml sources: - name: engineering source: confluence:ENG kb-id: shared-kb target-path: ENG - name: handbook source: ./handbook kb-id: shared-kb target-path: handbook ``` `target-path` is an optional destination directory. Filters still match paths inside each source, before this prefix is applied. In a group with multiple sources, Confluence entries default to their space key as the destination directory; other connectors keep their original paths. Explicit `target-path` values keep paths stable when sources are added or removed. Duplicate destination paths or a failed source scan abort the group before any KB changes. Existing single-source paths remain unchanged. Combining previously separate sources can relocate files; preview with `oikb sync --dry-run`. Use YAML mode for a shared KB: a standalone `oikb sync SOURCE --kb-id ID` only knows about that one source. ### Global Defaults Avoid repeating the same config across entries: ```yaml defaults: interval: 1h concurrency: 4 filter: max-size: 50mb notify: url: https://hooks.slack.com/services/T.../B.../xxx on: error sources: - name: docs source: github:myorg/docs kb-id: abc123 - name: handbook source: confluence:ENG kb-id: def456 interval: "0 6 * * 1-5" # overrides default ``` Per-entry values override defaults. Nested dicts (filter, notify) are deep-merged. ### Environment Variable Interpolation All string values support `${VAR}` and `${VAR:-default}`: ```yaml sources: - name: docs source: github:${GITHUB_ORG}/docs kb-id: ${KB_DOCS_ID} token: ${GITHUB_TOKEN} notify: url: ${SLACK_WEBHOOK:-https://hooks.slack.com/fallback} ``` This enables GitOps workflows where `.oikb.yaml` is committed to the repo but secrets come from the runtime environment. --- ## Sources ### Local Directories ```bash oikb sync ./path/to/docs --kb-id your-kb-id ``` Scans recursively. Supports any file type. ### GitHub ```bash oikb sync github:owner/repo --kb-id your-kb-id ``` Requires `GITHUB_TOKEN` for private repos. Syncs the default branch. To specify a branch or subdirectory: ```bash oikb sync github:owner/repo --branch develop --path docs/ ``` Or in `.oikb.yaml`: ```yaml sources: - name: api-docs source: github:myorg/api kb-id: abc123 branch: main path: docs/ ``` ### GitLab / Bitbucket ```bash oikb sync gitlab:owner/repo --kb-id your-kb-id oikb sync bitbucket:owner/repo --kb-id your-kb-id ``` Requires `GITLAB_TOKEN` or `BITBUCKET_TOKEN` respectively. ### Confluence ```bash oikb sync confluence:SPACE_KEY --kb-id your-kb-id # Preserve the Confluence page hierarchy in manifest paths. oikb sync 'confluence:SPACE_KEY?structure=hierarchical' --kb-id your-kb-id ``` With hierarchical structure enabled, existing `filter.include` and `filter.exclude` patterns can select page trees, for example `Engineering/Runbooks*`. Cloud uses REST API v2 by default. Set `CONFLUENCE_URL` to your instance URL, `CONFLUENCE_USER` to your email, and `CONFLUENCE_TOKEN` to your API token. For Server/Data Center, select v1 and omit the user to use a personal access token: ```bash export CONFLUENCE_API_VERSION=v1 export CONFLUENCE_URL=https://wiki.example.com/confluence export CONFLUENCE_TOKEN=your-personal-access-token unset CONFLUENCE_USER oikb sync confluence:ENG --kb-id your-kb-id ``` Include your installation's context path (such as `/confluence`) in the URL. Both versions support flat and hierarchical paths. A user plus token selects Basic authentication; a token alone selects Bearer authentication, as described in [Atlassian's PAT documentation](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html). Per-source `auth` keys `base_url`, `user`, `token`, and `api_version` override environment settings, so Cloud and Data Center can coexist in one config. Set `user: ""` to override an inherited user when using a PAT. Page titles that map to the same filename receive page-ID suffixes. Macro text and code blocks retain the existing extraction behavior; blank or index-only pages are skipped with a warning. ### BookStack ```bash # Sync all pages from all BookStack books as Markdown files. oikb sync bookstack: --kb-id your-kb-id # Sync one specific BookStack page. oikb sync 'bookstack:pages?include_ids=12' --kb-id your-kb-id # Sync one BookStack book as chapter files; pages outside chapters remain page files. oikb sync 'bookstack:books?include_ids=12&output=chapters' --kb-id your-kb-id # Sync one whole BookStack book as a PDF file. oikb sync 'bookstack:books?include_ids=12&output=books&format=pdf' --kb-id your-kb-id # Sync all pages from one shelf and preserve shelf/book/chapter paths. oikb sync 'bookstack:shelves?include_ids=5&structure=hierarchical' --kb-id your-kb-id ``` Requires `BOOKSTACK_URL`, `BOOKSTACK_TOKEN_ID`, and `BOOKSTACK_TOKEN_SECRET`. Defaults are `bookstack:books`, `output=pages`, `format=md`, and `structure=flat`. | Parameter | Values | Description | |---|---|---| | `output` | `pages`, `chapters`, `books` | Exports individual pages, chapters where pages are grouped in chapters, or whole books. | | `format` | `txt`, `md`, `html`, `pdf` | Exported file format. | | `structure` | `flat`, `hierarchical` | Keeps files flat or mirrors shelf/book/chapter paths where available. | | `include_ids` | comma-separated IDs | Limits the selected pages, books, or shelves. | | `exclude_ids` | comma-separated IDs | Excludes selected pages, books, or shelves. | Use `bookstack:pages`, `bookstack:books`, or `bookstack:shelves` to select what IDs refer to. `bookstack:` defaults to `bookstack:books`. When `bookstack:pages` is used, `output` is always treated as `pages`. With `output=chapters`, pages outside chapters remain individual page exports. ### Cloud Storage (S3 / GCS / Azure) ```bash oikb sync s3://bucket/prefix --kb-id your-kb-id oikb sync gcs://bucket/prefix --kb-id your-kb-id oikb sync azure://container/prefix --kb-id your-kb-id ``` Uses standard cloud SDK credentials (AWS profiles, service accounts, etc.). ### SharePoint Sync a SharePoint document library: ```bash oikb sync sharepoint:mysite.sharepoint.com --kb-id your-kb-id oikb sync sharepoint:mysite.sharepoint.com/Documents --kb-id your-kb-id ``` SharePoint supports two authentication methods: #### Client Secret (simpler setup) Set these environment variables: ```bash export SHAREPOINT_TENANT_ID=your-tenant-id export SHAREPOINT_CLIENT_ID=your-client-id export SHAREPOINT_CLIENT_SECRET=your-client-secret ``` #### Certificate Auth (recommended for production) Certificate-based auth is more secure and follows Microsoft best practices for production environments. First install the required extras: ```bash pip install oikb[sharepoint-cert] ``` Then set: ```bash export SHAREPOINT_TENANT_ID=your-tenant-id export SHAREPOINT_CLIENT_ID=your-client-id export SHAREPOINT_CERTIFICATE_PATH=/path/to/cert.pem # Optional — only needed if the PEM key is encrypted: export SHAREPOINT_CERTIFICATE_PASSWORD=your-password ``` The PEM file must contain both the private key and the certificate. The connector extracts the thumbprint and signs a JWT assertion automatically. > **Note:** `SHAREPOINT_CLIENT_SECRET` and `SHAREPOINT_CERTIFICATE_PATH` are mutually exclusive — set one or the other. #### .oikb.yaml example ```yaml sources: - name: legal-docs source: sharepoint:contoso.sharepoint.com/Legal kb-id: abc123 interval: 1h ``` ### Nextcloud Sync a Nextcloud folder via WebDAV: ```bash export NEXTCLOUD_URL=https://nextcloud.example.com export NEXTCLOUD_USER=svc_docs export NEXTCLOUD_PASSWORD=your-app-password oikb sync "nextcloud:/Documents" --kb-id your-kb-id oikb sync "nextcloud:/Team/Engineering Handbook" --kb-id your-kb-id ``` Use a Nextcloud app password when possible. The connector resolves the authenticated DAV user ID through the Nextcloud OCS user endpoint, so it works with both local users and LDAP-backed accounts. #### .oikb.yaml example ```yaml sources: - name: team-docs source: "nextcloud:/Documents" kb-id: abc123 interval: 1h ``` ### Zotero Sync PDF attachment text from Zotero collections into KB directories. A bare `zotero:` syncs all top-level collections and root library items into `_unfiled`; use `%%` for nested collections. ```bash pip install oikb[zotero] export ZOTERO_LIBRARY_ID=123456 export ZOTERO_API_KEY=... oikb sync "zotero:" --kb-id your-kb-id # syncs all top-level collections plus _unfiled oikb sync "zotero:Research" --kb-id your-kb-id # syncs only the 'Research' collection oikb sync "zotero:Research%%Machine Learning" --kb-id your-kb-id # syncs only the 'Machine Learning' subcollection ``` Optional settings: | Variable | Description | |---|---| | `ZOTERO_LIBRARY_TYPE` | `user` (default) or `group` | | `ZOTERO_INCLUDE_NOTES` | Append child notes when true | | `ZOTERO_INCLUDE_ANNOTATIONS` | Append PDF annotation text/comments when true | | `ZOTERO_CHECKSUM` | `version` (default) or `content` | | `ZOTERO_EXCLUDE` | Comma-separated collection paths to skip | | `ZOTERO_UNFILED_DIR` | Directory for root library items, default `_unfiled` | | `ZOTERO_WEBDAV_URL` | WebDAV Zotero storage base; fetches `.zip` on Zotero file 404 | | `ZOTERO_WEBDAV_USER` / `ZOTERO_WEBDAV_PASSWORD` | WebDAV credentials | ### All Connectors 46 connectors available. See the full list: | Category | Sources | |---|---| | **Git** | GitHub, GitLab, Bitbucket | | **Cloud Storage** | S3, GCS, Azure Blob, Dropbox, R2, Google Drive, SharePoint, Nextcloud, Egnyte, Oracle Cloud | | **Wikis & KBs** | Confluence, Notion, BookStack, Discourse, GitBook, Guru, Outline, Slab, Document360, DokuWiki, Google Sites | | **Ticketing** | Jira, Linear, Zendesk, Freshdesk, Asana, ClickUp, Airtable, ServiceNow, ProductBoard | | **Messaging** | Slack, Discord, Microsoft Teams, Gmail, Zulip | | **Meetings** | Gong, Fireflies | | **Forums** | XenForo | | **Sales & CRM** | Salesforce, HubSpot | | **Web** | Website / Sitemap crawler | | **Research** | Zotero | --- ## Filtering ### Include / Exclude Globs Control which files get synced: ```yaml sources: - name: docs source: github:owner/repo kb-id: abc123 filter: include: ["docs/**/*.md", "*.txt"] exclude: ["drafts/**", "**/*.tmp"] ``` - `include` — only sync files matching these patterns - `exclude` — skip files matching these patterns (applied after include) ### Max File Size Skip files above a size limit: ```yaml filter: max-size: 50mb ``` Accepts `b`, `kb`, `mb`, `gb`. Files exceeding the limit are warned and skipped: ``` ⚠ Skipping model.bin (2.1 GB) — exceeds max-size (50.0 MB) ``` Also available as CLI flag: `oikb sync ./docs --kb-id abc --max-file-size 50mb` ### Splitting Sources Across KBs Route different parts of a repo to different Knowledge Bases: ```yaml sources: - name: user-docs source: github:owner/repo kb-id: abc123 filter: include: ["docs/**"] - name: api-reference source: github:owner/repo kb-id: def456 filter: include: ["api/**"] ``` --- ## Daemon Mode For production deployments, run oikb as a long-lived daemon with scheduled sync and an HTTP API. ### Starting the Daemon ```bash oikb daemon --port 8080 ``` Reads `.oikb.yaml` and syncs each source on its configured interval. Specify a config file: ```bash oikb daemon --config /etc/oikb/config.yaml ``` ### Scheduling Simple intervals: ```yaml interval: 30m # every 30 minutes interval: 1h # every hour interval: 6h # every 6 hours ``` Cron expressions: ```yaml interval: "0 6 * * 1-5" # weekdays at 6am interval: "0 */6 * * *" # every 6 hours interval: "0 0 * * 0" # weekly on Sunday midnight ``` Both can be mixed in the same config. Auto-detected by field count. ### API Endpoints | Endpoint | Description | |---|---| | `GET /health` | Sync status for all sources (k8s readiness probe) | | `GET /health/ready` | Liveness probe | | `GET /metrics` | Prometheus metrics | | `GET /history` | Sync history (filterable by KB, errors) | | `POST /sync/{name-or-kb-id}` | Trigger immediate sync | | `POST /sync/{id}?dry_run=true` | Preview changes without uploading | | `POST /webhooks/github` | GitHub push webhook | | `POST /webhooks/gitlab` | GitLab push webhook | | `POST /webhooks/slack` | Slack event webhook | | `POST /webhooks/confluence` | Confluence update webhook | ### Authentication Secure endpoints with an API key: ```bash export OIKB_API_KEY=your-secret-key oikb daemon ``` All API requests require `Authorization: Bearer your-secret-key`. Docker secrets supported via `OIKB_API_KEY_FILE`. --- ## Webhooks Enable instant sync on push: ```yaml sources: - name: docs source: github:owner/repo kb-id: abc123 webhook: true github_secret: your-webhook-secret ``` Then add a webhook in GitHub → Settings → Webhooks pointing to `http://your-daemon:8080/webhooks/github`. Supported: GitHub, GitLab, Slack, Confluence. --- ## Enterprise Features ### Prometheus Metrics `GET /metrics` exports: | Metric | Type | Description | |---|---|---| | `oikb_sync_total` | Counter | Syncs by source and status | | `oikb_sync_duration_seconds` | Histogram | Sync duration | | `oikb_files_uploaded_total` | Counter | Files added + modified | | `oikb_files_deleted_total` | Counter | Files deleted | | `oikb_sync_errors_total` | Counter | Failed syncs | | `oikb_info` | Gauge | Build version | Add to your Prometheus config: ```yaml scrape_configs: - job_name: oikb static_configs: - targets: ["oikb:8080"] ``` ### Structured JSON Logging Enable for log aggregators (Datadog, Splunk, ELK, CloudWatch): ```bash oikb daemon --log-format json # or LOG_FORMAT=json oikb daemon ``` Output (one JSON object per line): ```json {"ts":"2025-05-21T06:00:01Z","level":"INFO","logger":"oikb.daemon","msg":"Synced github:owner/repo → abc123: 3 added (1823ms)"} ``` ### Failure Notifications POST to a webhook on sync error: ```yaml sources: - name: docs source: github:owner/repo kb-id: abc123 notify: url: https://hooks.slack.com/services/T.../B.../xxx on: error # error (default) | always ``` The payload includes a `text` field for native Slack compatibility, plus structured fields (`source`, `status`, `error`, `duration_ms`). ### Sync Locking Prevents overlapping syncs to the same KB. If a webhook fires while a scheduled sync is running, the duplicate is skipped: ``` INFO Skipping github:owner/repo — sync already running for abc123 ``` Automatic. No configuration needed. ### Dry-Run via API Preview changes without uploading: ```bash curl -X POST http://localhost:8080/sync/docs?dry_run=true \ -H "Authorization: Bearer $OIKB_API_KEY" ``` Returns: ```json {"dry_run": true, "result": {"added": 3, "modified": 1, "deleted": 0}} ``` --- ## Deployment ### Docker ```bash docker run -d \ -e OPEN_WEBUI_URL=http://open-webui:8080 \ -e OPEN_WEBUI_API_KEY=sk-... \ -e OIKB_API_KEY=your-daemon-key \ -e LOG_FORMAT=json \ -v ./.oikb.yaml:/app/.oikb.yaml:ro \ -p 8080:8080 \ ghcr.io/open-webui/oikb:latest daemon ``` ### Docker Compose ```yaml services: open-webui: image: ghcr.io/open-webui/open-webui:main ports: - "3000:8080" oikb: image: ghcr.io/open-webui/oikb:latest environment: - OPEN_WEBUI_URL=http://open-webui:8080 - OPEN_WEBUI_API_KEY=${OPEN_WEBUI_API_KEY} - OIKB_API_KEY=${OIKB_API_KEY} - LOG_FORMAT=json volumes: - ./.oikb.yaml:/app/.oikb.yaml:ro command: daemon ports: - "8080:8080" depends_on: - open-webui restart: unless-stopped healthcheck: test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8080/health/ready"] interval: 30s timeout: 5s retries: 3 start_period: 15s ``` ### GitHub Actions ```yaml - name: Sync docs to Open WebUI uses: docker://ghcr.io/open-webui/oikb:latest with: args: sync /github/workspace/docs --kb-id ${{ secrets.KB_ID }} env: OPEN_WEBUI_URL: ${{ secrets.OPEN_WEBUI_URL }} OPEN_WEBUI_API_KEY: ${{ secrets.OPEN_WEBUI_API_KEY }} ``` ### Kubernetes Minimal deployment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: oikb spec: replicas: 1 selector: matchLabels: app: oikb template: metadata: labels: app: oikb spec: containers: - name: oikb image: ghcr.io/open-webui/oikb:latest args: ["daemon"] ports: - containerPort: 8080 env: - name: OPEN_WEBUI_URL value: http://open-webui:8080 - name: OPEN_WEBUI_API_KEY valueFrom: secretKeyRef: name: oikb-secrets key: open-webui-api-key - name: OIKB_API_KEY valueFrom: secretKeyRef: name: oikb-secrets key: oikb-api-key - name: LOG_FORMAT value: json volumeMounts: - name: config mountPath: /app/.oikb.yaml subPath: .oikb.yaml livenessProbe: httpGet: path: /health/ready port: 8080 initialDelaySeconds: 5 periodSeconds: 30 readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 10 periodSeconds: 60 volumes: - name: config configMap: name: oikb-config --- apiVersion: v1 kind: Service metadata: name: oikb spec: selector: app: oikb ports: - port: 8080 ``` --- ## CLI Reference ``` oikb init Generate .oikb.yaml interactively oikb sync [SOURCE] Incremental sync oikb sync --dry-run Preview without uploading oikb sync --max-file-size 50mb Skip large files oikb sync --concurrency 4 Parallel uploads oikb sync --scan-secrets Block files with credentials oikb watch --kb-id ID Auto-sync on file change oikb daemon Start scheduled daemon oikb daemon --log-format json JSON logging oikb daemon --config /path/to/yaml Custom config path oikb diff --kb-id ID Preview changes oikb validate Check .oikb.yaml syntax oikb validate --deep Verify API, auth, and KB existence oikb ls --kb-id ID List files in a KB oikb status --kb-id ID Show KB info oikb history View sync history oikb history --json JSON output oikb history --errors Failed syncs only oikb reset --kb-id ID Delete all files in a KB oikb config set url Save Open WebUI URL oikb config set token Save API key oikb config show Show saved config ``` --- ## Troubleshooting ### "Connection refused" or "401 Unauthorized" Check your credentials: ```bash oikb config show # or verify env vars: echo $OPEN_WEBUI_URL echo $OPEN_WEBUI_API_KEY ``` ### "No sync entries found" You're running `oikb sync` without arguments and there's no `.oikb.yaml` in the current directory. Either: - Specify the source: `oikb sync ./docs --kb-id your-kb-id` - Create a config: `oikb init` ### Large syncs are slow - Enable concurrent uploads: `--concurrency 4` or `concurrency: 4` in yaml - Set `filter.max-size: 50mb` to skip large binaries - Use `filter.exclude` to skip unnecessary files ### Daemon won't start ```bash oikb validate # Check config syntax oikb validate --deep # Verify API + KB connectivity ``` ### How do I find my KB ID? Open WebUI → Knowledge → click a KB → the ID is in the URL: `http://localhost:3000/knowledge/8f3a2b1c-1234-5678-9abc-def012345678`