mirror of
https://github.com/open-webui/oikb.git
synced 2026-10-16 11:16:11 +00:00
Adapt PR #28 to the current connectors and sync lifecycle. Add v1 and Bearer authentication, context-path support, robust pagination, and stable names for colliding Confluence page titles. Preserve existing single-source paths, macro extraction, and empty-page warnings. Combine configured sources into one KB-wide manifest for CLI, scheduled, and webhook syncs. Keep per-source credentials, filters, and schedules; validate destination paths and close connectors on failed scans before allowing a diff. Document target-path and shared-KB configuration, and include server response details in upload errors. Validated with 39 passing tests, git diff --check, and uv build. Co-authored-by: tedk0n <4rfer0@gmail.com>
914 lines
24 KiB
Markdown
914 lines
24 KiB
Markdown
# oikb Guide
|
|
|
|
A complete guide to syncing content into Open WebUI Knowledge Bases.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
- [Installation](#installation)
|
|
- [Getting Started](#getting-started)
|
|
- [Your First Sync](#your-first-sync)
|
|
- [Watch Mode](#watch-mode)
|
|
- [Configuration File](#configuration-file)
|
|
- [Generating with oikb init](#generating-with-oikb-init)
|
|
- [Manual Setup](#manual-setup)
|
|
- [Global Defaults](#global-defaults)
|
|
- [Environment Variable Interpolation](#environment-variable-interpolation)
|
|
- [Sources](#sources)
|
|
- [Local Directories](#local-directories)
|
|
- [GitHub](#github)
|
|
- [GitLab / Bitbucket](#gitlab--bitbucket)
|
|
- [Confluence](#confluence)
|
|
- [BookStack](#bookstack)
|
|
- [Cloud Storage (S3 / GCS / Azure)](#cloud-storage-s3--gcs--azure)
|
|
- [SharePoint](#sharepoint)
|
|
- [Nextcloud](#nextcloud)
|
|
- [All Connectors](#all-connectors)
|
|
- [Filtering](#filtering)
|
|
- [Include / Exclude Globs](#include--exclude-globs)
|
|
- [Max File Size](#max-file-size)
|
|
- [Splitting Sources Across KBs](#splitting-sources-across-kbs)
|
|
- [Daemon Mode](#daemon-mode)
|
|
- [Starting the Daemon](#starting-the-daemon)
|
|
- [Scheduling](#scheduling)
|
|
- [API Endpoints](#api-endpoints)
|
|
- [Authentication](#authentication)
|
|
- [Webhooks](#webhooks)
|
|
- [Enterprise Features](#enterprise-features)
|
|
- [Prometheus Metrics](#prometheus-metrics)
|
|
- [Structured JSON Logging](#structured-json-logging)
|
|
- [Failure Notifications](#failure-notifications)
|
|
- [Sync Locking](#sync-locking)
|
|
- [Dry-Run via API](#dry-run-via-api)
|
|
- [Deployment](#deployment)
|
|
- [Docker](#docker)
|
|
- [Docker Compose](#docker-compose)
|
|
- [GitHub Actions](#github-actions)
|
|
- [Kubernetes](#kubernetes)
|
|
- [CLI Reference](#cli-reference)
|
|
- [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install oikb
|
|
```
|
|
|
|
For specific connectors with extra dependencies:
|
|
|
|
```bash
|
|
pip install oikb[gdrive] # Google Drive
|
|
pip install oikb[s3] # AWS S3
|
|
pip install oikb[confluence] # Confluence
|
|
pip install oikb[all] # Everything
|
|
```
|
|
|
|
Requires **Open WebUI 0.9.6+** and Python 3.10+.
|
|
|
|
---
|
|
|
|
## Getting Started
|
|
|
|
### Your First Sync
|
|
|
|
Set your Open WebUI credentials:
|
|
|
|
```bash
|
|
export OPEN_WEBUI_URL=http://localhost:3000
|
|
export OPEN_WEBUI_API_KEY=sk-your-api-key
|
|
```
|
|
|
|
Or save them permanently:
|
|
|
|
```bash
|
|
oikb config set url http://localhost:3000
|
|
oikb config set token sk-your-api-key
|
|
```
|
|
|
|
Find your Knowledge Base ID in Open WebUI → Knowledge → click a KB → copy the ID from the URL.
|
|
|
|
Sync a local directory:
|
|
|
|
```bash
|
|
oikb sync ./docs --kb-id your-kb-id
|
|
```
|
|
|
|
That's it. Only new and modified files are uploaded (SHA-256 diffing). Run it again and nothing happens — it's fully incremental.
|
|
|
|
### Preview Before Syncing
|
|
|
|
Always safe to preview first:
|
|
|
|
```bash
|
|
oikb sync ./docs --kb-id your-kb-id --dry-run
|
|
```
|
|
|
|
This shows exactly what would be added, modified, or deleted without touching the KB.
|
|
|
|
### Watch Mode
|
|
|
|
Auto-sync when files change:
|
|
|
|
```bash
|
|
oikb watch ./docs --kb-id your-kb-id
|
|
```
|
|
|
|
Uses filesystem events (not polling) so changes are picked up instantly.
|
|
|
|
---
|
|
|
|
## Configuration File
|
|
|
|
For multi-source setups or daemon mode, use a `.oikb.yaml` config file.
|
|
|
|
### Generating with oikb init
|
|
|
|
The fastest way to create one:
|
|
|
|
```bash
|
|
oikb init
|
|
```
|
|
|
|
This walks you through each source interactively and writes the file.
|
|
|
|
### Manual Setup
|
|
|
|
Create `.oikb.yaml` in your project root:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: docs
|
|
source: ./docs
|
|
kb-id: your-kb-id
|
|
|
|
- name: wiki
|
|
source: github:myorg/wiki
|
|
kb-id: another-kb-id
|
|
interval: 1h
|
|
```
|
|
|
|
Then sync all sources at once:
|
|
|
|
```bash
|
|
oikb sync
|
|
```
|
|
|
|
### Multiple sources in one Knowledge Base
|
|
|
|
Entries with the same `kb-id` are scanned together and sent as one manifest.
|
|
This prevents a sync of one source from deleting files belonging to another.
|
|
`oikb sync --name NAME`, daemon schedules, and webhooks all sync the entire
|
|
configured group. Each entry keeps its schedule; whenever one is due, the whole
|
|
group runs under the existing KB lock. Use the same `url` and `token` settings
|
|
for every entry in a group.
|
|
|
|
```yaml
|
|
sources:
|
|
- name: engineering
|
|
source: confluence:ENG
|
|
kb-id: shared-kb
|
|
target-path: ENG
|
|
- name: handbook
|
|
source: ./handbook
|
|
kb-id: shared-kb
|
|
target-path: handbook
|
|
```
|
|
|
|
`target-path` is an optional destination directory. Filters still match paths
|
|
inside each source, before this prefix is applied. In a group with multiple
|
|
sources, Confluence entries default to their space key as the destination
|
|
directory; other connectors keep their original paths. Explicit `target-path`
|
|
values keep paths stable when sources are added or removed. Duplicate destination
|
|
paths or a failed source scan abort the group before any KB changes.
|
|
|
|
Existing single-source paths remain unchanged. Combining previously separate
|
|
sources can relocate files; preview with `oikb sync --dry-run`. Use YAML mode
|
|
for a shared KB: a standalone `oikb sync SOURCE --kb-id ID` only knows about that
|
|
one source.
|
|
|
|
### Global Defaults
|
|
|
|
Avoid repeating the same config across entries:
|
|
|
|
```yaml
|
|
defaults:
|
|
interval: 1h
|
|
concurrency: 4
|
|
filter:
|
|
max-size: 50mb
|
|
notify:
|
|
url: https://hooks.slack.com/services/T.../B.../xxx
|
|
on: error
|
|
|
|
sources:
|
|
- name: docs
|
|
source: github:myorg/docs
|
|
kb-id: abc123
|
|
|
|
- name: handbook
|
|
source: confluence:ENG
|
|
kb-id: def456
|
|
interval: "0 6 * * 1-5" # overrides default
|
|
```
|
|
|
|
Per-entry values override defaults. Nested dicts (filter, notify) are deep-merged.
|
|
|
|
### Environment Variable Interpolation
|
|
|
|
All string values support `${VAR}` and `${VAR:-default}`:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: docs
|
|
source: github:${GITHUB_ORG}/docs
|
|
kb-id: ${KB_DOCS_ID}
|
|
token: ${GITHUB_TOKEN}
|
|
notify:
|
|
url: ${SLACK_WEBHOOK:-https://hooks.slack.com/fallback}
|
|
```
|
|
|
|
This enables GitOps workflows where `.oikb.yaml` is committed to the repo but secrets come from the runtime environment.
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
### Local Directories
|
|
|
|
```bash
|
|
oikb sync ./path/to/docs --kb-id your-kb-id
|
|
```
|
|
|
|
Scans recursively. Supports any file type.
|
|
|
|
### GitHub
|
|
|
|
```bash
|
|
oikb sync github:owner/repo --kb-id your-kb-id
|
|
```
|
|
|
|
Requires `GITHUB_TOKEN` for private repos. Syncs the default branch. To specify a branch or subdirectory:
|
|
|
|
```bash
|
|
oikb sync github:owner/repo --branch develop --path docs/
|
|
```
|
|
|
|
Or in `.oikb.yaml`:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: api-docs
|
|
source: github:myorg/api
|
|
kb-id: abc123
|
|
branch: main
|
|
path: docs/
|
|
```
|
|
|
|
### GitLab / Bitbucket
|
|
|
|
```bash
|
|
oikb sync gitlab:owner/repo --kb-id your-kb-id
|
|
oikb sync bitbucket:owner/repo --kb-id your-kb-id
|
|
```
|
|
|
|
Requires `GITLAB_TOKEN` or `BITBUCKET_TOKEN` respectively.
|
|
|
|
### Confluence
|
|
|
|
```bash
|
|
oikb sync confluence:SPACE_KEY --kb-id your-kb-id
|
|
|
|
# Preserve the Confluence page hierarchy in manifest paths.
|
|
oikb sync 'confluence:SPACE_KEY?structure=hierarchical' --kb-id your-kb-id
|
|
```
|
|
|
|
With hierarchical structure enabled, existing `filter.include` and
|
|
`filter.exclude` patterns can select page trees, for example
|
|
`Engineering/Runbooks*`.
|
|
|
|
Cloud uses REST API v2 by default. Set `CONFLUENCE_URL` to your instance URL,
|
|
`CONFLUENCE_USER` to your email, and `CONFLUENCE_TOKEN` to your API token.
|
|
|
|
For Server/Data Center, select v1 and omit the user to use a personal access token:
|
|
|
|
```bash
|
|
export CONFLUENCE_API_VERSION=v1
|
|
export CONFLUENCE_URL=https://wiki.example.com/confluence
|
|
export CONFLUENCE_TOKEN=your-personal-access-token
|
|
unset CONFLUENCE_USER
|
|
oikb sync confluence:ENG --kb-id your-kb-id
|
|
```
|
|
|
|
Include your installation's context path (such as `/confluence`) in the URL.
|
|
Both versions support flat and hierarchical paths. A user plus token selects
|
|
Basic authentication; a token alone selects Bearer authentication, as described
|
|
in [Atlassian's PAT documentation](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).
|
|
Per-source `auth` keys `base_url`, `user`, `token`, and `api_version` override
|
|
environment settings, so Cloud and Data Center can coexist in one config. Set
|
|
`user: ""` to override an inherited user when using a PAT.
|
|
|
|
Page titles that map to the same filename receive page-ID suffixes. Macro text
|
|
and code blocks retain the existing extraction behavior; blank or index-only
|
|
pages are skipped with a warning.
|
|
|
|
### BookStack
|
|
|
|
```bash
|
|
# Sync all pages from all BookStack books as Markdown files.
|
|
oikb sync bookstack: --kb-id your-kb-id
|
|
|
|
# Sync one specific BookStack page.
|
|
oikb sync 'bookstack:pages?include_ids=12' --kb-id your-kb-id
|
|
|
|
# Sync one BookStack book as chapter files; pages outside chapters remain page files.
|
|
oikb sync 'bookstack:books?include_ids=12&output=chapters' --kb-id your-kb-id
|
|
|
|
# Sync one whole BookStack book as a PDF file.
|
|
oikb sync 'bookstack:books?include_ids=12&output=books&format=pdf' --kb-id your-kb-id
|
|
|
|
# Sync all pages from one shelf and preserve shelf/book/chapter paths.
|
|
oikb sync 'bookstack:shelves?include_ids=5&structure=hierarchical' --kb-id your-kb-id
|
|
```
|
|
|
|
Requires `BOOKSTACK_URL`, `BOOKSTACK_TOKEN_ID`, and `BOOKSTACK_TOKEN_SECRET`.
|
|
|
|
Defaults are `bookstack:books`, `output=pages`, `format=md`, and `structure=flat`.
|
|
|
|
| Parameter | Values | Description |
|
|
|---|---|---|
|
|
| `output` | `pages`, `chapters`, `books` | Exports individual pages, chapters where pages are grouped in chapters, or whole books. |
|
|
| `format` | `txt`, `md`, `html`, `pdf` | Exported file format. |
|
|
| `structure` | `flat`, `hierarchical` | Keeps files flat or mirrors shelf/book/chapter paths where available. |
|
|
| `include_ids` | comma-separated IDs | Limits the selected pages, books, or shelves. |
|
|
| `exclude_ids` | comma-separated IDs | Excludes selected pages, books, or shelves. |
|
|
|
|
Use `bookstack:pages`, `bookstack:books`, or `bookstack:shelves` to select what IDs refer to. `bookstack:` defaults to `bookstack:books`.
|
|
When `bookstack:pages` is used, `output` is always treated as `pages`.
|
|
With `output=chapters`, pages outside chapters remain individual page exports.
|
|
|
|
### Cloud Storage (S3 / GCS / Azure)
|
|
|
|
```bash
|
|
oikb sync s3://bucket/prefix --kb-id your-kb-id
|
|
oikb sync gcs://bucket/prefix --kb-id your-kb-id
|
|
oikb sync azure://container/prefix --kb-id your-kb-id
|
|
```
|
|
|
|
Uses standard cloud SDK credentials (AWS profiles, service accounts, etc.).
|
|
|
|
### SharePoint
|
|
|
|
Sync a SharePoint document library:
|
|
|
|
```bash
|
|
oikb sync sharepoint:mysite.sharepoint.com --kb-id your-kb-id
|
|
oikb sync sharepoint:mysite.sharepoint.com/Documents --kb-id your-kb-id
|
|
```
|
|
|
|
SharePoint supports two authentication methods:
|
|
|
|
#### Client Secret (simpler setup)
|
|
|
|
Set these environment variables:
|
|
|
|
```bash
|
|
export SHAREPOINT_TENANT_ID=your-tenant-id
|
|
export SHAREPOINT_CLIENT_ID=your-client-id
|
|
export SHAREPOINT_CLIENT_SECRET=your-client-secret
|
|
```
|
|
|
|
#### Certificate Auth (recommended for production)
|
|
|
|
Certificate-based auth is more secure and follows Microsoft best practices for production environments.
|
|
|
|
First install the required extras:
|
|
|
|
```bash
|
|
pip install oikb[sharepoint-cert]
|
|
```
|
|
|
|
Then set:
|
|
|
|
```bash
|
|
export SHAREPOINT_TENANT_ID=your-tenant-id
|
|
export SHAREPOINT_CLIENT_ID=your-client-id
|
|
export SHAREPOINT_CERTIFICATE_PATH=/path/to/cert.pem
|
|
# Optional — only needed if the PEM key is encrypted:
|
|
export SHAREPOINT_CERTIFICATE_PASSWORD=your-password
|
|
```
|
|
|
|
The PEM file must contain both the private key and the certificate. The connector extracts the thumbprint and signs a JWT assertion automatically.
|
|
|
|
> **Note:** `SHAREPOINT_CLIENT_SECRET` and `SHAREPOINT_CERTIFICATE_PATH` are mutually exclusive — set one or the other.
|
|
|
|
#### .oikb.yaml example
|
|
|
|
```yaml
|
|
sources:
|
|
- name: legal-docs
|
|
source: sharepoint:contoso.sharepoint.com/Legal
|
|
kb-id: abc123
|
|
interval: 1h
|
|
```
|
|
|
|
### Nextcloud
|
|
|
|
Sync a Nextcloud folder via WebDAV:
|
|
|
|
```bash
|
|
export NEXTCLOUD_URL=https://nextcloud.example.com
|
|
export NEXTCLOUD_USER=svc_docs
|
|
export NEXTCLOUD_PASSWORD=your-app-password
|
|
|
|
oikb sync "nextcloud:/Documents" --kb-id your-kb-id
|
|
oikb sync "nextcloud:/Team/Engineering Handbook" --kb-id your-kb-id
|
|
```
|
|
|
|
Use a Nextcloud app password when possible. The connector resolves the
|
|
authenticated DAV user ID through the Nextcloud OCS user endpoint, so it works
|
|
with both local users and LDAP-backed accounts.
|
|
|
|
#### .oikb.yaml example
|
|
|
|
```yaml
|
|
sources:
|
|
- name: team-docs
|
|
source: "nextcloud:/Documents"
|
|
kb-id: abc123
|
|
interval: 1h
|
|
```
|
|
|
|
### Zotero
|
|
|
|
Sync PDF attachment text from Zotero collections into KB directories. A bare
|
|
`zotero:` syncs all top-level collections and root library items into
|
|
`_unfiled`; use `%%` for nested collections.
|
|
|
|
```bash
|
|
pip install oikb[zotero]
|
|
|
|
export ZOTERO_LIBRARY_ID=123456
|
|
export ZOTERO_API_KEY=...
|
|
|
|
oikb sync "zotero:" --kb-id your-kb-id # syncs all top-level collections plus _unfiled
|
|
oikb sync "zotero:Research" --kb-id your-kb-id # syncs only the 'Research' collection
|
|
oikb sync "zotero:Research%%Machine Learning" --kb-id your-kb-id # syncs only the 'Machine Learning' subcollection
|
|
|
|
```
|
|
|
|
Optional settings:
|
|
|
|
| Variable | Description |
|
|
|---|---|
|
|
| `ZOTERO_LIBRARY_TYPE` | `user` (default) or `group` |
|
|
| `ZOTERO_INCLUDE_NOTES` | Append child notes when true |
|
|
| `ZOTERO_INCLUDE_ANNOTATIONS` | Append PDF annotation text/comments when true |
|
|
| `ZOTERO_CHECKSUM` | `version` (default) or `content` |
|
|
| `ZOTERO_EXCLUDE` | Comma-separated collection paths to skip |
|
|
| `ZOTERO_UNFILED_DIR` | Directory for root library items, default `_unfiled` |
|
|
| `ZOTERO_WEBDAV_URL` | WebDAV Zotero storage base; fetches `<attachment-key>.zip` on Zotero file 404 |
|
|
| `ZOTERO_WEBDAV_USER` / `ZOTERO_WEBDAV_PASSWORD` | WebDAV credentials |
|
|
|
|
### All Connectors
|
|
|
|
46 connectors available. See the full list:
|
|
|
|
| Category | Sources |
|
|
|---|---|
|
|
| **Git** | GitHub, GitLab, Bitbucket |
|
|
| **Cloud Storage** | S3, GCS, Azure Blob, Dropbox, R2, Google Drive, SharePoint, Nextcloud, Egnyte, Oracle Cloud |
|
|
| **Wikis & KBs** | Confluence, Notion, BookStack, Discourse, GitBook, Guru, Outline, Slab, Document360, DokuWiki, Google Sites |
|
|
| **Ticketing** | Jira, Linear, Zendesk, Freshdesk, Asana, ClickUp, Airtable, ServiceNow, ProductBoard |
|
|
| **Messaging** | Slack, Discord, Microsoft Teams, Gmail, Zulip |
|
|
| **Meetings** | Gong, Fireflies |
|
|
| **Forums** | XenForo |
|
|
| **Sales & CRM** | Salesforce, HubSpot |
|
|
| **Web** | Website / Sitemap crawler |
|
|
| **Research** | Zotero |
|
|
|
|
---
|
|
|
|
## Filtering
|
|
|
|
### Include / Exclude Globs
|
|
|
|
Control which files get synced:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: docs
|
|
source: github:owner/repo
|
|
kb-id: abc123
|
|
filter:
|
|
include: ["docs/**/*.md", "*.txt"]
|
|
exclude: ["drafts/**", "**/*.tmp"]
|
|
```
|
|
|
|
- `include` — only sync files matching these patterns
|
|
- `exclude` — skip files matching these patterns (applied after include)
|
|
|
|
### Max File Size
|
|
|
|
Skip files above a size limit:
|
|
|
|
```yaml
|
|
filter:
|
|
max-size: 50mb
|
|
```
|
|
|
|
Accepts `b`, `kb`, `mb`, `gb`. Files exceeding the limit are warned and skipped:
|
|
|
|
```
|
|
⚠ Skipping model.bin (2.1 GB) — exceeds max-size (50.0 MB)
|
|
```
|
|
|
|
Also available as CLI flag: `oikb sync ./docs --kb-id abc --max-file-size 50mb`
|
|
|
|
### Splitting Sources Across KBs
|
|
|
|
Route different parts of a repo to different Knowledge Bases:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: user-docs
|
|
source: github:owner/repo
|
|
kb-id: abc123
|
|
filter:
|
|
include: ["docs/**"]
|
|
|
|
- name: api-reference
|
|
source: github:owner/repo
|
|
kb-id: def456
|
|
filter:
|
|
include: ["api/**"]
|
|
```
|
|
|
|
---
|
|
|
|
## Daemon Mode
|
|
|
|
For production deployments, run oikb as a long-lived daemon with scheduled sync and an HTTP API.
|
|
|
|
### Starting the Daemon
|
|
|
|
```bash
|
|
oikb daemon --port 8080
|
|
```
|
|
|
|
Reads `.oikb.yaml` and syncs each source on its configured interval. Specify a config file:
|
|
|
|
```bash
|
|
oikb daemon --config /etc/oikb/config.yaml
|
|
```
|
|
|
|
### Scheduling
|
|
|
|
Simple intervals:
|
|
|
|
```yaml
|
|
interval: 30m # every 30 minutes
|
|
interval: 1h # every hour
|
|
interval: 6h # every 6 hours
|
|
```
|
|
|
|
Cron expressions:
|
|
|
|
```yaml
|
|
interval: "0 6 * * 1-5" # weekdays at 6am
|
|
interval: "0 */6 * * *" # every 6 hours
|
|
interval: "0 0 * * 0" # weekly on Sunday midnight
|
|
```
|
|
|
|
Both can be mixed in the same config. Auto-detected by field count.
|
|
|
|
### API Endpoints
|
|
|
|
| Endpoint | Description |
|
|
|---|---|
|
|
| `GET /health` | Sync status for all sources (k8s readiness probe) |
|
|
| `GET /health/ready` | Liveness probe |
|
|
| `GET /metrics` | Prometheus metrics |
|
|
| `GET /history` | Sync history (filterable by KB, errors) |
|
|
| `POST /sync/{name-or-kb-id}` | Trigger immediate sync |
|
|
| `POST /sync/{id}?dry_run=true` | Preview changes without uploading |
|
|
| `POST /webhooks/github` | GitHub push webhook |
|
|
| `POST /webhooks/gitlab` | GitLab push webhook |
|
|
| `POST /webhooks/slack` | Slack event webhook |
|
|
| `POST /webhooks/confluence` | Confluence update webhook |
|
|
|
|
### Authentication
|
|
|
|
Secure endpoints with an API key:
|
|
|
|
```bash
|
|
export OIKB_API_KEY=your-secret-key
|
|
oikb daemon
|
|
```
|
|
|
|
All API requests require `Authorization: Bearer your-secret-key`. Docker secrets supported via `OIKB_API_KEY_FILE`.
|
|
|
|
---
|
|
|
|
## Webhooks
|
|
|
|
Enable instant sync on push:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: docs
|
|
source: github:owner/repo
|
|
kb-id: abc123
|
|
webhook: true
|
|
github_secret: your-webhook-secret
|
|
```
|
|
|
|
Then add a webhook in GitHub → Settings → Webhooks pointing to `http://your-daemon:8080/webhooks/github`.
|
|
|
|
Supported: GitHub, GitLab, Slack, Confluence.
|
|
|
|
---
|
|
|
|
## Enterprise Features
|
|
|
|
### Prometheus Metrics
|
|
|
|
`GET /metrics` exports:
|
|
|
|
| Metric | Type | Description |
|
|
|---|---|---|
|
|
| `oikb_sync_total` | Counter | Syncs by source and status |
|
|
| `oikb_sync_duration_seconds` | Histogram | Sync duration |
|
|
| `oikb_files_uploaded_total` | Counter | Files added + modified |
|
|
| `oikb_files_deleted_total` | Counter | Files deleted |
|
|
| `oikb_sync_errors_total` | Counter | Failed syncs |
|
|
| `oikb_info` | Gauge | Build version |
|
|
|
|
Add to your Prometheus config:
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: oikb
|
|
static_configs:
|
|
- targets: ["oikb:8080"]
|
|
```
|
|
|
|
### Structured JSON Logging
|
|
|
|
Enable for log aggregators (Datadog, Splunk, ELK, CloudWatch):
|
|
|
|
```bash
|
|
oikb daemon --log-format json
|
|
# or
|
|
LOG_FORMAT=json oikb daemon
|
|
```
|
|
|
|
Output (one JSON object per line):
|
|
|
|
```json
|
|
{"ts":"2025-05-21T06:00:01Z","level":"INFO","logger":"oikb.daemon","msg":"Synced github:owner/repo → abc123: 3 added (1823ms)"}
|
|
```
|
|
|
|
### Failure Notifications
|
|
|
|
POST to a webhook on sync error:
|
|
|
|
```yaml
|
|
sources:
|
|
- name: docs
|
|
source: github:owner/repo
|
|
kb-id: abc123
|
|
notify:
|
|
url: https://hooks.slack.com/services/T.../B.../xxx
|
|
on: error # error (default) | always
|
|
```
|
|
|
|
The payload includes a `text` field for native Slack compatibility, plus structured fields (`source`, `status`, `error`, `duration_ms`).
|
|
|
|
### Sync Locking
|
|
|
|
Prevents overlapping syncs to the same KB. If a webhook fires while a scheduled sync is running, the duplicate is skipped:
|
|
|
|
```
|
|
INFO Skipping github:owner/repo — sync already running for abc123
|
|
```
|
|
|
|
Automatic. No configuration needed.
|
|
|
|
### Dry-Run via API
|
|
|
|
Preview changes without uploading:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/sync/docs?dry_run=true \
|
|
-H "Authorization: Bearer $OIKB_API_KEY"
|
|
```
|
|
|
|
Returns:
|
|
|
|
```json
|
|
{"dry_run": true, "result": {"added": 3, "modified": 1, "deleted": 0}}
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment
|
|
|
|
### Docker
|
|
|
|
```bash
|
|
docker run -d \
|
|
-e OPEN_WEBUI_URL=http://open-webui:8080 \
|
|
-e OPEN_WEBUI_API_KEY=sk-... \
|
|
-e OIKB_API_KEY=your-daemon-key \
|
|
-e LOG_FORMAT=json \
|
|
-v ./.oikb.yaml:/app/.oikb.yaml:ro \
|
|
-p 8080:8080 \
|
|
ghcr.io/open-webui/oikb:latest daemon
|
|
```
|
|
|
|
### Docker Compose
|
|
|
|
```yaml
|
|
services:
|
|
open-webui:
|
|
image: ghcr.io/open-webui/open-webui:main
|
|
ports:
|
|
- "3000:8080"
|
|
|
|
oikb:
|
|
image: ghcr.io/open-webui/oikb:latest
|
|
environment:
|
|
- OPEN_WEBUI_URL=http://open-webui:8080
|
|
- OPEN_WEBUI_API_KEY=${OPEN_WEBUI_API_KEY}
|
|
- OIKB_API_KEY=${OIKB_API_KEY}
|
|
- LOG_FORMAT=json
|
|
volumes:
|
|
- ./.oikb.yaml:/app/.oikb.yaml:ro
|
|
command: daemon
|
|
ports:
|
|
- "8080:8080"
|
|
depends_on:
|
|
- open-webui
|
|
restart: unless-stopped
|
|
healthcheck:
|
|
test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8080/health/ready"]
|
|
interval: 30s
|
|
timeout: 5s
|
|
retries: 3
|
|
start_period: 15s
|
|
```
|
|
|
|
### GitHub Actions
|
|
|
|
```yaml
|
|
- name: Sync docs to Open WebUI
|
|
uses: docker://ghcr.io/open-webui/oikb:latest
|
|
with:
|
|
args: sync /github/workspace/docs --kb-id ${{ secrets.KB_ID }}
|
|
env:
|
|
OPEN_WEBUI_URL: ${{ secrets.OPEN_WEBUI_URL }}
|
|
OPEN_WEBUI_API_KEY: ${{ secrets.OPEN_WEBUI_API_KEY }}
|
|
```
|
|
|
|
### Kubernetes
|
|
|
|
Minimal deployment:
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: oikb
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: oikb
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: oikb
|
|
spec:
|
|
containers:
|
|
- name: oikb
|
|
image: ghcr.io/open-webui/oikb:latest
|
|
args: ["daemon"]
|
|
ports:
|
|
- containerPort: 8080
|
|
env:
|
|
- name: OPEN_WEBUI_URL
|
|
value: http://open-webui:8080
|
|
- name: OPEN_WEBUI_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: oikb-secrets
|
|
key: open-webui-api-key
|
|
- name: OIKB_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: oikb-secrets
|
|
key: oikb-api-key
|
|
- name: LOG_FORMAT
|
|
value: json
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /app/.oikb.yaml
|
|
subPath: .oikb.yaml
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /health/ready
|
|
port: 8080
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 30
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: 8080
|
|
initialDelaySeconds: 10
|
|
periodSeconds: 60
|
|
volumes:
|
|
- name: config
|
|
configMap:
|
|
name: oikb-config
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: oikb
|
|
spec:
|
|
selector:
|
|
app: oikb
|
|
ports:
|
|
- port: 8080
|
|
```
|
|
|
|
---
|
|
|
|
## CLI Reference
|
|
|
|
```
|
|
oikb init Generate .oikb.yaml interactively
|
|
oikb sync [SOURCE] Incremental sync
|
|
oikb sync --dry-run Preview without uploading
|
|
oikb sync --max-file-size 50mb Skip large files
|
|
oikb sync --concurrency 4 Parallel uploads
|
|
oikb sync --scan-secrets Block files with credentials
|
|
oikb watch <dir> --kb-id ID Auto-sync on file change
|
|
oikb daemon Start scheduled daemon
|
|
oikb daemon --log-format json JSON logging
|
|
oikb daemon --config /path/to/yaml Custom config path
|
|
oikb diff <source> --kb-id ID Preview changes
|
|
oikb validate Check .oikb.yaml syntax
|
|
oikb validate --deep Verify API, auth, and KB existence
|
|
oikb ls --kb-id ID List files in a KB
|
|
oikb status --kb-id ID Show KB info
|
|
oikb history View sync history
|
|
oikb history --json JSON output
|
|
oikb history --errors Failed syncs only
|
|
oikb reset --kb-id ID Delete all files in a KB
|
|
oikb config set url <url> Save Open WebUI URL
|
|
oikb config set token <token> Save API key
|
|
oikb config show Show saved config
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### "Connection refused" or "401 Unauthorized"
|
|
|
|
Check your credentials:
|
|
|
|
```bash
|
|
oikb config show
|
|
# or verify env vars:
|
|
echo $OPEN_WEBUI_URL
|
|
echo $OPEN_WEBUI_API_KEY
|
|
```
|
|
|
|
### "No sync entries found"
|
|
|
|
You're running `oikb sync` without arguments and there's no `.oikb.yaml` in the current directory. Either:
|
|
- Specify the source: `oikb sync ./docs --kb-id your-kb-id`
|
|
- Create a config: `oikb init`
|
|
|
|
### Large syncs are slow
|
|
|
|
- Enable concurrent uploads: `--concurrency 4` or `concurrency: 4` in yaml
|
|
- Set `filter.max-size: 50mb` to skip large binaries
|
|
- Use `filter.exclude` to skip unnecessary files
|
|
|
|
### Daemon won't start
|
|
|
|
```bash
|
|
oikb validate # Check config syntax
|
|
oikb validate --deep # Verify API + KB connectivity
|
|
```
|
|
|
|
### How do I find my KB ID?
|
|
|
|
Open WebUI → Knowledge → click a KB → the ID is in the URL:
|
|
`http://localhost:3000/knowledge/8f3a2b1c-1234-5678-9abc-def012345678`
|