openwebui-oikb/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

[0.5.0] - 2026-09-16

Added

• Multiple sources in one Knowledge Base: combine sources such as a Confluence space and a local handbook without one sync deleting the other's files. Sources listed together in the same configuration are synced as a group, including scheduled runs and updates triggered by source changes. You can choose a destination folder for each source to keep matching filenames separate.
• Confluence Server and Data Center: sync spaces from self-hosted Confluence, including installations under an address such as /confluence. Sign in with a personal access token, and use Cloud and Data Center sources together with separate connection settings.
• Confluence page folders: optionally preserve parent and child pages as folders, so you can include or exclude a whole section, such as Engineering runbooks.
• SharePoint sites: sync document libraries from team sites and nested subsites, in addition to the main site.
• SharePoint government clouds: connect to Microsoft GCC High and Department of Defense environments.
• Multiple Zotero libraries: sync more than one personal or group library from one configuration, with separate access keys, collection choices, notes, annotations, and PDF download settings for each library.
• Separate source sign-ins: supply credentials for GitHub, GitLab, Bitbucket, SharePoint, and Confluence individually in the configuration or when running a sync or preview. This lets one setup access sources that require different accounts.

Fixed

• Google Drive subfolders: files inside nested folders are now included instead of being silently skipped.
• Large Outline collections: syncing continues past the first 100 documents instead of leaving the rest out.
• Zotero collection selection: selecting a named collection or subcollection now finds it correctly when its top-level collection has no parent.
• SharePoint filenames: files and folders containing characters such as # or % can now be found and downloaded.
• SharePoint certificate sign-in: valid certificates no longer fail with a "key was not found" error caused by how oikb identifies the certificate.
• GitHub and GitLab access: syncs now use the credentials supplied for each source instead of ignoring them. Configuration checks also use the supplied source credentials.
• Confluence space selection: spaces can be selected by their short names, such as ENG, without looking up a numeric space ID. Cloud addresses that already end in /wiki are accepted.
• Confluence page content: text inside macros and code blocks is retained, including code indentation and line breaks. Blank pages and pages containing only an automatic index are skipped with a warning instead of repeatedly failing to upload.
• Confluence matching titles: pages whose titles produce the same filename now receive distinct names, so one page does not replace another.
• Confluence large spaces: page lists continue correctly when Confluence limits the number returned at once. If the list cannot be completed, the sync stops before changing the Knowledge Base.
• Knowledge Base file listings: file listing and status commands now read the dedicated file list, avoiding failures when the Knowledge Base summary contains no file details. Connection checks read the reported total file count.
• Empty files: empty content is skipped with a warning instead of being sent for upload.
• Upload errors: failures now include the reason returned by Open WebUI, such as a text extraction failure.

Changed

• Shared Knowledge Base setup: list all sources for the same Knowledge Base in one configuration. Selecting any one of those sources syncs the whole group. Running a standalone sync for just one source does not protect files from the others. Preview changes before combining existing sources, since destination folders can move files.
• Contributor credits: added explicit credit for thiswillbeyourgithub's Zotero work and hexsm's BookStack export work already included in 0.4.0.

[0.4.0] - 2026-07-17

Added

• Nextcloud: sync folders from Nextcloud using app-password login. Empty or missing folders are handled cleanly instead of stopping the run.
• Zotero: sync Zotero PDF attachment text into Knowledge Bases. Supports whole libraries, nested collections, unfiled items, optional notes, optional annotations, collection excludes, content checksums, and WebDAV fallback for stored PDFs.
• BookStack: choose exactly what to sync from BookStack. You can sync selected pages, selected books, selected shelves, chapters, whole-book exports, Markdown, plain text, HTML, or PDF, with flat or folder-style paths.
• GitLab wikis: sync project wiki pages with gitlab:group/project?wiki=true, including subgroup projects and older GitLab installs that do not return page content in the list response.
• Automatic publishing: pushing a version bump and changelog entry to the release branch now creates the GitHub release, publishes to PyPI, attaches release files, and publishes versioned Docker images.

Fixed

• SharePoint: file downloads now follow Microsoft Graph redirects, so SharePoint files no longer fail with 302 Found.
• Google Drive: Shared Drives are now included when listing, finding, exporting, and downloading files.
• Jira: large issue lists now keep paging until all matching issues are found.
• GitLab: subgroup project paths are resolved more reliably, including sources that also include a folder path.
• Filesystem: local sync now blocks paths that escape the configured source folder.
• Docker Compose: the sample service now starts in /app, so the mounted .oikb.yaml is found where the container expects it.
• Docker health checks: the bundled compose health check now works in the Alpine-based image.
• Sync warnings: source files that are listed but unavailable can now show as warnings instead of failing the entire sync.
• CLI output: multi-source syncs now print upload errors and warnings instead of only exiting with a failure code.
• Daemon status: scheduled syncs now report partial success, warnings, and upload errors in health, history, dry-run, and notification output.
• Daemon shutdown: Docker shutdown now asks running syncs to stop and waits briefly, instead of leaving background work loose.
• History storage: concurrent daemon syncs now use separate SQLite connections, avoiding cross-thread database failures.
• Release version output: oikb --version now matches the package version.

Changed

• Docker images now use a smaller Alpine-based runtime and run as a non-root user.
• The connector count is now 46 in the README and guide.
• The guide now includes BookStack, Nextcloud, and Zotero examples.

[0.3.6] - 2026-05-28

Added

• SharePoint: certificate-based authentication via SHAREPOINT_CERTIFICATE_PATH env var — more secure alternative to client secret auth. Supports encrypted PEM keys with SHAREPOINT_CERTIFICATE_PASSWORD. Requires pip install oikb[sharepoint-cert].

[0.3.5] - 2025-05-21

Added

• Status dashboard at GET / — monospace status page rendered client-side from /health. Shows source name, status dot, last sync time, duration, file counts, next sync. Polls every 10s. Zero dependencies.

[0.3.4] - 2025-05-21

Added

• oikb validate --deep — verifies Open WebUI connectivity, API key validity, and that each KB ID exists. Catches config errors before deployment.

Fixed

• docker-compose.yaml updated from stale watch mode to production daemon mode with healthcheck, LOG_FORMAT=json, and OIKB_API_KEY support.
• validate now applies defaults: and env var interpolation to entries before checking (was only doing raw yaml).

[0.3.3] - 2025-05-21

Added

• oikb init — interactive wizard that generates .oikb.yaml. Prompts for source, KB ID, name, and interval. Outputs next-step commands. Reduces onboarding from "read docs and write YAML" to "answer 4 questions."

[0.3.2] - 2025-05-21

Added

• Global defaults: key in .oikb.yaml — set interval, concurrency, filter, notify, or any other key once and have it apply to all entries. Per-entry values override defaults. Deep merges nested dicts (filter, notify).

[0.3.1] - 2025-05-21

Added

• Per-KB sync locking — prevents overlapping syncs to the same Knowledge Base. If a webhook fires while a scheduled sync is running, the duplicate is skipped with a log message.
• Dry-run via API — POST /sync/{id}?dry_run=true previews changes without uploading, returns added/modified/deleted counts.
• Environment variable interpolation in .oikb.yaml — ${VAR} and ${VAR:-default} syntax in all string values. Enables GitOps workflows where secrets come from the runtime.

[0.3.0] - 2025-05-21

Added

• Webhook failure notifications — notify key in .oikb.yaml entries POSTs a JSON payload to any HTTP endpoint on sync error. Includes a text field for native Slack incoming webhook compatibility. Supports on: error (default) or on: always.

[0.2.9] - 2025-05-21

Added

• filter.max-size — skip files exceeding a size limit (e.g. 50mb, 1gb). Configurable per-entry in .oikb.yaml or via --max-file-size CLI flag. Oversized files are warned and excluded before diffing.

[0.2.8] - 2025-05-21

Added

• Structured JSON logging for the daemon — oikb daemon --log-format json or LOG_FORMAT=json env var. Outputs one JSON object per line with ts, level, logger, msg, and optional source/kb_id/duration_ms fields. Compatible with Datadog, Splunk, ELK, CloudWatch, and Loki.

[0.2.7] - 2025-05-21

Added

• Cron expression support for daemon scheduling — use interval: "0 6 * * 1-5" alongside simple intervals (30m, 1h). Auto-detected, no config flag needed.
• croniter>=2.0 added as a core dependency.

[0.2.6] - 2025-05-21

Added

• Prometheus metrics endpoint (GET /metrics) on the daemon — exports oikb_sync_total, oikb_sync_duration_seconds, oikb_files_uploaded_total, oikb_files_deleted_total, oikb_sync_errors_total, and oikb_info build metadata. All counters labeled by source.
• prometheus-client>=0.20 added as a core dependency.

[0.2.5] - 2025-05-21

Added

• Rich progress bars for sync operations — spinner during scan/diff, progress bar during uploads with file count and elapsed time.
• rich>=13.0 added as a core dependency.

[0.2.4] - 2025-05-21

Added

• oikb validate command — validate .oikb.yaml without running syncs.
• --concurrency flag for oikb sync — opt-in parallel uploads (also configurable per-entry in .oikb.yaml).
• Upload retry with exponential backoff — transient 5xx errors retry up to 3 times.

Fixed

• Daemon now applies filter.include/filter.exclude from .oikb.yaml (previously silently ignored).
• Connector HTTP clients are now properly closed after sync completes.
• GitHub connector caches default branch lookup — eliminates redundant API calls per file.

Changed

• Removed 30+ no-op optional dependencies from pyproject.toml ([github], [confluence], etc.) that just re-declared httpx (already a core dep). Only extras with real dependencies remain: s3, gcs, azure, dropbox, r2, gdrive, gmail, gsites, web, oracle.

[0.2.3] - 2025-05-21

Removed

• Breaking: routes key in .oikb.yaml. Use multiple entries with filter.include instead — each entry is one source → one KB.

[0.2.2] - 2025-05-21

Added

• name key for .oikb.yaml entries — use friendly names to target syncs via CLI (--name wiki) or API (POST /sync/wiki).

Changed

• Daemon sync endpoint changed from /sync/{source} to /sync/{identifier} — accepts name or kb-id (UUIDs).

[0.2.1] - 2025-05-20

Added

• API key authentication for daemon endpoints (OIKB_API_KEY env var, Docker secrets _FILE supported).
• Configurable fields, format, and query filters for Jira and ServiceNow connectors.

Changed

• .oikb.yaml top-level key renamed from sync: to sources: (backward compatible).
• /health endpoints remain public; /history and /sync require auth when key is set.

[0.2.0] - 2025-05-20

Added

• oikb daemon command: long-lived scheduler with FastAPI server, /health and /history endpoints.
• oikb history command: view sync history from local SQLite database.
• Webhook support: /webhooks/github, /webhooks/gitlab, /webhooks/slack, /webhooks/confluence for real-time sync triggers.
• --json output flag for history command.
• 13 new connectors: Document360, Slab, Outline, Google Sites, Egnyte, Oracle Cloud Storage, ProductBoard, XenForo, Zulip, Gong, Fireflies, DokuWiki, ServiceNow. Total: 44 connectors.
• Selective sync filters: filter.include / filter.exclude glob patterns to narrow sync scope.
• Daemon doubles as an OpenAPI tool server for Open WebUI (Settings → Connections → Tool Server).

Changed

• FastAPI and uvicorn are now core dependencies.

[0.1.3] - 2025-05-20

Changed

• Renamed .oikb.yaml config key from kb to kb-id to align with the CLI flag.

[0.1.2] - 2025-05-20

Changed

• Messaging connectors (Slack, Discord, Teams) now split messages by day for truly incremental sync. Past days are immutable so their checksums never change.

[0.1.1] - 2025-05-20

Added

• 28 new connectors bringing the total to 31, covering code repos, cloud storage, wikis, ticketing, messaging, CRM, and web:
  • Code Repos: GitLab (gitlab:owner/repo), Bitbucket (bitbucket:owner/repo)
  • Cloud Storage: Google Cloud Storage (gs://), Azure Blob (az://), Dropbox (dropbox:), Cloudflare R2 (r2://)
  • Wikis & Knowledge Bases: Confluence (confluence:), Notion (notion:), BookStack (bookstack:), Discourse (discourse:), GitBook (gitbook:), Guru (guru:)
  • Ticketing & Tasks: Jira (jira:), Linear (linear:), Zendesk (zendesk:), Freshdesk (freshdesk:), Asana (asana:), ClickUp (clickup:), Airtable (airtable:)
  • Messaging: Slack (slack:), Discord (discord:), Microsoft Teams (teams:), Gmail (gmail:)
  • Sales & CRM: Salesforce (salesforce:), HubSpot (hubspot:)
  • Other: Google Drive (gdrive:), SharePoint (sharepoint:), Web Crawler (web:)
• Context manager support for OikbClient
• .gitignore added to the project

Changed

• Renamed --kb to --kb-id across all commands
• Modernized README with 30+ connector showcase organized by category
• Added Open WebUI 0.9.6+ version requirement note
• Updated author to Tim Baek
• Moved import json to top-level in client.py
• Removed unused imports from sync.py, config.py
• Fixed Optional[str] type hints to use str | None syntax
• Removed redundant fnmatch check in ignore.py
• Empty source message now respects --quiet flag

[0.1.0] - 2025-05-20

Added

• Initial release of oikb, a CLI tool for syncing content to Open WebUI Knowledge Bases.
• Incremental sync (oikb sync) with SHA-256 diffing. Supports local directories, GitHub repos, and S3 buckets.
• Dry-run preview (oikb diff).
• Watch mode (oikb watch) with debounced filesystem monitoring via watchdog.
• List files (oikb ls), status (oikb status), reset (oikb reset) commands.
• Configuration (oikb config) via config file, env vars, or CLI flags.
• Declarative config (.oikb.yaml) for multi-source sync.
• Ignore patterns (.oikbignore) with gitignore-style file exclusion.
• GitHub connector (github:owner/repo) via Trees API. No local clone needed.
• S3 connector (s3://bucket/prefix) via boto3 using ETags as checksums.
• Docker image with multi-arch support (amd64 + arm64).
• GitHub Actions workflows for Docker builds (GHCR) and releases.