The merge jobs needed the whole build matrix, so the alpine/arm64 build
failing on pymupdf blocked the default image too and no 0.5.0 tag was
published. Each merge job now runs unless the workflow was cancelled and
publishes only when both of its own arches built.
Refs #126
The published image runs Python 3.11, whose mimetypes table has no .md
entry, and the slim base ships no /etc/mime.types. httpx then labels
every Markdown upload application/octet-stream. Open WebUI only reads a
file as plain text when the type is text/* or the extension is a known
source extension, so with a content extraction engine configured
(Docling, Tika) each .md went through that engine and came back
re-rendered. Docling in particular breaks GFM tables that contain
inline code.
Fall back to text/markdown for .md and .markdown when mimetypes has no
answer. Registering the type globally with mimetypes.add_type would
also work but changes process-wide state for connectors that embed
oikb.
Adapt PR #28 to the current connectors and sync lifecycle. Add v1 and
Bearer authentication, context-path support, robust pagination, and
stable names for colliding Confluence page titles. Preserve existing
single-source paths, macro extraction, and empty-page warnings.
Combine configured sources into one KB-wide manifest for CLI, scheduled,
and webhook syncs. Keep per-source credentials, filters, and schedules;
validate destination paths and close connectors on failed scans before
allowing a diff. Document target-path and shared-KB configuration, and
include server response details in upload errors.
Validated with 39 passing tests, git diff --check, and uv build.
Co-authored-by: tedk0n <4rfer0@gmail.com>
`validate --deep`, `status`, and `ls` derived the KB file list from
`GET /knowledge/{id}`, whose `files` field is a server-hydrated
convenience that some Open WebUI versions return as null. Reading
`len(kb.get("files", []))` then crashed with "object of type 'NoneType'
has no len()" because .get()'s default only applies to a missing key,
not an explicit null. `ls` silently printed "(empty)" for the same
reason. Sync was unaffected as it uses the server-side /sync/diff
endpoint.
Point the file listing at the real `GET /knowledge/{id}/files`
endpoint (returns {items, total}): fix list_kb_files, add
count_kb_files for the authoritative total, and use them in the
affected commands.
Fixes#89
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(zotero): support multiple libraries via per-source library ID
Embed an optional numeric library ID in the source string to allow
multiple Zotero libraries to be synced from a single daemon instance:
source: "zotero:123456:" # all collections in library 123456
source: "zotero:123456:Research%%ML" # specific collection
When a library ID is present the connector resolves credentials from
per-library env vars before falling back to the global defaults:
ZOTERO_API_KEY_<library_id> → ZOTERO_API_KEY
ZOTERO_LIBRARY_TYPE_<library_id> → ZOTERO_LIBRARY_TYPE
Existing source strings ("zotero:" or "zotero:Collection%%Sub") are
fully backward-compatible — collection names are not purely numeric so
there is no ambiguity when parsing.
* refactor(zotero): per-library env var resolution via _zotero_env helper
Introduce _zotero_env(key, library_id, default) which checks
ZOTERO_<KEY>_<library_id> before falling back to ZOTERO_<KEY>. Apply
it uniformly to every configurable setting so that each source entry
in .oikb.yaml can be tuned independently when multiple libraries are
in use:
ZOTERO_CHECKSUM_<id> overrides ZOTERO_CHECKSUM
ZOTERO_INCLUDE_NOTES_<id> ZOTERO_INCLUDE_NOTES
ZOTERO_INCLUDE_ANNOTATIONS_<id> ZOTERO_INCLUDE_ANNOTATIONS
ZOTERO_LIBRARY_TYPE_<id> ZOTERO_LIBRARY_TYPE
ZOTERO_EXCLUDE_<id> ZOTERO_EXCLUDE
ZOTERO_UNFILED_DIR_<id> ZOTERO_UNFILED_DIR
ZOTERO_WEBDAV_{URL,USER,PASSWORD}_<id> ZOTERO_WEBDAV_*
Single-library setups are unaffected — the helper falls through to the
global vars unchanged.
* fix(sync): skip empty file content before upload
Files with no extractable text (e.g. Zotero items with no fulltext and
no downloadable PDF) were uploaded as zero-byte payloads, causing
Open WebUI to reject them with EMPTY_CONTENT errors on every sync cycle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Pierre-Louis Suckrow <pierreroutine@mac.speedport.ip>
Co-authored-by: Pierre-Louis Suckrow <pierreroutine@MacBook-Air-von-Pierre-Louis.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add support for non-root sites and subsites
* Refactor SharePointConnector initialization
* Add site_path parameter to SharePoint connector
* Clean up comments in parse_sharepoint_source function
Removed redundant comments and examples from the parse_sharepoint_source function.
* Enhance SharePointConnector for multi-cloud support
Updated SharePoint connector to support multiple cloud environments by adding cloud-specific endpoints. Adjusted token retrieval methods to use the appropriate graph base URL based on the selected cloud.
* fix: reorder default param
* fix: pass source token to gitlab and github connectors
* Add auth parameter to _resolve_connector function
Updated the _resolve_connector function to accept an optional auth parameter, allowing for source-specific credentials to be passed. Modified calls to _resolve_connector throughout the code to include the auth argument where applicable.
* Add auth parameter to connector resolution
`<[^>]+>` treats `<![CDATA[...]]>` as a single tag and deletes it whole, so
everything Confluence keeps in a macro body — code blocks, info and note panels,
expand sections — never reaches the Knowledge Base. The page still syncs, quietly
missing the part that was usually worth reading. Across one 321-page space five
pages were affected, the worst losing 1.3KB of a documented HTML snippet.
Macro bodies are now parked before the markup passes and restored after the
whitespace pass, so a code block survives with its own line breaks, indentation
and angle brackets intact rather than having half of itself stripped as tags.
Block ends become newlines too: everything used to collapse onto one line, which
reads as a single thought and chunks as one.
Link labels are dropped along with their links. `<ac:plain-text-link-body>` holds
the title of another page in the space, and that page syncs as its own file, so
keeping labels turns every section index into a document that is nothing but
titles already in the Knowledge Base.
That leaves pages which genuinely have no text — a blank page, or one holding
only a children-display macro. They extract to zero bytes, and Open WebUI
extracts text as part of POST /files/ and answers a bare 400 for a file it can
get nothing out of, so each one fails on every run for as long as it exists and
the space can never sync clean. read_file now raises SourceFileUnavailable for
them, which run_sync already reports as a warning rather than an error.
Note for anyone with a space already synced: a Confluence checksum is built from
the page version rather than the page text, so pages pick up the better
extraction when their version next changes. Salting the manifest to force one
re-upload would fix that immediately but re-uploads every page of every
Confluence space, so it seemed a call for a maintainer rather than this patch.
* Enhance SharePointConnector for multi-cloud support
Updated SharePoint connector to support multiple cloud environments by adding cloud-specific endpoints. Adjusted token retrieval methods to use the appropriate graph base URL based on the selected cloud.
* fix: reorder default param
* feat: add support for non-root sites and subsites
* Refactor SharePointConnector initialization
* Add site_path parameter to SharePoint connector
* Clean up comments in parse_sharepoint_source function
Removed redundant comments and examples from the parse_sharepoint_source function.
* Enhance SharePointConnector for multi-cloud support
Updated SharePoint connector to support multiple cloud environments by adding cloud-specific endpoints. Adjusted token retrieval methods to use the appropriate graph base URL based on the selected cloud.
* fix: reorder default param
* fix: pass source token to gitlab and github connectors
* Add auth parameter to _resolve_connector function
Updated the _resolve_connector function to accept an optional auth parameter, allowing for source-specific credentials to be passed. Modified calls to _resolve_connector throughout the code to include the auth argument where applicable.
* Add auth parameter to connector resolution
Certificate auth built the x5t header from cert.signature_hash_algorithm,
which for any SHA-256-signed certificate is a 32-byte digest. The Microsoft
identity platform matches the registered key on its SHA-1 thumbprint, so
every such certificate was rejected:
AADSTS700027: The certificate with identifier used to sign the client
assertion is not registered on application. [Reason - The key was not found.]
Verified against a live Entra app registration: identical assertion, only
the x5t digest swapped, turns the 401 into an issued token.
Folder and file names were interpolated into the /root:/{path}: addressing
form as-is. A name containing '#' truncates the request at the fragment
marker, so walking a library with a folder like '#Archive' asks Graph for
/drives/{id}/root:/ and gets 400; '?', '%' and other reserved characters
are wrong in the same way, and read_file has the same flaw on download.
Encode with quote() so only the separators stay structural.
The Drive API does not return items from Shared Drives (formerly Team
Drives) unless requests opt in with supportsAllDrives and
includeItemsFromAllDrives. Without them, syncing a folder inside a
Shared Drive listed no files and reported "nothing to do", even with
correct service-account permissions.
Add these flags to all files.list calls in _walk_folder and _find_file,
and pass supportsAllDrives to the file get/get_media download calls so
Shared Drive files can be fetched.
Fixes#52
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
While syncing a Nextcloud source, a single missing folder currently aborts the entire sync process:
PROPFIND ... failed: HTTP 404
This can happen when:
a configured folder has been deleted or renamed;
a shared folder is no longer available;
permissions change between sync runs.
Proposed change:
Catch HTTPStatusError with status code 404 during folder traversal (_walk_folder).
Log a warning indicating that the folder was skipped.
Continue processing the remaining folders/files instead of failing the whole synchronization job.
Previously, GitLab source parser used a simple split limit to extract the owner namespace and repository name (e.g. `source.split("/", 2)`). This caused parsing failures for repositories located in nested subgroups (e.g., `gitlab:group/subgroup/project/docs/api`), as the subgroup path would get incorrectly split.
This commit fixes subgroup parsing by:
1. Supporting an explicit `project:path` separator format (e.g., `gitlab:group/subgroup/project:docs/api`) in `parse_gitlab_source`.
2. Adding a lazy resolution helper `_ensure_resolved()` to the connector. It probes the segments against the GitLab API (`/projects/{candidate}`) to dynamically identify the boundary between the project path and the repository file path.
3. Merging paths safely and falling back to the default split behavior if the API check fails or the client is offline.
Refactors `SyncHistory` to use a `queue.Queue` connection pool instead of `threading.local()`.
Previously, connections tied to `threading.local()` in a background worker pool would remain open until the thread died, leading to a steady leak of open file descriptors and locked connections.
Changes:
- Replaced `threading.local()` with a bounded `queue.Queue` pool (size=5) of `sqlite3.Connection` objects configured with `check_same_thread=False`.
- Safely wrapped the initial schema connection in a `try...finally` block to ensure `close()` is called, fixing a silent file lock leak.
- Added `PRAGMA synchronous=NORMAL` alongside `journal_mode=WAL` for drastically improved concurrent write performance without sacrificing durability.
- Updated all database operations to borrow from the pool using a context manager, allowing true multi-threaded concurrency (concurrent readers/writers via WAL) while strictly bounding the total number of open database connections.
- Updated `close()` to gracefully drain the queue and explicitly terminate all pooled connections.
Resolve the file path and verify that it remains within the connector's root directory using `is_relative_to()`. Raises a `PermissionError` if a path traversal attempt is detected.
- Upgrade Docker base image to python:3.14-alpine3.23
- Use Astral uv for multi-stage dependency installs and virtual env management
- Switch to running under non-root appuser (UID/GID 1000)
- Replace healthcheck curl command with wget to support Alpine
- Update docker-compose examples in README.md and docs/guide.md
Add SHAREPOINT_CERTIFICATE_PATH as a secure alternative to
SHAREPOINT_CLIENT_SECRET for authenticating with SharePoint via
Microsoft Graph API.
- JWT assertion flow using cryptography + PyJWT
- SHAREPOINT_CERTIFICATE_PASSWORD for encrypted PEM keys
- Mutual exclusivity validation (secret vs cert)
- New optional extra: pip install oikb[sharepoint-cert]
Closes#3
2KB inline HTML page, monospace, dark background. Polls /health
every 10s and renders source statuses with colored dots, last
sync time, duration, file counts. Zero dependencies.
