ouroboros/server.py
Ouroboros ae53779ccd release: Ouroboros v6.26.0-rc.1 — systemic hardening: immune system, custody, memory integrity, native multimodal chat
Owner-approved big release (plan: большой_релиз_v6.26.0). Workstreams:

- WS-A lint gate: ruff F-rules step in CI quick-test; fixed the real F823
  NameError class (supervisor/events.py utc_now_iso shadowing); make lint/health.
- WS-B memory integrity: atomic dialogue_blocks with corrupt-quarantine
  (.corrupt-<ts>.bak + memory_store_corrupt event); honest scratchpad journal
  (block_append_failed, corrupt storage renders as corruption not "(empty)");
  full Pattern Register window (16K cap vs 3K cut); merge-aware scratchpad
  consolidation under the sidecar lock; backlog writes through locked helpers;
  chat omission notes ("[N older unconsolidated messages omitted]").
- WS-C provider SSOT: provider registry (prefixes/credentials/resolution) in
  provider_models.py replacing 5 duplicated knowledge sites; credential-aware
  consolidation model; pricing empty-fetch retry; direct-route correctness
  (o-series max_completion_tokens, reasoning_effort, Anthropic error bodies,
  per-request timeouts on cached clients); deep/plan review budgets reserve
  output headroom inside the 1M window (min(SSOT, window − output − margin)).
- WS-D races/state: utils.update_json_locked (locked RMW, loud TimeoutError);
  task_results merge under per-file lock (cancel-latch safe); update_state
  migration for owner binding, evolution counters, budget_line, post-task
  activation; queue-lock coverage (enqueue, timeouts, worker health, task_done,
  snapshot); visible supervisor death (supervisor_error + owner notice + ghost
  consciousness/chat-agent cleanup on re-init); chat.jsonl rotation via
  os.replace under the append lock; unbounded _outbox removed; host-service
  port probe + actual-port panic sweep; skill lifecycle lane deadline
  (OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC, dedupe-leak-safe); consilium
  force-plan structured flag (plan_review_aggregate from the FULL result);
  atomic writers (update-intent, repo-manifest, metrics cache, post-task pair);
  review_state.save_state raises on lock timeout (advisory ledger honesty);
  apply_pending_request no-ops while a campaign is active;
  api_update_apply kills workers only AFTER update validation, with respawn
  on aborted checkout and "interrupted" terminal status.
- WS-E immune hardening: triad anti-refusal coverage contract (empty array
  needs NO_FINDINGS sentinel or bare-[] body; refusal prose with [] is
  parse_failure and never enters quorum); BIBLE P3 "Owner-chosen enforcement,
  loud advisory" bound + CHECKLISTS sync; every advisory pass-through of a
  blocking signal writes review_advisory_override + persistent
  advisory_overrides (surfaced by review_status); ToolEntry.mutates_worktree
  with dispatcher-level before/after worktree diff invalidation (covers error
  paths; read-only runs no longer invalidate; redundant manual calls removed);
  scope review fails closed without its checklist; advisory final write is a
  locked re-read merge; synthesis severity defaults to critical with WARNING
  fallbacks; find removed from SAFE_SHELL_COMMANDS; anti-thrashing state
  survives advisory criticals; preflight env scrubs secret-class variables;
  .git/index.lock age gate in worker startup checks.
- WS-F security: file-browser symlink containment on the RESOLVED path across
  all endpoints (out-of-root symlink targets are listed but inert; tests
  rewritten to the new contract deliberately); HMAC-signed session cookies
  (server-side persisted key, 30-day TTL, Secure on TLS) replacing the
  permanent password-derived cookie; password-class settings mask to a
  constant placeholder; conservative SSRF guard for the MAIN agent (link-local
  /cloud-metadata only, LAN stays reachable, per-request route re-validation);
  ClawHub zip-slip hardening (":"/backslash segments rejected + post-join
  containment) and lazy no-proxy OuroborosHub opener; single-execution
  signature dispatch for extension handlers (no TypeError re-run after side
  effects); onboarding postMessage origin checks; SHA256-pinned
  python-standalone download with pipefail; payload-resident dependency
  fingerprints only corroborate durable deps.json; skill payload re-hash
  immediately before spawn (TOCTOU narrowing).
- WS-G process custody: ouroboros/process_custody.py — spawn_supervised
  chokepoint + durable data/state/process_ledger.jsonl (pid, pgid,
  fingerprint{start_time, cmd_sha256}, purpose, scope task|session|daemon,
  owner_task, session_id); platform_layer.process_start_time primitive;
  startup + periodic reaper killing ONLY strict-fingerprint matches from dead
  generations/tasks (never by command-line class); migrations: services
  (the orphan hole), workspace executor + local model + extension companions
  (ledger write-through), worker_pids (write-through; legacy path retained);
  parent lifelines (ppid watchdog, group-suicide only as group leader) in
  worker_main, extension runner, Claude readonly child; conformance test
  pinning the Popen allowlist; ARCHITECTURE/DEVELOPMENT/CHECKLISTS entries.
- WS-H native multimodal chat: supports_vision capability map (static
  prefixes + OpenRouter /models input_modalities overlay); web chat uploads
  ride the WS frame as structured attachments (additive ChatInbound field)
  and image uploads become NATIVE image blocks via the existing Path B;
  browser screenshots inject natively for vision models via the multipart
  user-merge (tool result stays a string; file persisted under
  data/uploads/screenshots for re-view); K=3 newest-image eviction with
  caption placeholders carrying the vlm_query re-view path; image-aware token
  estimates (fixed ~1.1K-token equivalent instead of base64 length, fixing
  permanent emergency-compaction wedges); compaction renders images as
  captions (no base64 into the summarizer); GigaChat/local lanes emit explicit
  "[image omitted: model has no vision]"; internal _caption/_source_path
  metadata stripped from provider payloads.
- WS-I housekeeping (partial): SETTLED_STATUSES SSOT (+ cycle-safe mirror pin);
  owner_inject.py renamed to owner_mailbox.py; version-neutral envelope
  wording; files.py import-block cleanup. Remaining WS-I/WS-J/WS-K items are
  deferred with the owner's context-budget priority on review+release.

Review notes: triad+scope ran via scripts/run_external_review.py on the core
pack across 3 rounds to convergence (scope responded=PASS each round; round-2
criticals fixed: update_apply kill-order + respawn, lifecycle dedupe leak on
lane timeout, OUROBOROS_MAX_ROUNDS hot-reload + docs, toggle_evolution
NameError, async Anthropic timeout forwarding, supports_vision local check,
budget-update lock visibility, ChatInbound additive attachment contract,
file-browser doc sync). The FULL combined diff exceeds every triad model's
context window (~1.59M tokens > 1.05M) — reviewed in packs; remaining
cross-pack findings were verified as slicing artifacts. Adversarial critics
(GPT; Gemini/Opus rounds) ran on the working tree. Deliberate tradeoffs:
metadata-based eviction captions (no light-LLM call in the hot path);
worker ledger records use live-cmdline fingerprints with a synthetic-arg
fallback only where the OS offers no cmdline.
2026-06-10 16:08:53 +03:00

1306 lines
54 KiB
Python

"""Self-editable Starlette/uvicorn entry point for UI and supervisor runtime."""
import asyncio
import json
import logging
import socket
import os
import pathlib
import sys
import threading
import time
import uuid
from ouroboros.utils import utc_now_iso
from typing import Any, Dict, Optional
from starlette.applications import Starlette
from starlette.routing import Route, Mount
import uvicorn
from ouroboros.server_control import (
execute_panic_stop as _execute_panic_stop_impl,
restart_current_process as _restart_current_process_impl,
)
from ouroboros.server_auth import (
NetworkAuthGate,
get_network_auth_startup_warning,
validate_network_auth_configuration,
)
from ouroboros.server_entrypoint import find_free_port, parse_server_args, write_port_file
from ouroboros.server_web import NoCacheStaticFiles, make_index_page, resolve_web_dir
from ouroboros.gateway import collect_routes
from ouroboros.gateway import settings as _gateway_settings
from ouroboros.gateway.ws import (
broadcast_ws,
broadcast_ws_sync,
close_all_ws,
has_ws_clients as _has_ws_clients,
set_event_loop as _set_ws_event_loop,
)
REPO_DIR = pathlib.Path(os.environ.get("OUROBOROS_REPO_DIR", pathlib.Path(__file__).parent))
DATA_DIR = pathlib.Path(os.environ.get("OUROBOROS_DATA_DIR",
pathlib.Path.home() / "Ouroboros" / "data"))
DEFAULT_HOST = os.environ.get("OUROBOROS_SERVER_HOST", "127.0.0.1")
DEFAULT_PORT = int(os.environ.get("OUROBOROS_SERVER_PORT", "8765"))
PORT_FILE = DATA_DIR / "state" / "server_port"
sys.path.insert(0, str(REPO_DIR))
if not os.environ.get("OUROBOROS_AGENT_PYTHON"):
_agent_python = sys.executable
if isinstance(_agent_python, str) and _agent_python:
os.environ["OUROBOROS_AGENT_PYTHON"] = _agent_python
_LOG_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
_pytest_default_real_data_dir = (
"pytest" in sys.modules
and not os.environ.get("OUROBOROS_DATA_DIR")
and DATA_DIR == pathlib.Path.home() / "Ouroboros" / "data"
)
if _pytest_default_real_data_dir:
logging.basicConfig(level=logging.INFO, format=_LOG_FORMAT, handlers=[logging.StreamHandler()])
else:
_log_dir = DATA_DIR / "logs"
_log_dir.mkdir(parents=True, exist_ok=True)
from logging.handlers import RotatingFileHandler
_file_handler = RotatingFileHandler(
_log_dir / "server.log", maxBytes=2 * 1024 * 1024, backupCount=3, encoding="utf-8",
)
_file_handler.setFormatter(logging.Formatter(_LOG_FORMAT))
logging.basicConfig(level=logging.INFO, format=_LOG_FORMAT, handlers=[_file_handler, logging.StreamHandler()])
log = logging.getLogger("server")
RESTART_EXIT_CODE = 42
PANIC_EXIT_CODE = 99
_restart_requested = threading.Event()
_LAUNCHER_MANAGED = str(os.environ.get("OUROBOROS_MANAGED_BY_LAUNCHER", "") or "").strip() == "1"
# Captured in main() for Settings LAN-reachability metadata.
_BIND_HOST = DEFAULT_HOST
def _has_active_evolution_transaction() -> bool:
try:
path = DATA_DIR / "state" / "evolution_campaign.json"
if not path.is_file():
return False
raw = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(raw, dict):
return False
if raw.get("status") not in {"active", "paused"}:
return False
tx = raw.get("active_transaction")
return isinstance(tx, dict) and not str(tx.get("commit_sha") or "").strip()
except Exception:
return False
def _restart_current_process(host: str, port: int) -> None:
_restart_current_process_impl(host, port, repo_dir=REPO_DIR, log=log)
from ouroboros.config import (
load_settings, save_settings, apply_settings_to_env as _apply_settings_to_env,
)
from ouroboros.server_runtime import (
apply_runtime_provider_defaults,
has_startup_ready_provider,
needs_local_model_autostart,
setup_remote_if_configured,
ws_heartbeat_loop,
)
_supervisor_ready = threading.Event()
_supervisor_error: Optional[str] = None
_event_loop: Optional[asyncio.AbstractEventLoop] = None
_supervisor_thread: Optional[threading.Thread] = None
_consciousness: Any = None
def _describe_bg_consciousness_state(requested_enabled: bool) -> dict:
snapshot = _consciousness.status_snapshot() if _consciousness else {}
running = bool(snapshot.get("running"))
paused = bool(snapshot.get("paused"))
next_wakeup_sec = int(snapshot.get("next_wakeup_sec") or 0)
idle_reason = str(snapshot.get("last_idle_reason") or "")
detail = "Background consciousness is off."
status = "disabled"
if requested_enabled and running and paused:
status = "paused"
detail = "Paused while another foreground task is active."
elif requested_enabled and running and idle_reason == "thinking":
status = "running"
detail = "Background consciousness is thinking now."
elif requested_enabled and running and idle_reason == "budget_blocked":
status = "budget_blocked"
detail = "Background consciousness hit its budget allocation and is waiting."
elif requested_enabled and running:
status = "running"
detail = (
"Background consciousness is idle between wakeups."
+ (f" Next wakeup in {next_wakeup_sec}s." if next_wakeup_sec > 0 else "")
)
elif requested_enabled:
status = "stopped"
detail = "Enabled in state, but the background thread is not running."
if idle_reason == "error_backoff" and snapshot.get("last_error"):
status = "error_backoff"
detail = f"Waiting to retry after an internal error: {snapshot['last_error']}"
return {
"enabled": requested_enabled,
"status": status,
"detail": detail,
**snapshot,
}
def _start_supervisor_if_needed(settings: dict) -> bool:
"""Start the supervisor once when runtime providers become available."""
global _supervisor_thread, _supervisor_error
if not has_startup_ready_provider(settings):
return False
if _supervisor_thread and _supervisor_thread.is_alive():
return False
_supervisor_error = None
_supervisor_thread = threading.Thread(
target=_run_supervisor,
args=(settings,),
daemon=True,
name="supervisor-main",
)
_supervisor_thread.start()
return True
def _process_bridge_updates(bridge, offset: int, ctx: Any) -> int:
from supervisor.message_bus import coerce_chat_identity
updates = bridge.get_updates(offset=offset, timeout=1)
for upd in updates:
offset = int(upd["update_id"]) + 1
msg = upd.get("message") or {}
if not msg:
continue
chat_id = coerce_chat_identity((msg.get("chat") or {}).get("id"), 1)
user_id = coerce_chat_identity((msg.get("from") or {}).get("id"), chat_id or 1)
text = str(msg.get("text") or "")
source = str(msg.get("source") or "web")
sender_label = str(msg.get("sender_label") or "")
sender_session_id = str(msg.get("sender_session_id") or "")
client_message_id = str(msg.get("client_message_id") or "")
transport = msg.get("transport") if isinstance(msg.get("transport"), dict) else {}
image_base64 = str(msg.get("image_base64") or "")
image_mime = str(msg.get("image_mime") or "image/jpeg")
image_caption = str(msg.get("image_caption") or "")
suppress_chat_log = bool(msg.get("suppress_chat_log"))
task_constraint = msg.get("task_constraint") if isinstance(msg.get("task_constraint"), dict) else None
task_metadata = msg.get("task_metadata") if isinstance(msg.get("task_metadata"), dict) else None
image_data = (
(image_base64, image_mime, image_caption)
if image_base64
else None
)
log_text = text or image_caption or ("(image attached)" if image_base64 else "")
now_iso = utc_now_iso()
st = ctx.load_state()
owner_id = st.get("owner_id")
lowered = text.strip().lower()
is_slash_command = lowered.startswith("/")
is_external_transport = source != "web"
external_identity_present = (not is_external_transport) or (chat_id > 0 and user_id > 0)
# Global owner = primary chat for outbound notices (web on desktop, the
# first transport on headless Colab). Bound once, on the first message.
if owner_id is None and external_identity_present:
owner_id = user_id
from supervisor.message_bus import log_chat
if not suppress_chat_log:
log_chat(
"in",
chat_id,
user_id,
log_text,
source=source,
sender_label=sender_label,
sender_session_id=sender_session_id,
client_message_id=client_message_id,
transport=transport,
)
if source != "web":
bridge.broadcast({
"type": "photo" if image_base64 else "chat",
"role": "user",
"content": text,
"caption": image_caption,
"image_base64": image_base64,
"mime": image_mime,
"ts": now_iso,
"source": source,
"sender_label": sender_label,
"sender_session_id": sender_session_id,
"client_message_id": client_message_id,
"transport": transport,
"chat_id": chat_id,
})
# Atomic owner-binding + activity stamp: the old load→(log/broadcast)→save
# span could overwrite concurrent budget/state writers with stale data.
def _stamp_owner_activity(live: dict) -> None:
if live.get("owner_id") is None and external_identity_present:
live["owner_id"] = user_id
live["owner_chat_id"] = chat_id
live["last_owner_message_at"] = now_iso
ctx.update_state(_stamp_owner_activity)
if not text and not image_base64:
continue
if is_external_transport and is_slash_command:
if not external_identity_present:
ctx.send_with_budget(chat_id, "⚠️ Command ignored: this transport did not provide owner identity.")
continue
# External transports authorize slash commands against a SEPARATE
# owner-external slot, so the local web owner (1/1) can never lock out
# a real Telegram owner. The first external slash binds it (TOFU) and
# asks for a resend instead of executing.
owner_ext_id = st.get("owner_external_id")
owner_ext_chat_id = st.get("owner_external_chat_id")
if owner_ext_id is None:
def _bind_external_owner(live: dict) -> None:
if live.get("owner_external_id") is None:
live["owner_external_id"] = user_id
live["owner_external_chat_id"] = chat_id
live["owner_external_bound_at"] = now_iso
ctx.update_state(_bind_external_owner)
ctx.send_with_budget(chat_id, "✅ Owner chat registered. Send the command again to execute it.")
continue
try:
owner_ext_id_int = int(owner_ext_id or 0)
owner_ext_chat_id_int = int(owner_ext_chat_id or 0)
except (TypeError, ValueError):
owner_ext_id_int = 0
owner_ext_chat_id_int = 0
if owner_ext_id_int != user_id or owner_ext_chat_id_int != chat_id:
ctx.send_with_budget(chat_id, "⚠️ Command ignored: this transport is not the bound owner chat.")
continue
if lowered.startswith("/panic"):
ctx.send_with_budget(chat_id, "🛑 PANIC: killing everything. App will close.")
_execute_panic_stop(ctx.consciousness, ctx.kill_workers)
elif lowered.startswith("/restart"):
ctx.send_with_budget(chat_id, "♻️ Restarting.")
ok, restart_msg = ctx.safe_restart(reason="owner_restart", unsynced_policy="rescue_and_reset")
if not ok:
ctx.send_with_budget(chat_id, f"⚠️ Restart cancelled: {restart_msg}")
continue
state_dir = DATA_DIR / "state"
owner_restart_flag = state_dir / "owner_restart_no_resume.flag"
stable_skip_flag = state_dir / "panic_stop.flag"
try:
state_dir.mkdir(parents=True, exist_ok=True)
owner_restart_flag.write_text("owner_restart", encoding="utf-8")
# Pair owner flag with panic_stop for stable-build auto-resume compatibility.
stable_skip_flag.write_text("owner_restart_no_resume", encoding="utf-8")
except Exception:
owner_restart_flag.unlink(missing_ok=True)
stable_skip_flag.unlink(missing_ok=True)
log.warning("Failed to write owner restart no-resume flag", exc_info=True)
ctx.send_with_budget(chat_id, "⚠️ Restart cancelled: could not write restart state.")
continue
try:
ctx.kill_workers(
force=True,
terminal_status="cancelled",
result_reason="Owner restart stopped this task before process restart.",
)
except Exception:
owner_restart_flag.unlink(missing_ok=True)
stable_skip_flag.unlink(missing_ok=True)
log.warning("Restart cancelled because worker shutdown failed", exc_info=True)
try:
ctx.send_with_budget(chat_id, "⚠️ Restart cancelled: failed to stop workers.")
except Exception:
pass
continue
try:
ctx.send_with_budget(chat_id, "Stopping active task. New settings apply to the next message.")
except Exception:
log.warning("Failed to send owner restart stop notice; continuing restart", exc_info=True)
_request_restart_exit()
elif lowered == "/review" or lowered.startswith("/review "):
# Target the requesting chat so the ack and results return to the
# external transport owner, not the default web owner_chat_id.
ctx.queue_deep_self_review_task(reason="owner:/review", force=True, chat_id=chat_id)
elif lowered.startswith("/evolve"):
parts = lowered.split()
action = parts[1] if len(parts) > 1 else "on"
turn_on = action not in ("off", "stop", "0")
objective = ""
if turn_on and len(parts) > 2:
objective = text.split(None, 2)[2].strip()
if turn_on:
from supervisor.queue import evolution_block_reason
block = evolution_block_reason()
if block:
ctx.send_with_budget(chat_id, block)
continue
st2 = ctx.load_state()
st2["evolution_mode_enabled"] = bool(turn_on)
if turn_on:
st2["evolution_consecutive_failures"] = 0
# Owner-initiated evolution must not inherit a stale post-task one-shot
# autostop, which would disable the owner's campaign after one cycle.
st2["post_task_autostop"] = False
ctx.save_state(st2)
try:
from supervisor.queue import pause_evolution_campaign, start_evolution_campaign
if turn_on:
start_evolution_campaign(objective, source="owner_chat")
else:
pause_evolution_campaign("disabled via owner chat")
except Exception:
log.warning("Failed to update evolution campaign state", exc_info=True)
if not turn_on:
# Cancel the live evolution worker too — pruning PENDING alone
# leaves a mid-cycle task running (and eligible for retry).
from supervisor.queue import cancel_running_evolution_tasks
cancelled = cancel_running_evolution_tasks("disabled via owner chat")
ctx.PENDING[:] = [t for t in ctx.PENDING if str(t.get("type")) != "evolution"]
ctx.sort_pending()
ctx.persist_queue_snapshot(reason="evolve_off")
if cancelled:
ctx.send_with_budget(
chat_id,
f"🛑 Cancelled running evolution task(s): {', '.join(cancelled)}",
)
ctx.send_with_budget(chat_id, f"🧬 Evolution campaign: {'ON' if turn_on else 'OFF'}")
elif lowered.startswith("/bg"):
parts = lowered.split()
action = parts[1] if len(parts) > 1 else "status"
if action in ("start", "on", "1"):
result = ctx.consciousness.start()
_bg_s = ctx.load_state()
_bg_s["bg_consciousness_enabled"] = True
ctx.save_state(_bg_s)
ctx.send_with_budget(chat_id, f"🧠 {result}")
elif action in ("stop", "off", "0"):
result = ctx.consciousness.stop()
_bg_s = ctx.load_state()
_bg_s["bg_consciousness_enabled"] = False
ctx.save_state(_bg_s)
ctx.send_with_budget(chat_id, f"🧠 {result}")
else:
bg_status = "running" if ctx.consciousness.is_running else "stopped"
ctx.send_with_budget(chat_id, f"🧠 Background consciousness: {bg_status}")
elif lowered.startswith("/status"):
from supervisor.state import status_text
from supervisor.queue import SOFT_TIMEOUT_SEC, HARD_TIMEOUT_SEC
status = status_text(ctx.WORKERS, ctx.PENDING, ctx.RUNNING, SOFT_TIMEOUT_SEC, HARD_TIMEOUT_SEC)
ctx.send_with_budget(chat_id, status)
else:
ctx.consciousness.inject_observation(f"Message from my human: {log_text}")
agent = ctx.get_chat_agent()
def _run_constrained_or_resume(cid, txt, img, constraint, metadata, resume_consciousness: bool):
try:
ctx.handle_chat_direct(
cid,
txt,
img,
task_constraint=constraint,
task_metadata=metadata,
)
finally:
if resume_consciousness:
ctx.consciousness.resume()
if agent._busy:
if task_constraint or task_metadata:
threading.Thread(
target=_run_constrained_or_resume,
args=(chat_id, text or image_caption, image_data, task_constraint, task_metadata, False),
daemon=True,
).start()
else:
agent.inject_message(text or image_caption, image_data=image_data)
else:
ctx.consciousness.pause()
threading.Thread(
target=_run_constrained_or_resume,
args=(chat_id, text or image_caption, image_data, task_constraint, task_metadata, True),
daemon=True,
).start()
return offset
def _runtime_branch_defaults() -> tuple[str, str]:
branch_dev = "ouroboros"
branch_stable = "ouroboros-stable"
if not _LAUNCHER_MANAGED:
return branch_dev, branch_stable
try:
from supervisor import git_ops as git_ops_module
if hasattr(git_ops_module, "managed_branch_defaults"):
return git_ops_module.managed_branch_defaults(REPO_DIR)
except Exception:
pass
return branch_dev, branch_stable
def _bootstrap_supervisor_repo(settings: dict, git_ops_module=None):
if git_ops_module is None:
from supervisor import git_ops as git_ops_module
branch_dev, branch_stable = _runtime_branch_defaults()
git_ops_module.init(
repo_dir=REPO_DIR,
drive_root=DATA_DIR,
remote_url="",
branch_dev=branch_dev,
branch_stable=branch_stable,
)
git_ops_module.ensure_repo_present()
setup_remote_if_configured(settings, log)
if _LAUNCHER_MANAGED:
policy = "rescue_and_block" if _has_active_evolution_transaction() else "rescue_and_reset"
ok, msg = git_ops_module.safe_restart(reason="bootstrap", unsynced_policy=policy)
if not ok and policy == "rescue_and_block":
try:
from supervisor.queue import pause_evolution_campaign
from supervisor.state import load_state, save_state
st = load_state()
st["evolution_mode_enabled"] = False
save_state(st)
pause_evolution_campaign(f"bootstrap blocked to protect active evolution transaction: {msg}")
except Exception:
log.debug("Failed to pause evolution after blocked bootstrap", exc_info=True)
return ok, msg
log.info("Local-dev server start detected — skipping bootstrap git reset.")
deps_ok, deps_msg = git_ops_module.sync_runtime_dependencies(reason="bootstrap_local_dev")
if not deps_ok:
return False, f"Failed local-dev deps sync: {deps_msg}"
import_result = git_ops_module.import_test()
if import_result.get("ok"):
return True, "OK: local-dev bootstrap"
return False, f"Local-dev import test failed (rc={import_result.get('returncode', -1)})"
def _run_supervisor(settings: dict) -> None:
"""Initialize and run the supervisor loop. Called in a background thread."""
global _supervisor_error, _supervisor_thread, _consciousness
_apply_settings_to_env(settings)
# Supervisor revival (e.g. settings POST after a loop death) must not leak
# the previous generation: the old BackgroundConsciousness daemon thread
# would keep burning budget unreachable by /bg stop, and the cached direct
# chat agent stays bound to the OLD event queue (messages to a dead queue).
if _consciousness is not None:
try:
_consciousness.stop()
except Exception:
log.debug("Failed to stop previous consciousness instance", exc_info=True)
_consciousness = None
try:
from supervisor import workers as _workers_mod
_workers_mod._chat_agent = None
except Exception:
log.debug("Failed to reset cached chat agent", exc_info=True)
try:
from supervisor.message_bus import init as bus_init
from supervisor.message_bus import LocalChatBridge
bridge = LocalChatBridge(settings)
bridge._broadcast_fn = broadcast_ws_sync
from ouroboros.utils import set_log_sink
set_log_sink(bridge.push_log)
bus_init(
drive_root=DATA_DIR,
total_budget_limit=float(settings.get("TOTAL_BUDGET", 10.0)),
budget_report_every=10,
chat_bridge=bridge,
)
from supervisor.state import init as state_init, init_state, load_state, save_state, update_state
from supervisor.state import append_jsonl, update_budget_from_usage, rotate_chat_log_if_needed
state_init(DATA_DIR, float(settings.get("TOTAL_BUDGET", 10.0)))
init_state()
from supervisor.git_ops import safe_restart
ok, msg = _bootstrap_supervisor_repo(settings)
if not ok:
log.error("Supervisor bootstrap failed: %s", msg)
from supervisor.queue import (
enqueue_task, enforce_task_timeouts, enqueue_evolution_task_if_needed,
persist_queue_snapshot, restore_pending_from_snapshot,
cancel_task_by_id, queue_deep_self_review_task, sort_pending,
)
from supervisor.workers import (
init as workers_init, get_event_q, WORKERS, PENDING, RUNNING,
spawn_workers, kill_workers, assign_tasks, ensure_workers_healthy,
handle_chat_direct, _get_chat_agent, auto_resume_after_restart,
)
max_workers = int(settings.get("OUROBOROS_MAX_WORKERS", 10))
soft_timeout = int(settings.get("OUROBOROS_SOFT_TIMEOUT_SEC", 600))
hard_timeout = int(settings.get("OUROBOROS_HARD_TIMEOUT_SEC", 1800))
# Managed manifest branch defaults must drive worker commit/restart flows too.
_workers_branch_dev, _workers_branch_stable = _runtime_branch_defaults()
workers_init(
repo_dir=REPO_DIR, drive_root=DATA_DIR, max_workers=max_workers,
soft_timeout=soft_timeout, hard_timeout=hard_timeout,
total_budget_limit=float(settings.get("TOTAL_BUDGET", 10.0)),
branch_dev=_workers_branch_dev, branch_stable=_workers_branch_stable,
)
from supervisor.events import dispatch_event
from supervisor.message_bus import send_with_budget
from ouroboros.consciousness import BackgroundConsciousness
import types
import queue as _queue_mod
kill_workers()
spawn_workers(max_workers)
restored_pending = restore_pending_from_snapshot()
persist_queue_snapshot(reason="startup")
try:
from ouroboros.headless import prune_headless_task_drives, prune_task_drives
from ouroboros.utils import sweep_stale_temp_files
prune_report = prune_headless_task_drives(DATA_DIR)
task_drive_report = prune_task_drives(DATA_DIR)
# Reap orphaned atomic-write temp files (.*.tmp.*) left by a hard kill.
sweep_stale_temp_files(DATA_DIR)
if (
prune_report.get("pruned")
or prune_report.get("errors")
or task_drive_report.get("pruned")
or task_drive_report.get("errors")
):
append_jsonl(DATA_DIR / "logs" / "events.jsonl", {
"ts": utc_now_iso(),
"type": "headless_task_drive_prune",
"report": prune_report,
"task_drives": task_drive_report,
})
except Exception:
log.debug("Headless task drive prune failed", exc_info=True)
try:
from ouroboros.process_custody import reap_orphaned_processes
reaped = reap_orphaned_processes(DATA_DIR)
if reaped:
log.info("Process custody reaper killed %d orphaned process(es): %s", len(reaped), reaped)
except Exception:
log.debug("Process custody startup reap failed", exc_info=True)
try:
from ouroboros import subagent_worktrees
worktree_report = subagent_worktrees.prune_orphans()
if worktree_report.get("removed"):
append_jsonl(DATA_DIR / "logs" / "events.jsonl", {
"ts": utc_now_iso(),
"type": "subagent_worktree_prune",
"report": worktree_report,
})
except Exception:
log.debug("Subagent worktree prune failed", exc_info=True)
try:
from ouroboros.observability import prune_observability_blobs
from ouroboros.tools.services import prune_service_logs
observability_report = prune_observability_blobs(DATA_DIR)
service_report = prune_service_logs(DATA_DIR)
if (
observability_report.get("enabled")
or observability_report.get("manifest_count")
or observability_report.get("blob_count")
or observability_report.get("deleted_manifests")
or observability_report.get("deleted_blobs")
or observability_report.get("errors")
or service_report.get("deleted_dirs")
or service_report.get("deleted_files")
or service_report.get("errors")
):
append_jsonl(DATA_DIR / "logs" / "events.jsonl", {
"ts": utc_now_iso(),
"type": "runtime_artifact_prune",
"observability": observability_report,
"services": service_report,
})
except Exception:
log.debug("Runtime artifact prune failed", exc_info=True)
if restored_pending > 0:
st_boot = load_state()
if st_boot.get("owner_chat_id"):
send_with_budget(int(st_boot["owner_chat_id"]),
f"♻️ Restored pending queue from snapshot: {restored_pending} tasks.")
auto_resume_after_restart()
def _get_owner_chat_id() -> Optional[int]:
try:
st = load_state()
cid = st.get("owner_chat_id")
return int(cid) if cid else None
except Exception:
return None
_consciousness = BackgroundConsciousness(
drive_root=DATA_DIR, repo_dir=REPO_DIR,
event_queue=get_event_q(), owner_chat_id_fn=_get_owner_chat_id,
)
_bg_st = load_state()
if _bg_st.get("bg_consciousness_enabled"):
_consciousness.start()
log.info("Background consciousness auto-restored from saved state.")
branch_dev, branch_stable = _runtime_branch_defaults()
_event_ctx = types.SimpleNamespace(
DRIVE_ROOT=DATA_DIR, REPO_DIR=REPO_DIR,
BRANCH_DEV=branch_dev, BRANCH_STABLE=branch_stable,
bridge=bridge, WORKERS=WORKERS, PENDING=PENDING, RUNNING=RUNNING,
MAX_WORKERS=max_workers,
send_with_budget=send_with_budget, load_state=load_state, save_state=save_state,
update_state=update_state,
update_budget_from_usage=update_budget_from_usage, append_jsonl=append_jsonl,
enqueue_task=enqueue_task, cancel_task_by_id=cancel_task_by_id,
queue_deep_self_review_task=queue_deep_self_review_task, persist_queue_snapshot=persist_queue_snapshot,
safe_restart=safe_restart, kill_workers=kill_workers, spawn_workers=spawn_workers,
sort_pending=sort_pending, consciousness=_consciousness,
soft_timeout=soft_timeout, hard_timeout=hard_timeout,
get_chat_agent=_get_chat_agent, handle_chat_direct=handle_chat_direct,
request_restart=_request_restart_exit,
)
except Exception as exc:
_supervisor_error = f"Supervisor init failed: {exc}"
_consciousness = None
log.critical("Supervisor initialization failed", exc_info=True)
_supervisor_ready.set()
_supervisor_thread = None
return
_supervisor_ready.set()
log.info("Supervisor ready.")
offset = 0
crash_count = 0
_last_custody_reap = [time.time()]
while not _restart_requested.is_set():
try:
rotate_chat_log_if_needed(DATA_DIR)
ensure_workers_healthy()
event_q = get_event_q()
while True:
try:
evt = event_q.get_nowait()
except _queue_mod.Empty:
break
if evt.get("type") == "restart_request":
_handle_restart_in_supervisor(evt, _event_ctx)
continue
dispatch_event(evt, _event_ctx)
if _restart_requested.is_set():
break
enforce_task_timeouts()
try:
from supervisor.queue import check_scheduled_tasks
check_scheduled_tasks()
except Exception:
log.warning("Scheduled task check failed", exc_info=True)
# Periodic custody reap: catches task-scoped processes whose owning
# task finished (or was SIGKILLed) within this server generation.
if time.time() - _last_custody_reap[0] > 600:
_last_custody_reap[0] = time.time()
try:
from ouroboros.process_custody import reap_orphaned_processes
from supervisor.queue import RUNNING as _running_tasks
reap_orphaned_processes(
DATA_DIR, running_task_ids=set(_running_tasks.keys()),
)
except Exception:
log.debug("Periodic custody reap failed", exc_info=True)
try:
from ouroboros.post_task_evolution import apply_pending_request
from supervisor import state as _pte_state
apply_pending_request(_pte_state.DRIVE_ROOT)
except Exception:
log.debug("Post-task evolution apply failed", exc_info=True)
enqueue_evolution_task_if_needed()
assign_tasks()
persist_queue_snapshot(reason="main_loop")
offset = _process_bridge_updates(bridge, offset, _event_ctx)
crash_count = 0
time.sleep(0.5)
except Exception as exc:
crash_count += 1
log.error("Supervisor loop crash #%d: %s", crash_count, exc, exc_info=True)
if crash_count >= 3:
# Visible death: previously the loop returned with
# _supervisor_ready still set and no _supervisor_error, so
# tasks silently stopped being assigned with a healthy-looking
# /api/state. Record the failure and tell the owner.
_supervisor_error = f"Supervisor loop died after 3 consecutive crashes: {exc}"
_supervisor_ready.clear()
log.critical("Supervisor exceeded max retries: %s", _supervisor_error)
try:
st = load_state()
if st.get("owner_chat_id"):
send_with_budget(
int(st["owner_chat_id"]),
"🛑 Supervisor loop died after repeated crashes; tasks are no "
"longer being assigned. Saving settings or restarting the app "
f"will revive it. Last error: {exc}",
)
except Exception:
log.debug("Failed to notify owner about supervisor death", exc_info=True)
return
time.sleep(min(30, 2 ** crash_count))
_supervisor_thread = None
def _handle_restart_in_supervisor(evt: Dict[str, Any], ctx: Any) -> None:
"""Handle agent restart request via graceful shutdown + exit(42)."""
st = ctx.load_state()
if st.get("owner_chat_id"):
ctx.send_with_budget(
int(st["owner_chat_id"]),
f"♻️ Restart requested by agent: {evt.get('reason')}",
)
ok, msg = ctx.safe_restart(
reason="agent_restart_request", unsynced_policy="rescue_and_block",
)
if not ok:
try:
from supervisor.queue import pause_evolution_campaign
st["evolution_mode_enabled"] = False
ctx.save_state(st)
pause_evolution_campaign(f"agent restart blocked to protect local changes: {msg}")
except Exception:
log.debug("Failed to pause evolution after blocked agent restart", exc_info=True)
if st.get("owner_chat_id"):
ctx.send_with_budget(int(st["owner_chat_id"]), f"⚠️ Restart skipped: {msg}")
return
cleanup_status, cleanup_reason = _shutdown_task_cleanup_args(restart_requested=True)
ctx.kill_workers(force=True, terminal_status=cleanup_status, result_reason=cleanup_reason)
st2 = ctx.load_state()
st2["session_id"] = uuid.uuid4().hex
ctx.save_state(st2)
ctx.persist_queue_snapshot(reason="pre_restart_exit")
_request_restart_exit()
def _request_restart_exit() -> None:
"""Signal server shutdown with restart exit code."""
_restart_requested.set()
def _shutdown_task_cleanup_args(restart_requested: bool) -> tuple[str, str]:
"""Return ``(terminal_status, result_reason)`` for tasks torn down by a
graceful server shutdown.
A graceful shutdown — a requested restart (exit 42) or an external
stop/restart signal (SIGTERM/SIGINT) — is not a worker crash storm, so a
still-running task is finalized as ``cancelled`` with an honest reason
instead of the default crash-storm text the supervisor uses for real
worker deaths.
"""
if restart_requested:
reason = (
"Server restarted before this task finished; the task was "
"interrupted by the restart, not a worker crash."
)
else:
reason = (
"Server shut down (external stop/restart signal) before this task "
"finished; the task was interrupted, not a worker crash."
)
return "cancelled", reason
def _execute_panic_stop(consciousness, kill_workers_fn) -> None:
_execute_panic_stop_impl(
consciousness,
kill_workers_fn,
data_dir=DATA_DIR,
panic_exit_code=PANIC_EXIT_CODE,
log=log,
)
APP_START = time.time()
def _sync_gateway_settings_module() -> None:
"""Keep legacy server.* monkeypatch tests wired to gateway.settings."""
_gateway_settings.load_settings = load_settings
_gateway_settings.save_settings = save_settings
_gateway_settings._apply_settings_to_env = _apply_settings_to_env
_gateway_settings.apply_runtime_provider_defaults = apply_runtime_provider_defaults
async def api_settings_get(request):
_sync_gateway_settings_module()
return await _gateway_settings.api_settings_get(request)
async def api_settings_post(request):
_sync_gateway_settings_module()
return await _gateway_settings.api_settings_post(request)
web_dir = resolve_web_dir(REPO_DIR)
web_dir.mkdir(parents=True, exist_ok=True)
index_page = make_index_page(web_dir)
routes = [
Route("/", endpoint=index_page),
*collect_routes(
data_dir=DATA_DIR,
settings_handlers={
"api_onboarding": _gateway_settings.api_onboarding,
"api_claude_code_status": _gateway_settings.api_claude_code_status,
"api_claude_code_install": _gateway_settings.api_claude_code_install,
"api_settings_get": api_settings_get,
"api_settings_post": api_settings_post,
},
),
Mount("/static", app=NoCacheStaticFiles(directory=str(web_dir)), name="static"),
]
from contextlib import asynccontextmanager, suppress
@asynccontextmanager
async def lifespan(app):
global _event_loop
_event_loop = asyncio.get_running_loop()
_set_ws_event_loop(_event_loop)
ws_heartbeat_task = asyncio.create_task(
ws_heartbeat_loop(_has_ws_clients, broadcast_ws),
name="ws-heartbeat",
)
settings, provider_defaults_changed, _provider_default_keys = apply_runtime_provider_defaults(load_settings())
if provider_defaults_changed:
save_settings(settings, allow_elevation=True)
_apply_settings_to_env(settings)
# Pin boot-time runtime-mode after env apply; save_settings compares to this owner baseline.
from ouroboros.config import initialize_runtime_mode_baseline
initialize_runtime_mode_baseline()
has_local = needs_local_model_autostart(settings)
lifespan_drive_root = pathlib.Path(
app.state.drive_root
if hasattr(app, "state") and hasattr(app.state, "drive_root")
else DATA_DIR
)
default_real_data_dir = pathlib.Path.home() / "Ouroboros" / "data"
pytest_default_real_data_dir = (
(bool(os.environ.get("PYTEST_CURRENT_TEST")) or "pytest" in sys.modules)
and lifespan_drive_root == default_real_data_dir
and not os.environ.get("OUROBOROS_DATA_DIR")
)
# Source-mode must seed native skills too, matching packaged launcher layout.
try:
if pytest_default_real_data_dir:
log.info("Skipping native skills bootstrap against real DATA_DIR during pytest")
else:
from ouroboros.launcher_bootstrap import ensure_data_skills_seeded
ensure_data_skills_seeded()
except Exception:
log.warning("Native skills bootstrap failed", exc_info=True)
if has_startup_ready_provider(settings):
_start_supervisor_if_needed(settings)
else:
_supervisor_ready.set()
log.info("No supported provider or local routing configured. Supervisor not started.")
if has_local and settings.get("LOCAL_MODEL_SOURCE"):
from ouroboros.local_model_autostart import auto_start_local_model
threading.Thread(
target=auto_start_local_model, args=(settings,),
daemon=True, name="local-model-autostart",
).start()
host_service_task = None
host_service_server = None
extension_reconcile_task = None
try:
from ouroboros.event_bus import init_global_event_bus
from ouroboros.extension_companion import init_global_supervisor
from ouroboros.gateway.host_service import (
DEFAULT_HOST_SERVICE_HOST,
create_host_service_app,
host_service_port,
)
init_global_event_bus().set_loop(_event_loop)
init_global_supervisor(lifespan_drive_root)
host_service_app = create_host_service_app(lifespan_drive_root)
host_port = host_service_port()
# Probe the port first: uvicorn's Server.startup() calls sys.exit(1) on a
# bind error, and SystemExit raised inside an asyncio task escapes
# run_forever and takes down the WHOLE main server (a stale prior
# instance still holding the port is exactly the realistic trigger).
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as _probe:
_probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
_probe.bind((DEFAULT_HOST_SERVICE_HOST, host_port))
except OSError as bind_exc:
raise RuntimeError(
f"Host Service port {host_port} is busy: {bind_exc}"
) from bind_exc
host_service_config = uvicorn.Config(
host_service_app,
host=DEFAULT_HOST_SERVICE_HOST,
port=host_port,
log_level="warning",
)
host_service_server = uvicorn.Server(host_service_config)
host_service_task = asyncio.create_task(
host_service_server.serve(),
name="host-service-api",
)
log.info("Host Service API listening on %s:%d", DEFAULT_HOST_SERVICE_HOST, host_port)
except Exception:
log.warning("Failed to start Host Service API", exc_info=True)
try:
from ouroboros.skill_review_runner import reconcile_stale_review_jobs
if pytest_default_real_data_dir:
log.info("Skipping stale skill-review reconciliation against real DATA_DIR during pytest")
else:
reconcile_stale_review_jobs(lifespan_drive_root)
except Exception:
log.warning("Stale skill-review reconciliation at startup failed", exc_info=True)
# Reload enabled+reviewed extensions across restarts.
try:
from ouroboros.config import (
get_skills_repo_path,
load_settings as _load_settings,
)
from ouroboros.extension_loader import reload_all as _reload_extensions
from ouroboros.extension_loader import set_ws_broadcaster as _set_extension_ws_broadcaster
_set_extension_ws_broadcaster(broadcast_ws_sync)
repo_path = get_skills_repo_path()
if pytest_default_real_data_dir:
log.info("Skipping extension reload_all against real DATA_DIR during pytest")
else:
_reload_extensions(lifespan_drive_root, _load_settings, repo_path=repo_path or None)
except Exception:
log.error("Extension reload_all at startup failed", exc_info=True)
try:
from ouroboros.mcp_client import (
reconfigure_from_settings as _mcp_reconfigure_startup,
refresh_all_background as _mcp_refresh_background_startup,
)
_mcp_reconfigure_startup(settings)
_mcp_refresh_background_startup(reason="startup")
except Exception:
log.warning("MCP startup reconfigure failed", exc_info=True)
try:
from ouroboros.config import get_skills_repo_path
from ouroboros.config import load_settings as _load_settings
from ouroboros.extension_reconcile_queue import extension_reconcile_pickup_loop
if pytest_default_real_data_dir:
log.info("Skipping extension reconcile pickup against real DATA_DIR during pytest")
else:
extension_reconcile_task = asyncio.create_task(
extension_reconcile_pickup_loop(
lifespan_drive_root,
_load_settings,
repo_path_getter=lambda: get_skills_repo_path() or None,
),
name="extension-reconcile-pickup",
)
except Exception:
log.warning("Failed to start extension reconcile pickup task", exc_info=True)
try:
yield
finally:
if extension_reconcile_task is not None:
extension_reconcile_task.cancel()
with suppress(asyncio.CancelledError, asyncio.TimeoutError):
await asyncio.wait_for(extension_reconcile_task, timeout=30)
if host_service_server is not None:
try:
host_service_server.should_exit = True
except Exception:
pass
if host_service_task is not None:
with suppress(asyncio.CancelledError, asyncio.TimeoutError):
await asyncio.wait_for(host_service_task, timeout=5)
if not host_service_task.done():
host_service_task.cancel()
with suppress(asyncio.CancelledError, asyncio.TimeoutError):
await asyncio.wait_for(host_service_task, timeout=2)
ws_heartbeat_task.cancel()
with suppress(asyncio.CancelledError):
await ws_heartbeat_task
log.info("Server shutting down...")
try:
from ouroboros.local_model import get_manager
get_manager().stop_server()
except Exception:
pass
try:
from ouroboros.tools.shell import kill_all_tracked_subprocesses
kill_all_tracked_subprocesses()
except Exception:
pass
try:
from ouroboros.workspace_executor import kill_all_foreground
kill_all_foreground(lifespan_drive_root)
except Exception:
pass
try:
from ouroboros.tools.services import kill_all_services
kill_all_services(lifespan_drive_root)
except Exception:
pass
try:
from ouroboros.extension_companion import get_global_supervisor
supervisor = get_global_supervisor()
if supervisor is not None:
supervisor.stop_all()
except Exception:
pass
try:
restart_requested = _restart_requested.is_set()
# Record an explicit shutdown cause so a task interrupted by the
# shutdown is never later read as a worker crash storm.
try:
from ouroboros.utils import append_jsonl, utc_now_iso
append_jsonl(
lifespan_drive_root / "logs" / "supervisor.jsonl",
{
"ts": utc_now_iso(),
"type": "server_shutdown",
"cause": "restart_requested" if restart_requested else "external_signal",
"restart_exit": restart_requested,
},
)
except Exception:
log.debug("Failed to record server_shutdown event", exc_info=True)
from supervisor.workers import kill_workers
cleanup_status, cleanup_reason = _shutdown_task_cleanup_args(restart_requested)
kill_workers(force=True, terminal_status=cleanup_status, result_reason=cleanup_reason)
except Exception:
pass
try:
from supervisor.message_bus import get_bridge
get_bridge().shutdown()
except Exception:
pass
app = NetworkAuthGate(Starlette(routes=routes, lifespan=lifespan))
app.app.state.drive_root = pathlib.Path(DATA_DIR) # type: ignore[attr-defined]
app.app.state.repo_dir = pathlib.Path(REPO_DIR) # type: ignore[attr-defined]
app.app.state.broadcast_ws_sync = broadcast_ws_sync # type: ignore[attr-defined]
app.app.state.app_start = APP_START # type: ignore[attr-defined]
app.app.state.supervisor_ready_event = _supervisor_ready # type: ignore[attr-defined]
app.app.state.get_supervisor_error = lambda: _supervisor_error # type: ignore[attr-defined]
app.app.state.describe_bg_consciousness_state = _describe_bg_consciousness_state # type: ignore[attr-defined]
app.app.state.request_restart = _request_restart_exit # type: ignore[attr-defined]
app.app.state.runtime_branch_defaults = _runtime_branch_defaults # type: ignore[attr-defined]
app.app.state.bind_host = _BIND_HOST # type: ignore[attr-defined]
app.app.state.port_file = PORT_FILE # type: ignore[attr-defined]
app.app.state.default_port = DEFAULT_PORT # type: ignore[attr-defined]
app.app.state.start_supervisor_if_needed = _start_supervisor_if_needed # type: ignore[attr-defined]
_ACTUAL_BOUND_PORT: Optional[int] = None
def _actual_bound_port() -> int:
"""Port the server actually bound (set in main(); DEFAULT_PORT before that)."""
return _ACTUAL_BOUND_PORT if _ACTUAL_BOUND_PORT else DEFAULT_PORT
def _emergency_process_cleanup(*, port_sweep: bool = True) -> None:
"""Kill child processes, workers, companions, and runtime port holders."""
try:
from ouroboros.tools.shell import kill_all_tracked_subprocesses
kill_all_tracked_subprocesses()
except Exception:
pass
try:
from ouroboros.workspace_executor import kill_all_foreground
kill_all_foreground(DATA_DIR, wait=False)
except Exception:
pass
try:
from ouroboros.tools.services import kill_all_services
kill_all_services(DATA_DIR, wait=False)
except Exception:
pass
try:
from supervisor.workers import kill_workers
if _restart_requested.is_set():
# A restart that hung past the uvicorn shutdown timeout still reaches
# here; finalize running tasks as an honest interrupted-by-restart,
# not a worker crash storm.
cleanup_status, cleanup_reason = _shutdown_task_cleanup_args(True)
kill_workers(
force=True,
archive_service_logs=False,
terminal_status=cleanup_status,
result_reason=cleanup_reason,
)
else:
kill_workers(force=True, archive_service_logs=False)
except Exception:
pass
import multiprocessing
from ouroboros.platform_layer import force_kill_pid, kill_process_on_port
for child in multiprocessing.active_children():
try:
force_kill_pid(child.pid)
except (ProcessLookupError, PermissionError):
pass
# Reap the Process object so it does not linger as a zombie / keep
# active_children non-empty if the main process exits before it dies.
try:
child.join(timeout=2)
except Exception:
pass
if port_sweep:
# Sweep the ACTUALLY bound port (find_free_port may have moved off
# DEFAULT_PORT); the old hardcoded 8765/8766 pair could kill an
# unrelated process on a custom-port install.
kill_process_on_port(_actual_bound_port())
try:
from ouroboros.extension_companion import panic_kill_all
from ouroboros.gateway.host_service import host_service_port
panic_kill_all()
if port_sweep:
kill_process_on_port(host_service_port())
except Exception:
pass
def main() -> int:
try:
saved_host = str(load_settings().get("OUROBOROS_SERVER_HOST") or "").strip()
except Exception:
saved_host = ""
default_host = os.environ.get("OUROBOROS_SERVER_HOST", "").strip() or saved_host or DEFAULT_HOST
args = parse_server_args(default_host, DEFAULT_PORT)
global _BIND_HOST
_BIND_HOST = args.host
app.app.state.bind_host = args.host # type: ignore[attr-defined]
auth_warning = get_network_auth_startup_warning(args.host)
if auth_warning:
log.warning(auth_warning)
auth_error = validate_network_auth_configuration(args.host)
if auth_error:
log.error(auth_error)
return 2
actual_port = find_free_port(args.host, args.port)
if actual_port != args.port:
log.info("Port %d busy on %s, using %d instead", args.port, args.host, actual_port)
global _ACTUAL_BOUND_PORT
_ACTUAL_BOUND_PORT = actual_port
write_port_file(PORT_FILE, actual_port)
log.info("Starting Ouroboros server on %s:%d", args.host, actual_port)
config = uvicorn.Config(
app,
host=args.host,
port=actual_port,
log_level="warning",
ws_ping_interval=20,
ws_ping_timeout=20,
)
server = uvicorn.Server(config)
_uvicorn_exited = threading.Event()
def _check_restart():
"""Monitor restart signal, then shut down uvicorn."""
while not _restart_requested.is_set():
time.sleep(0.5)
log.info("Restart requested — closing WebSocket clients and shutting down server.")
loop = _event_loop
if loop:
try:
future = asyncio.run_coroutine_threadsafe(close_all_ws(), loop)
future.result(timeout=3)
except Exception:
pass
server.should_exit = True
# Force-exit only if uvicorn never returns; direct-server mode needs cleanup/re-exec time.
force_exit_timeout_sec = 5 if _LAUNCHER_MANAGED else 30
if _uvicorn_exited.wait(timeout=force_exit_timeout_sec):
return
log.warning(
"Uvicorn did not exit within %ss — running emergency cleanup before os._exit(%d)",
force_exit_timeout_sec,
RESTART_EXIT_CODE,
)
_emergency_process_cleanup()
os._exit(RESTART_EXIT_CODE)
threading.Thread(target=_check_restart, daemon=True).start()
try:
server.run()
finally:
_uvicorn_exited.set()
if _restart_requested.is_set():
log.info("Exiting with code %d (restart signal).", RESTART_EXIT_CODE)
_emergency_process_cleanup(port_sweep=False)
if not _LAUNCHER_MANAGED:
_restart_current_process(args.host, actual_port)
os._exit(RESTART_EXIT_CODE)
return 0
if __name__ == "__main__":
sys.exit(main())