mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-21 22:30:39 +00:00
Add scheduled client backup health reports (#3041)
Client backup reports can now be scheduled weekly or monthly as PDF or CSV, scoped to current VMs and containers by resource or client tag. Reports show recorded outcomes, daily failure rates, last retained success, failure details and missing evidence. Weekly reports cover seven completed local calendar days. Monthly reports cover the previous calendar month. Backup executions and observation coverage are retained in each organization's recovery store. Shrinking provider task inventories and restarts preserve recorded failures. Unavailable history, permission limits, collection gaps and incomplete multi-guest attribution remain explicit. A successful task does not certify that a backup is retained or restorable. This reports Proxmox VE backup jobs, including jobs writing to PBS. The existing scheduler, branding, delivery checks, bounded retries and deliberate uncertain-delivery recovery are reused. The public adapter supplies scoped evidence to a private PDF/CSV renderer. Pricing, feature entitlements and retention remain unchanged. Validation: - Frontend: 136 focused tests, full type check, lint, production build and bundle budgets. - Race checks cover recovery persistence, collector quality, reporting, organization isolation and runtime registration. Private rendering and runtime build checks cover the composed Pro implementation. - Live runtime checks on `/settings/support/reporting` and `/alerts/notifications` at 1440x1000, 820x844 and 390x844 in light and dark themes, with actual pixel review. The 61-workload selection remains bounded and keyboard usable. All five generated PDF pages were inspected. - Real client-scoped CSV and MIME attachment contents, retained failure after an empty provider inventory and restart, retry after restart, permanent rejection, uncertain acknowledgement and deliberate recovery. An unmatched client scope produces missing evidence with blank counts.
This commit is contained in:
rcourtman
GitHub
commit
0d5b877425
56 changed files with 2536 additions and 100 deletions
|
|
@ -9968,3 +9968,14 @@ organizations. `monitor_host_agents_test.go` exercises discovery delivery
|
|||
through both monitor initialization paths, including explicit and legacy
|
||||
default organizations. These notifications refresh inventory presentation;
|
||||
they do not grant enrollment, removal, re-enrollment or command authority.
|
||||
|
||||
### Backup health reporting preserves lifecycle authority
|
||||
|
||||
The reporting API's `backup_health` schedule kind reads current tenant inventory
|
||||
and retained recovery task evidence. It adds no agent enrollment, credential,
|
||||
command or remediation authority. It reuses file-report delivery/retry/restart
|
||||
handling and the existing advanced-reporting entitlement. Task history data
|
||||
quality is independent of delivery status. `reporting_backup_health_test.go`
|
||||
covers current scope, missing history, unchanged entitlement rejection and retry
|
||||
after handler recreation. Collection and persistence proofs remain owned by
|
||||
monitoring and storage-recovery.
|
||||
|
|
|
|||
|
|
@ -12181,6 +12181,57 @@ schedule persistence and retention checks do not keep using a stopped monitor.
|
|||
Clearing the dependency fails closed. `reporting_monitor_reload_test.go` covers
|
||||
replacement inventory/persistence and reads overlapping replacement.
|
||||
|
||||
### Client backup health reports use durable execution evidence
|
||||
|
||||
The additive `backup_health` schedule kind creates branded PDF or CSV reports
|
||||
for current organization VMs and system containers, selected explicitly, by
|
||||
current tags, or all when neither filter is supplied. It has no performance
|
||||
report resource cap. The organization snapshot and recovery store are resolved
|
||||
from the same request organization. Unavailable scope fails closed. Overlapping
|
||||
VMIDs across source connections cannot cross client scope. Historical deleted
|
||||
workloads or past tag membership are not reconstructed.
|
||||
|
||||
Reports summarize retained Proxmox VE backup task outcomes, including jobs
|
||||
writing to PBS, rather than inferring execution from backup artifacts. Native
|
||||
PBS writer, verification and synchronization jobs are not counted again.
|
||||
Weekly backup windows use the previous seven complete local calendar days.
|
||||
Monthly windows use the previous calendar month. Daily outcomes use the schedule
|
||||
timezone and completion cutoff. Unresolved
|
||||
outcomes, source observation gaps, unsupported providers, unqualified multi-guest
|
||||
jobs and unreadable history are explicit. Unobserved zero counts remain blank
|
||||
or unavailable, and failure percentages use recorded completed attempts only.
|
||||
Task success does not certify a retained or restorable backup.
|
||||
|
||||
`last_report_data_state` and `last_report_data_message` are server-owned,
|
||||
additive schedule fields. They describe the last generated report's evidence,
|
||||
independently of successful or failed delivery. Creation and edits cannot forge
|
||||
them. Weekly/monthly windows, original retry windows, notification preflight,
|
||||
SMTP outcomes, bounded retries, restart recovery, file retention and existing
|
||||
branding/advanced-reporting entitlement checks are shared with file reports.
|
||||
Pricing, grants and customer entitlements remain unchanged.
|
||||
|
||||
Proofs: `internal/api/reporting_backup_health_test.go` covers retained failures,
|
||||
more than fifty workloads, current client scope, missing history, server-owned
|
||||
quality fields, entitlement rejection and retry after handler recreation.
|
||||
`pulse-enterprise/internal/reportingadmin/backup_health_test.go` covers fixed-schema
|
||||
CSV, missing versus observed zero, spreadsheet formula escaping and PDF generation. The final-source
|
||||
Reports browser receipt covers scheduling and delivered attachment contents.
|
||||
|
||||
The public helper is a tenant/history adapter behind the existing reporting
|
||||
admin binder. `GenerateBackupHealthReportFunc` carries scoped evidence to the
|
||||
private renderer registered through `BusinessHooks` before router/scheduler
|
||||
creation. Missing rendering fails clearly rather than falling back to a public
|
||||
paid implementation. `pkg/reporting` exposes DTOs and existing PDF branding
|
||||
primitives. The new PDF/CSV implementation lives in pulse-enterprise. Exact-path
|
||||
boundary exemptions admit only that shared adapter, while Go and shell regression
|
||||
proofs continue rejecting neighboring implementations and private licensing imports.
|
||||
|
||||
Monitor dependency replacement also rebinds the shared recovery manager for
|
||||
existing and future tenant monitors. Single-monitor replacements receive it
|
||||
through the canonical dependency initializer. This keeps unattended collection
|
||||
writing to the report's store after a connection configuration reload.
|
||||
`recovery_monitor_reload_test.go` and the live report proof cover this lifecycle.
|
||||
|
||||
### Explicit historical incident archives
|
||||
|
||||
`internal/api/router.go` pins a read-only legacy archive to each organization's
|
||||
|
|
@ -12422,3 +12473,10 @@ the tenant monitor from each request context. Payload and route authority are
|
|||
unchanged. `TestChartServiceInitializationIsConcurrentAndRouterScoped` in
|
||||
`internal/api/contract_test.go` and `TestMultiTenant_TenantIsolation` are the
|
||||
direct race-detector proofs. This is a separate CI stabilization slice.
|
||||
|
||||
The additive reporting-catalog `backupHealthAvailable` flag describes runtime
|
||||
renderer presence independently of entitlement. New backup schedules are offered
|
||||
only when it is present, while existing schedules remain editable after a runtime
|
||||
change. Legacy catalogs default to false. API and catalog-model tests cover
|
||||
absence and registration so staggered source upgrades do not offer unavailable
|
||||
rendering. No price or entitlement changes.
|
||||
|
|
|
|||
|
|
@ -3704,3 +3704,18 @@ joined to billing accounts. This adds no price, entitlement, refund or billing
|
|||
behavior. Receiver intake and migration tests pin the additive schema boundary.
|
||||
Deploying the compatible receiver and release adoption remain separate from
|
||||
source verification. Count and disclosure semantics belong to security-privacy.
|
||||
|
||||
### Business backup health report renderer
|
||||
|
||||
New backup health PDF/CSV rendering lives in pulse-enterprise's reportingadmin
|
||||
package behind `GenerateBackupHealthReportFunc`. BusinessHooks installs that
|
||||
private callback without forking the canonical reporting handlers, scheduler or
|
||||
delivery system. Public code owns task collection, retained evidence, tenant
|
||||
scope, effective branding and the shared rendering contract. The public adapter
|
||||
fails explicitly when the private renderer is absent.
|
||||
|
||||
Existing advanced-reporting and branding entitlements, pricing, grants and
|
||||
metric/recovery retention remain unchanged. This source change does not deploy
|
||||
a runtime or change customer subscriptions. Private renderer tests cover formats,
|
||||
missing/observed-zero distinction, formula-safe CSV and PDF generation. Runtime
|
||||
hook tests and final-source Pro browser delivery qualify registration/composition.
|
||||
|
|
|
|||
|
|
@ -9770,3 +9770,38 @@ WebKit exercise the real policy producer, consuming drawer, shared clock and
|
|||
existing table evidence. No route, guest check, backup, threshold, collector or
|
||||
permission changes. This presentation result does not establish thaw, covered
|
||||
filesystem writes, native restart recovery or containing released software.
|
||||
|
||||
### Client backup report scheduling and data-quality states
|
||||
|
||||
Reports adds Client backup health to the existing schedule editor. PDF/CSV,
|
||||
weekly/monthly cadence, current workload/tag scope and existing email/disk
|
||||
delivery controls reuse the schedule workflow. Empty selection and tags mean
|
||||
all current VMs and system containers. The picker restricts this kind to those
|
||||
workload types and has no performance-report selection cap. Its existing default
|
||||
cap remains unchanged for other consumers. Switching kinds retains only valid
|
||||
workload selections and preserves the existing digest/inventory constraints.
|
||||
|
||||
The schedule card presents last report data quality separately from delivery
|
||||
acceptance. Missing/incomplete history never reads as zero failures or completed
|
||||
delivery failure. The browser payload cannot supply server-owned quality fields.
|
||||
Copy states the PVE task scope, PBS destinations and unsupported-provider state.
|
||||
Pricing and current advanced-reporting/branding entitlement behavior are intact.
|
||||
|
||||
Proofs: `ReportingPanel.test.tsx`, `ResourcePicker.test.tsx` and
|
||||
`reportingSchedulesModel.backupHealth.test.ts` under Settings tests cover report
|
||||
kinds, quality/delivery coexistence, unrestricted workload selection, payloads
|
||||
and kind switching. Final-source browser proof covers the actual editor, client
|
||||
scope, reload, delivery/recovery states and narrow/desktop layouts.
|
||||
|
||||
The shared picker uses semantic theme colours for workload names, selection,
|
||||
tags and chips. Large selections remain in a bounded scrollable summary so
|
||||
client filters and delivery controls stay reachable. Toggle rows expose pressed
|
||||
state, and keyboard removal scrolls selected chips into view. The browser matrix
|
||||
covers light and dark themes at all three widths.
|
||||
|
||||
The additive reporting-catalog `backupHealthAvailable` flag describes runtime
|
||||
renderer presence independently of entitlement. New backup schedules are offered
|
||||
only when it is present, while existing schedules remain editable after a runtime
|
||||
change. Legacy catalogs default to false. API and catalog-model tests cover
|
||||
absence and registration so staggered source upgrades do not offer unavailable
|
||||
rendering. No price or entitlement changes.
|
||||
|
|
|
|||
|
|
@ -7403,3 +7403,42 @@ The canonical Docker-ingest source guard follows both guest enumeration routes
|
|||
through `publishPVEGuests` to the canonical state update. Publication reconciliation
|
||||
updates its former direct-call assertions without changing Docker ingestion or
|
||||
the runtime ownership gate; the behavioral withdrawal controls remain required.
|
||||
|
||||
### Backup history collection carries completeness
|
||||
|
||||
`GetBackupTaskHistory` in `pkg/proxmox/backup_history.go` enumerates a bounded
|
||||
ninety-day PVE task window with pagination and a fixed collection cutoff. Offline
|
||||
nodes, failed reads, repeated/capped pages and unconfirmed `Sys.Audit` authority
|
||||
remain explicit partial evidence. A successful HTTP response alone cannot prove
|
||||
visibility of other users' backup jobs. The cluster client retains its canonical
|
||||
transport failover wrapper. Older adapters can still supply live tasks but
|
||||
cannot qualify report observation coverage.
|
||||
|
||||
`pollBackupTasks` persists mapped task evidence synchronously through the
|
||||
organization's recovery manager before recording collection quality. The
|
||||
continuity allowance derives from the configured backup interval/cycle cadence.
|
||||
An unavailable manager or failed write cannot certify complete observation.
|
||||
Multi-guest attribution requires a nontruncated, contiguous completed task log
|
||||
with a terminal marker and parsed guests. Incomplete logs remain unqualified
|
||||
and eligible for later collection. No backup artifact or restore assurance is
|
||||
created by an execution outcome.
|
||||
|
||||
Proofs: `pkg/proxmox/backup_history_test.go` covers pagination, permissions,
|
||||
partial reads and offline nodes. `internal/monitoring/backup_task_history_test.go`
|
||||
covers durable failures surviving an empty poll and log qualification.
|
||||
|
||||
Backup collection starts with a bounded retained-history backfill, then uses a
|
||||
durable collection checkpoint with a day of overlap. The oldest known pending
|
||||
execution remains in range until completion, including after restart. Failed
|
||||
reads do not advance the checkpoint. A capped backfill may advance a later
|
||||
incremental read without backdating observation coverage. Another node's
|
||||
failure cannot be hidden by a truncation reason. The live API/state keeps its
|
||||
existing 200 provider tasks per node plus their guest details, while the recovery
|
||||
store retains executions beyond that live window. Checkpoint, long-running-job,
|
||||
failed-read, capped-backfill and bounded-live-payload regression tests exercise
|
||||
these boundaries.
|
||||
|
||||
Recovery rollup and alert readers take a locked snapshot of the manager pointer,
|
||||
matching the existing setter during live monitor dependency rebinding. The
|
||||
`TestBackupHealthRecoveryRebindingCanOverlapActiveReaders` race proof exercises
|
||||
rebinding while both consumers remain active.
|
||||
|
|
|
|||
|
|
@ -2840,17 +2840,23 @@
|
|||
"pkg/aicontracts/orchestrator_deps.go",
|
||||
"pkg/extensions/ai_autofix.go",
|
||||
"pkg/extensions/audit_admin.go",
|
||||
"pkg/extensions/reporting_admin.go",
|
||||
"pkg/pulsecli/actions.go",
|
||||
"pkg/pulsecli/api_client.go",
|
||||
"pkg/pulsecli/fleet.go",
|
||||
"pkg/pulsecli/root.go",
|
||||
"pkg/reporting/backup_health.go",
|
||||
"pkg/reporting/catalog.go",
|
||||
"pkg/reporting/pdf_brand.go",
|
||||
"pulse-mobile:config/mobile-api-surface.json",
|
||||
"pulse-mobile:src/api/coreMobileContract.ts",
|
||||
"pulse-mobile:src/generated/coreCompatibility.ts",
|
||||
"scripts/generate-pulse-intelligence-docs.go",
|
||||
"scripts/generate-types.go",
|
||||
"scripts/release_control/generate_mobile_compatibility.py",
|
||||
"scripts/release_control/mobile_compatibility.py"
|
||||
"scripts/release_control/mobile_compatibility.py",
|
||||
"scripts/repo-boundary-paid-surface.allowlist",
|
||||
"scripts/repo-boundary-regression.py"
|
||||
],
|
||||
"verification": {
|
||||
"allow_same_subsystem_tests": false,
|
||||
|
|
@ -2858,6 +2864,29 @@
|
|||
"exact_files": [],
|
||||
"require_explicit_path_policy_coverage": true,
|
||||
"path_policies": [
|
||||
{
|
||||
"id": "client-backup-health-reports",
|
||||
"label": "client-scoped backup health rendering and schedule proof",
|
||||
"match_prefixes": [],
|
||||
"match_files": [
|
||||
"internal/api/enterprise_extension_reporting_admin.go",
|
||||
"internal/api/reporting_backup_health.go",
|
||||
"internal/api/reporting_catalog_handlers.go",
|
||||
"pkg/extensions/reporting_admin.go",
|
||||
"pkg/reporting/backup_health.go",
|
||||
"pkg/reporting/catalog.go",
|
||||
"pkg/reporting/pdf_brand.go",
|
||||
"scripts/repo-boundary-paid-surface.allowlist",
|
||||
"scripts/repo-boundary-regression.py"
|
||||
],
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/api/reporting_backup_health_test.go",
|
||||
"internal/api/reporting_boundary_test.go",
|
||||
"scripts/repo-boundary-regression.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "api-reference-docs",
|
||||
"label": "canonical and published API reference synchronization proof",
|
||||
|
|
@ -3252,6 +3281,7 @@
|
|||
"internal/api/report_schedule_delivery_test.go",
|
||||
"internal/api/report_schedule_inventory_test.go",
|
||||
"internal/api/report_schedules_test.go",
|
||||
"internal/api/reporting_backup_health_test.go",
|
||||
"internal/config/report_schedule_delivery_test.go"
|
||||
]
|
||||
},
|
||||
|
|
@ -3269,6 +3299,7 @@
|
|||
"exact_files": [
|
||||
"internal/api/report_schedule_inventory_test.go",
|
||||
"internal/api/reporting_availability_test.go",
|
||||
"internal/api/reporting_backup_health_test.go",
|
||||
"internal/api/reporting_handlers_test.go",
|
||||
"internal/api/reporting_monitor_reload_test.go",
|
||||
"internal/api/reporting_org_scope_test.go",
|
||||
|
|
@ -3280,14 +3311,20 @@
|
|||
"label": "router dependency replacement proof",
|
||||
"match_prefixes": [],
|
||||
"match_files": [
|
||||
"internal/api/router.go",
|
||||
"internal/api/router_helpers.go"
|
||||
],
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/api/contract_test.go",
|
||||
"internal/api/multi_tenant_setters_additional_test.go",
|
||||
"internal/api/recovery_monitor_reload_test.go",
|
||||
"internal/api/reporting_monitor_reload_test.go",
|
||||
"internal/api/router_helpers_more_test.go"
|
||||
"internal/api/router_helpers_more_test.go",
|
||||
"internal/api/router_resource_wiring_test.go",
|
||||
"internal/api/router_set_monitor_agents_test.go",
|
||||
"internal/api/router_webhook_cidr_reload_test.go"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -5365,6 +5402,7 @@
|
|||
"frontend-modern/src/components/Settings/NetworkSettingsPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/OperationsPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/RecoverySettingsPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/reportingCatalogModel.ts",
|
||||
"frontend-modern/src/components/Settings/ReportingFormField.tsx",
|
||||
"frontend-modern/src/components/Settings/ReportingPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/reportingPanelModel.ts",
|
||||
|
|
@ -5736,6 +5774,7 @@
|
|||
"match_files": [
|
||||
"frontend-modern/scripts/header-audit.mjs",
|
||||
"frontend-modern/src/components/Settings/AvailabilitySettingsPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/reportingCatalogModel.ts",
|
||||
"frontend-modern/src/components/Settings/ReportingFormField.tsx",
|
||||
"frontend-modern/src/components/Settings/ReportingPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/reportingPanelModel.ts",
|
||||
|
|
@ -5755,9 +5794,11 @@
|
|||
"exact_files": [
|
||||
"frontend-modern/src/__tests__/App.architecture.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/AvailabilitySettingsPanel.test.tsx",
|
||||
"frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/ReportingPanel.test.tsx",
|
||||
"frontend-modern/src/components/Settings/__tests__/reportingPanelModel.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/reportingResourceTypes.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/reportingSchedulesModel.backupHealth.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/reportingSchedulesModel.delivery.test.ts",
|
||||
"frontend-modern/src/components/Settings/__tests__/ResourcePicker.test.tsx",
|
||||
"frontend-modern/src/components/Settings/__tests__/settingsArchitecture.test.ts",
|
||||
|
|
@ -6176,6 +6217,7 @@
|
|||
"pkg/agents/docker/report.go",
|
||||
"pkg/agents/host/report.go",
|
||||
"pkg/agents/kubernetes/report.go",
|
||||
"pkg/proxmox/backup_history.go",
|
||||
"pkg/proxmox/ceph.go",
|
||||
"pkg/proxmox/client.go",
|
||||
"pkg/proxmox/cluster_client.go",
|
||||
|
|
@ -6189,6 +6231,21 @@
|
|||
"exact_files": [],
|
||||
"require_explicit_path_policy_coverage": true,
|
||||
"path_policies": [
|
||||
{
|
||||
"id": "backup-history-collection",
|
||||
"label": "paginated backup execution history and collection quality proof",
|
||||
"match_prefixes": [],
|
||||
"match_files": [
|
||||
"internal/monitoring/backup_task_history.go",
|
||||
"pkg/proxmox/backup_history.go"
|
||||
],
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/monitoring/backup_task_history_test.go",
|
||||
"pkg/proxmox/backup_history_test.go"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "maintenance-verification-runtime",
|
||||
"label": "post-maintenance verification monitoring proof",
|
||||
|
|
@ -6622,6 +6679,7 @@
|
|||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/monitoring/backup_task_history_test.go",
|
||||
"internal/monitoring/monitor_alert_intent_test.go",
|
||||
"internal/monitoring/monitor_backups_dir_storage_test.go",
|
||||
"internal/monitoring/monitor_full_coverage_test.go",
|
||||
|
|
@ -6726,6 +6784,7 @@
|
|||
"internal/monitoring/proxmox_large_cluster_poll_budget_test.go",
|
||||
"internal/monitoring/pve_protection_observation_test.go",
|
||||
"internal/monitoring/ratetracker_test.go",
|
||||
"internal/monitoring/recovery_rollups_resilience_test.go",
|
||||
"internal/monitoring/reload_test.go",
|
||||
"internal/monitoring/truenas_poller_test.go",
|
||||
"internal/unifiedresources/code_standards_test.go"
|
||||
|
|
@ -7772,6 +7831,7 @@
|
|||
"internal/telemetry/service_health_test.go",
|
||||
"internal/telemetry/telemetry_test.go",
|
||||
"pkg/server/reporting_engine_test.go",
|
||||
"pkg/server/server_test.go",
|
||||
"pkg/server/service_health_test.go",
|
||||
"pkg/server/telemetry_licensed_features_test.go",
|
||||
"pkg/server/telemetry_pulse_intelligence_test.go",
|
||||
|
|
@ -8085,11 +8145,13 @@
|
|||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/recovery/backup_health_test.go",
|
||||
"internal/recovery/idgen/idgen_test.go",
|
||||
"internal/recovery/manager/manager_test.go",
|
||||
"internal/recovery/mapper/proxmox/mapper_test.go",
|
||||
"internal/recovery/posture_test.go",
|
||||
"internal/recovery/recovery_test.go",
|
||||
"internal/recovery/store/backup_history_test.go",
|
||||
"internal/recovery/store/store_posture_test.go",
|
||||
"internal/recovery/store/store_queryplan_test.go",
|
||||
"internal/recovery/store/store_rollups_test.go",
|
||||
|
|
|
|||
|
|
@ -3398,3 +3398,15 @@ the request organization opt in through `RequireOrgAdmin`,
|
|||
closed. `internal/api/org_admin_session_authority_test.go` holds the session
|
||||
matrix and `TestSettingsWriteScopeSplitsInstanceAndOrgManagerSession` the
|
||||
settings scope split.
|
||||
|
||||
### Backup report renderer hook preserves authority
|
||||
|
||||
The server supplies `GenerateBackupHealthReportFunc` from BusinessHooks before
|
||||
router and scheduler construction, and clears it for runtimes without the hook.
|
||||
The callback receives already-scoped recovery evidence and effective branding.
|
||||
It adds no route, organization selection, credential, action or licensing
|
||||
authority. Existing report administrator/settings scopes and advanced-reporting
|
||||
checks remain before generation and delivery. Missing private rendering fails
|
||||
explicitly. `pkg/server/server_test.go` retains runtime identity separation.
|
||||
Backup reporting tests cover missing renderer and unchanged entitlement refusal.
|
||||
The live disposable Pro runtime proof exercises actual registered rendering.
|
||||
|
|
|
|||
|
|
@ -6993,3 +6993,47 @@ their own tenant; discovery results and successful node mutations do not
|
|||
establish backup coverage, protection freshness, restore readiness or recovery
|
||||
point evidence. The delivery change adds no persisted storage or recovery
|
||||
state and gives these events no storage mutation authority.
|
||||
|
||||
### Durable backup execution history and observation coverage
|
||||
|
||||
The per-organization recovery SQLite store retains backup task execution rows
|
||||
under the existing ninety-day policy. Task rows stay `KindOther`. Success never
|
||||
mints a restore artifact or protected posture. A shrinking provider task window
|
||||
does not reconcile away retained executions. Later running observations cannot
|
||||
erase a committed terminal outcome. Source identity is validated before writes.
|
||||
|
||||
Collection-quality rows and continuous observation intervals share that store.
|
||||
Task evidence commits before collection success can extend coverage. Failed
|
||||
writes, failed/partial reads, downtime beyond the actual polling cadence and
|
||||
older observations cannot fabricate or bridge complete history. Returned old
|
||||
tasks never backdate Pulse's observation period. Report reads take outcomes and
|
||||
coverage from one database snapshot, and fail explicitly on query/decode limits.
|
||||
All early returns from point upserts release their transaction, including
|
||||
errors scoped inside a point loop.
|
||||
|
||||
`BuildBackupHealth` scopes outcomes to current authoritative workload identities,
|
||||
using source/VMID only when unique within that selection. Ambiguous and
|
||||
unattributed multi-guest evidence reduces completeness without exposing another
|
||||
client's task details. Reports distinguish incomplete observations from an
|
||||
observed period with no attempts. Successful tasks do not prove backup retention
|
||||
or restorability. Metric retention and existing entitlements do not change.
|
||||
Proofs: `internal/recovery/store/backup_history_test.go` and
|
||||
`internal/recovery/backup_health_test.go` cover restart, shrinking inventory,
|
||||
failure/downtime boundaries, write failure, client isolation and DST/cutoffs.
|
||||
|
||||
Backup collection starts with a bounded retained-history backfill, then uses a
|
||||
durable collection checkpoint with a day of overlap. The oldest known pending
|
||||
execution remains in range until completion, including after restart. Failed
|
||||
reads do not advance the checkpoint. A capped backfill may advance a later
|
||||
incremental read without backdating observation coverage. Another node's
|
||||
failure cannot be hidden by a truncation reason. The live API/state keeps its
|
||||
existing 200 provider tasks per node plus their guest details, while the recovery
|
||||
store retains executions beyond that live window. Checkpoint, long-running-job,
|
||||
failed-read, capped-backfill and bounded-live-payload regression tests exercise
|
||||
these boundaries.
|
||||
|
||||
Router monitor replacement retains the shared per-organization recovery manager
|
||||
for existing and future monitors. Configuration reload must not silently stop
|
||||
durable task collection while reporting continues reading the original store.
|
||||
`internal/api/recovery_monitor_reload_test.go` and a live connection-reload report
|
||||
exercise dependency retention and actual history persistence.
|
||||
|
|
|
|||
96
frontend-modern/browser-verification/c30a36b2044d9c07.json
Normal file
96
frontend-modern/browser-verification/c30a36b2044d9c07.json
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
{
|
||||
"version": 1,
|
||||
"base_sha": "20fa477f4d5db803cd451fdd0f00340b058d3d6e",
|
||||
"verified_at": "2026-10-17T16:32:43.089Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Settings/ReportScheduleEditor.tsx",
|
||||
"frontend-modern/src/components/Settings/ReportingPanel.tsx",
|
||||
"frontend-modern/src/components/Settings/ResourcePicker.tsx",
|
||||
"frontend-modern/src/components/Settings/reportingCatalogModel.ts",
|
||||
"frontend-modern/src/components/Settings/reportingSchedulesModel.ts",
|
||||
"frontend-modern/src/components/Settings/useReportingPanelState.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Settings/ReportScheduleEditor.tsx": "083ea8933986cc20d9b5b8eda7eaf77b81639d4befc2fc91a72e3be9a7f0ca74",
|
||||
"frontend-modern/src/components/Settings/ReportingPanel.tsx": "5436f0a25062041a057a48811d5d200922e0c3559a541abd59626de16ab5c0c8",
|
||||
"frontend-modern/src/components/Settings/ResourcePicker.tsx": "fe272be96251301d94009aa02cb1b9f5fb24d2756db32cc259413a2fb8cf9215",
|
||||
"frontend-modern/src/components/Settings/reportingCatalogModel.ts": "05b836ebaaced053103ab16bdf75ad5273de959117b5a6a3dc49c34e4c135f1b",
|
||||
"frontend-modern/src/components/Settings/reportingSchedulesModel.ts": "0339d7d04a7999a603313615f6d528353910f856859840805e8af5a04aaf0afe",
|
||||
"frontend-modern/src/components/Settings/useReportingPanelState.ts": "17dfa24401869fb8d30db64fbf91e5aca919a97579bfdd1cdfdeb3d448fa593a"
|
||||
},
|
||||
"routes": [
|
||||
"/settings/support/reporting",
|
||||
"/alerts/notifications"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 820,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"With the extension flag omitted as on legacy/community catalogs, a new unavailable backup schedule is not offered. Remaining delivery proof uses the unmodified Pro catalog and real private renderer.",
|
||||
"In light and dark themes at 1440/820/390 widths: editor focus, report-kind switching, PDF/CSV, weekly/monthly controls, empty/search/keyboard selection/remove, all 61 workloads beyond performance cap, bounded scrollable selected chips including keyboard removal of the last item, clear selection and delivery disabled states exercised.",
|
||||
"Normal Proxmox polling writes real task evidence. Tag-scoped delivered CSV contains the client failure, excludes another client, includes 31 current workloads, exposes partial history and missing attempts. Accepted MIME attachment equals the saved file.",
|
||||
"After provider task inventory becomes empty and a real server restart, retained failure evidence still appears in a new report.",
|
||||
"Temporary SMTP rejection retries exactly once after a real restart, preserves the original backup period, and keeps data quality separate from delivery status.",
|
||||
"Permanent rejection stops retries. Lost acknowledgement remains unconfirmed. Duplicate-risk recovery cancel sends nothing, deliberate confirmation succeeds.",
|
||||
"Saved/reloaded client tags and backup kind survive editing. Monthly PDF saves to disk without mail. Disable persists, details survive polling and delete cancel/confirm work.",
|
||||
"A client tag matching no current workloads produces an explicit no-observation report; period and every daily count remain blank rather than a false zero-failure result."
|
||||
],
|
||||
"interactions": [
|
||||
"Create, focus, search and keyboard-toggle workloads, select 61, scroll selected chips and keyboard-remove the final workload, clear all, switch kinds, formats, cadence and delivery in light and dark at all three widths.",
|
||||
"Refuse missing email configuration, save a disabled draft, follow Email settings to Notifications and enable the persisted schedule.",
|
||||
"Generate real client-scoped CSV, compare MIME attachment bytes, remove provider task inventory and restart before generating another retained-history report.",
|
||||
"Retry temporary SMTP failure across restart with the same window, retain permanent failure and uncertain acknowledgement, cancel deliberate recovery and then confirm it.",
|
||||
"Edit and reload a monthly PDF disk report, disable, keep expanded details through polling, cancel deletion and then confirm it.",
|
||||
"Generate a no-matching-client report and check blank period and daily counts rather than a false zero-failure result."
|
||||
],
|
||||
"environment": "Real private runtime and disposable signed Pro grant, no dev/mock modes, synthetic Proxmox and local SMTP only.",
|
||||
"backend_content_sha256": {
|
||||
"internal/api/router.go": "3b78ff44989f69da7c66e099fecf325d42ab8dbb84dc9934354a3411cfe346fc",
|
||||
"internal/api/router_helpers.go": "c2c7cc946e87899df7ec60fb4320eebc88472e66fe276f46967fa4d20beec27a",
|
||||
"internal/api/reporting_catalog_handlers.go": "ea5303f3b5f809db6127ae4334c0a0b21088266482245c3b79ca76412d6c3ca3",
|
||||
"internal/api/enterprise_extension_reporting_admin.go": "6cfa596dbbffdd6f1e8222fd6752941b386e2178037ccfd8286e2639fce3f640",
|
||||
"pkg/server/server.go": "89ffcd7890be7bc0d1f77e91ca2702f412235cbab4ee74599892c7371d606848",
|
||||
"pkg/extensions/reporting_admin.go": "e4a6dba1a9b898c2a3994605259de066eddaa3d91f8ba1bab2cfa1088a0bf596",
|
||||
"pkg/reporting/catalog.go": "9a8ee2fc320efa38ec96443b7ed8cb33ca55c4b1840e91d83dfd9b2f7bc5aed3",
|
||||
"pkg/reporting/pdf_brand.go": "f1ed4e46302fb8a160a4add49f96e47fdf0ae48fa6a8e62b8f206fefcf184c56",
|
||||
"internal/api/metrics_reporting_handlers.go": "cd6b2beddafbeeaeef8ff6ea2a17c287505dbfe1b4c0a6d91ad90b800d70aa7e",
|
||||
"internal/api/report_schedules.go": "f90786da7a45b0ab68488764dd6bc778adee4fdbb74bf709b69f88a158a72760",
|
||||
"internal/api/reporting_backup_health.go": "14e64dfc01c14f9eccac0f60936f81c13f17053b30b131ad82b473a2a9eec0fe",
|
||||
"internal/config/report_schedules.go": "76f5e6c7da0807a842d20f85119bb341feba9ae7cc62b1d083a3838eb45808f0",
|
||||
"internal/monitoring/monitor_backups.go": "bac66f98218d085d65459732139bbacb4b5358043cfcbf1cd39cf06dcc58eb42",
|
||||
"internal/monitoring/backup_task_history.go": "f0bd57e866b7e74a445abda6075ea827af17850601c24ba6f6c290ef921c8e12",
|
||||
"internal/recovery/backup_health.go": "965e0df25f048b058b8bd8bc4472fa0e1ccc761b5f59047be1f0534e9b506bd0",
|
||||
"internal/recovery/model/backup_history.go": "06e249e7c6b5e814c3264e80059628c4b0a7208ae34cbe0f5cc5f016fd9c2df5",
|
||||
"internal/recovery/mapper/proxmox/mapper.go": "81145147d735f506dbc66d29cbe7d409f39f8350d9fa0be4ae9c048026155d39",
|
||||
"internal/recovery/store/backup_history.go": "9e02131d7928badb5c285317483da3f220c849c0c8354eda5819bd7c9622c93f",
|
||||
"internal/recovery/store/store.go": "8013620f6862a4ebac44434ac50a4b15a3ce8e4e0e5b0274010acfce4609e98b",
|
||||
"pkg/proxmox/backup_history.go": "6d946cdf375d1d1b2861c516ad4ec37dc2f7c07e30a3469bf310e4ec010c923e",
|
||||
"pkg/reporting/backup_health.go": "d2bc8f046e513bcc3abacf89e7b33576633ac6a231b4ed3dd4ccd12c02b6c190",
|
||||
"internal/monitoring/recovery_rollups.go": "315c163c3a684db8f6a3636ccaf46ad09c8c86d7b04eecffd98b874bff87bb8a"
|
||||
},
|
||||
"private_content_sha256": {
|
||||
"internal/reportingadmin/backup_health.go": "4208478e4baf570581c1c9222ccbd52aa299b1500b6d47b64361e8aa90bb1ca7",
|
||||
"internal/enterpriseruntime/runtime.go": "0509ac6e2724cd68bb58a3f9348013006aa350aa5c16f079db08e6aebb3e7363",
|
||||
"go.mod": "2506c706eee4a617d3796a9b6c65145dd87785095b8f54b28eebb4b24e6abb4b",
|
||||
"go.sum": "62d138996eb5a28047c905ec8a264354f4345f6242692dba8a0c60264649b9b7"
|
||||
},
|
||||
"artifact_sha256": {
|
||||
"client-backup-health.pdf": "32f9c60029c7ca5ebd8e4b2918c26a1eb3ec6e901f5310ed01ff096eb60d8a6d",
|
||||
"client-backup-health.csv": "8dc7b7496d14dae0c249f33d260add879ecf03404136bada6a602b13a754822b",
|
||||
"no-matching-client.csv": "6fefcc346c80b3b61b39bef165127cba0e46b3eb1a97af2d9258050cad1435e8"
|
||||
},
|
||||
"evidence": "/opt/pulse-release-worker/task-proofs/client-backup-reports-20261017/browser-evidence",
|
||||
"pixel_review": "Reviewed current desktop, intermediate and narrow screenshots of the backup editor, unselected and selected workload names, light/dark colours, search and focus states, the 61-workload scrollable selection and reachable client/delivery controls. Selected rows, names, tags and keyboard focus remain visible in both themes. Reviewed accepted delivery with a distinct amber incomplete-data message, retrying/rejected/unconfirmed states, saved PDF details and unmatched scope. Save/Cancel and expanded actions remain readable and reachable without horizontal overflow. Reviewed every page of the five-page PDF after final source, including repeated daily column headings, workload continuation headings, missing values, failure details, footer and corrected UTC metadata. No clipping, overlap or obscured controls found in the checked states."
|
||||
}
|
||||
|
|
@ -52,10 +52,15 @@ export default function ReportScheduleEditor(props: { state: EditorState }) {
|
|||
const performanceReport = () => reportingCatalog()?.performanceReport ?? null;
|
||||
const isDigestForm = () => scheduleForm().kind === 'patrol_digest';
|
||||
const isInventoryForm = () => scheduleForm().kind === 'vm_inventory';
|
||||
const isBackupHealthForm = () => scheduleForm().kind === 'backup_health';
|
||||
// Existing digest schedules stay editable after Patrol is switched off.
|
||||
const inventoryAvailable = () => !!reportingCatalog()?.vmInventoryExport || isInventoryForm();
|
||||
const showReportTypeSelect = () =>
|
||||
inventoryAvailable() || patrolDigestAvailable() || isDigestForm();
|
||||
inventoryAvailable() ||
|
||||
patrolDigestAvailable() ||
|
||||
isDigestForm() ||
|
||||
!!performanceReport() ||
|
||||
isBackupHealthForm();
|
||||
return (
|
||||
<section class="space-y-4 border-y border-base-300 py-4">
|
||||
<Show when={showReportTypeSelect()}>
|
||||
|
|
@ -67,12 +72,17 @@ export default function ReportScheduleEditor(props: { state: EditorState }) {
|
|||
help={
|
||||
isDigestForm()
|
||||
? 'Emails what Patrol checked, found, fixed, and spent over the last seven days. Nothing is saved to disk.'
|
||||
: isInventoryForm()
|
||||
? 'Exports the current inventory of all VMs in this organization as CSV each time it runs, including newly discovered VMs.'
|
||||
: 'A performance report for the resources you pick, as a PDF or CSV.'
|
||||
: isBackupHealthForm()
|
||||
? 'Reports backup successes, failures and missing evidence for client VMs and containers over the previous seven complete days or previous calendar month. Uses Proxmox VE task history, including backups writing to PBS.'
|
||||
: isInventoryForm()
|
||||
? 'Exports the current inventory of all VMs in this organization as CSV each time it runs, including newly discovered VMs.'
|
||||
: 'A performance report for the resources you pick, as a PDF or CSV.'
|
||||
}
|
||||
>
|
||||
<option value="resources">{reportScheduleKindLabel('resources')}</option>
|
||||
<Show when={reportingCatalog()?.backupHealthAvailable || isBackupHealthForm()}>
|
||||
<option value="backup_health">{reportScheduleKindLabel('backup_health')}</option>
|
||||
</Show>
|
||||
<Show when={inventoryAvailable()}>
|
||||
<option value="vm_inventory">{reportScheduleKindLabel('vm_inventory')}</option>
|
||||
</Show>
|
||||
|
|
@ -94,9 +104,11 @@ export default function ReportScheduleEditor(props: { state: EditorState }) {
|
|||
placeholder={
|
||||
isDigestForm()
|
||||
? 'Patrol weekly summary'
|
||||
: isInventoryForm()
|
||||
? 'Monthly VM inventory'
|
||||
: 'Monthly client report'
|
||||
: isBackupHealthForm()
|
||||
? 'Monthly client backup health'
|
||||
: isInventoryForm()
|
||||
? 'Monthly VM inventory'
|
||||
: 'Monthly client report'
|
||||
}
|
||||
/>
|
||||
</FormField>
|
||||
|
|
@ -170,7 +182,7 @@ export default function ReportScheduleEditor(props: { state: EditorState }) {
|
|||
onInput={(e) => updateScheduleForm({ time: e.currentTarget.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<Show when={scheduleForm().kind === 'resources'}>
|
||||
<Show when={scheduleForm().kind === 'resources' || isBackupHealthForm()}>
|
||||
<FormSelect
|
||||
label="Format"
|
||||
value={scheduleForm().format}
|
||||
|
|
@ -184,13 +196,18 @@ export default function ReportScheduleEditor(props: { state: EditorState }) {
|
|||
</Show>
|
||||
</div>
|
||||
|
||||
<Show when={scheduleForm().kind === 'resources'}>
|
||||
<Show when={scheduleForm().kind === 'resources' || isBackupHealthForm()}>
|
||||
<FormField
|
||||
label="Resources"
|
||||
helpText="Use explicit resources, tags, or both. Scheduled reports use the previous reporting boundary."
|
||||
helpText={
|
||||
isBackupHealthForm()
|
||||
? 'Leave resources and tags empty for all current VMs and containers. Pick workloads, tags, or both for a client. Non-Proxmox workloads show unavailable task outcomes.'
|
||||
: 'Use explicit resources, tags, or both. Scheduled reports use the previous reporting boundary.'
|
||||
}
|
||||
>
|
||||
<ResourcePicker
|
||||
maxSelection={performanceReport()?.multiResourceMax}
|
||||
maxSelection={isBackupHealthForm() ? null : performanceReport()?.multiResourceMax}
|
||||
resourceTypes={isBackupHealthForm() ? ['vm', 'system-container'] : undefined}
|
||||
selected={scheduleResources}
|
||||
onSelectionChange={setScheduleResources}
|
||||
/>
|
||||
|
|
|
|||
|
|
@ -282,8 +282,7 @@ export function ReportingPanel() {
|
|||
<div class="space-y-1">
|
||||
<h4 class="text-base font-semibold text-base-content">Scheduled reports</h4>
|
||||
<p class="text-sm text-muted">
|
||||
Schedule performance reports or a CSV of the current VM inventory, delivered by
|
||||
email or saved on this server.
|
||||
Email reports on a schedule or save them on this server.
|
||||
<Show when={patrolDigestAvailable()}>
|
||||
{' '}
|
||||
A Patrol weekly summary can also be emailed: what Patrol checked, found,
|
||||
|
|
@ -427,6 +426,21 @@ export function ReportingPanel() {
|
|||
{formatReportScheduleTime(schedule.next_run_at)}
|
||||
</p>
|
||||
</Show>
|
||||
<Show
|
||||
when={
|
||||
schedule.kind === 'backup_health' && schedule.last_report_data_message
|
||||
}
|
||||
>
|
||||
<p
|
||||
class={
|
||||
schedule.last_report_data_state === 'complete'
|
||||
? 'text-muted'
|
||||
: 'text-amber-700 dark:text-amber-300'
|
||||
}
|
||||
>
|
||||
Last report data: {schedule.last_report_data_message}
|
||||
</p>
|
||||
</Show>
|
||||
<details>
|
||||
<summary
|
||||
class="cursor-pointer py-2"
|
||||
|
|
|
|||
|
|
@ -33,7 +33,8 @@ export interface SelectedResource {
|
|||
}
|
||||
|
||||
interface ResourcePickerProps {
|
||||
maxSelection?: number;
|
||||
maxSelection?: number | null;
|
||||
resourceTypes?: ResourceType[];
|
||||
selected: Accessor<SelectedResource[]>;
|
||||
onSelectionChange: (items: SelectedResource[]) => void;
|
||||
}
|
||||
|
|
@ -43,19 +44,33 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
const [search, setSearch] = createSignal('');
|
||||
const [typeFilter, setTypeFilter] = createSignal<TypeFilter>('all');
|
||||
const [tagFilter, setTagFilter] = createSignal('');
|
||||
const maxSelection = () => props.maxSelection ?? DEFAULT_MAX_SELECTION;
|
||||
const maxSelection = () =>
|
||||
props.maxSelection === null
|
||||
? Number.POSITIVE_INFINITY
|
||||
: (props.maxSelection ?? DEFAULT_MAX_SELECTION);
|
||||
|
||||
// Filter to reportable resource types across infrastructure, workloads, storage, and recovery.
|
||||
const reportableResources = createMemo(() => {
|
||||
return resources().filter(
|
||||
(r) =>
|
||||
REPORTABLE_RESOURCE_TYPES.has(r.type) &&
|
||||
(!props.resourceTypes || props.resourceTypes.includes(r.type)),
|
||||
);
|
||||
});
|
||||
const typeFilterOptions = createMemo<FilterOption<TypeFilter>[]>(() =>
|
||||
RESOURCE_PICKER_TYPE_FILTERS.map((type) => ({
|
||||
RESOURCE_PICKER_TYPE_FILTERS.filter(
|
||||
(type) =>
|
||||
!props.resourceTypes ||
|
||||
type === 'all' ||
|
||||
reportableResources().some((resource) =>
|
||||
matchesReportableResourceTypeFilter(resource, type),
|
||||
),
|
||||
).map((type) => ({
|
||||
value: type,
|
||||
label: getResourcePickerTypeFilterLabel(type),
|
||||
})),
|
||||
);
|
||||
|
||||
// Filter to reportable resource types across infrastructure, workloads, storage, and recovery.
|
||||
const reportableResources = createMemo(() => {
|
||||
return resources().filter((r) => REPORTABLE_RESOURCE_TYPES.has(r.type));
|
||||
});
|
||||
|
||||
// Apply filters
|
||||
const filteredResources = createMemo(() => {
|
||||
let result = reportableResources();
|
||||
|
|
@ -191,7 +206,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
<Show
|
||||
when={reportableResources().length > 0}
|
||||
fallback={
|
||||
<div class="p-8 text-center text-slate-400">
|
||||
<div class="p-8 text-center text-muted">
|
||||
<p class="text-sm">{getResourcePickerEmptyState(false).title}</p>
|
||||
<p class="text-xs mt-1 text-muted">
|
||||
{getResourcePickerEmptyState(false).description}
|
||||
|
|
@ -202,7 +217,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
<Show
|
||||
when={filteredResources().length > 0}
|
||||
fallback={
|
||||
<div class="p-6 text-center text-slate-400 text-sm">
|
||||
<div class="p-6 text-center text-muted text-sm">
|
||||
{getResourcePickerEmptyState(true).title}
|
||||
</div>
|
||||
}
|
||||
|
|
@ -217,13 +232,16 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
{(resource) => {
|
||||
const badge = getResourceTypePresentation(resource.type) || {
|
||||
label: resource.type,
|
||||
badgeClasses: 'bg-slate-500 text-slate-400',
|
||||
badgeClasses: 'bg-surface-alt text-base-content',
|
||||
};
|
||||
return (
|
||||
<button
|
||||
class={`w-full flex items-start sm:items-center gap-3 px-3 py-2 text-left transition-colors border-b border-border last:border-b-0 ${
|
||||
isSelected(resource.id) ? 'bg-blue-600' : 'hover:bg-slate-800'
|
||||
class={`w-full flex items-start sm:items-center gap-3 px-3 py-2 text-left text-base-content transition-colors border-b border-border last:border-b-0 ${
|
||||
isSelected(resource.id)
|
||||
? 'bg-blue-50 dark:bg-blue-900/25'
|
||||
: 'hover:bg-surface-hover'
|
||||
}`}
|
||||
aria-pressed={isSelected(resource.id)}
|
||||
onClick={() => toggleResource(resource)}
|
||||
>
|
||||
{/* Checkbox */}
|
||||
|
|
@ -231,7 +249,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
class={`w-4 h-4 rounded border shrink-0 flex items-center justify-center ${
|
||||
isSelected(resource.id)
|
||||
? 'bg-blue-600 border-blue-600'
|
||||
: 'border-slate-600'
|
||||
: 'border-base-content/40'
|
||||
}`}
|
||||
>
|
||||
<Show when={isSelected(resource.id)}>
|
||||
|
|
@ -261,7 +279,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
{/* Name and ID */}
|
||||
<div class="flex-1 min-w-0">
|
||||
<div
|
||||
class="text-sm text-white sm:truncate wrap-break-word"
|
||||
class="text-sm text-base-content sm:truncate wrap-break-word"
|
||||
title={getPreferredInfrastructureDisplayName(resource)}
|
||||
>
|
||||
{getPreferredInfrastructureDisplayName(resource)}
|
||||
|
|
@ -276,7 +294,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
<Show when={resource.tags && resource.tags.length > 0}>
|
||||
<For each={resource.tags?.slice(0, 2)}>
|
||||
{(tag) => (
|
||||
<span class="text-xs px-1.5 py-0.5 rounded-sm bg-surface-hover text-slate-300">
|
||||
<span class="text-xs px-1.5 py-0.5 rounded-sm bg-surface-alt text-base-content">
|
||||
{tag}
|
||||
</span>
|
||||
)}
|
||||
|
|
@ -302,7 +320,7 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
<div class="hidden sm:flex gap-1 shrink-0">
|
||||
<For each={resource.tags?.slice(0, 2)}>
|
||||
{(tag) => (
|
||||
<span class="text-xs px-1.5 py-0.5 rounded-sm bg-surface-hover text-slate-300">
|
||||
<span class="text-xs px-1.5 py-0.5 rounded-sm bg-surface-alt text-base-content">
|
||||
{tag}
|
||||
</span>
|
||||
)}
|
||||
|
|
@ -351,22 +369,26 @@ export function ResourcePicker(props: ResourcePickerProps) {
|
|||
<span class="text-xs sm:text-sm text-muted">
|
||||
{props.selected().length} selected
|
||||
<Show when={props.selected().length >= maxSelection()}>
|
||||
<span class="text-amber-400 ml-1">(max)</span>
|
||||
<span class="text-amber-600 dark:text-amber-400 ml-1">(max)</span>
|
||||
</Show>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Selected summary chips */}
|
||||
<Show when={props.selected().length > 0}>
|
||||
<div class="flex flex-wrap gap-1.5">
|
||||
<div
|
||||
class="flex max-h-40 flex-wrap gap-1.5 overflow-y-auto py-1"
|
||||
role="group"
|
||||
aria-label="Selected resources"
|
||||
>
|
||||
<For each={props.selected()}>
|
||||
{(item) => (
|
||||
<span class="inline-flex items-center gap-1 pl-2 pr-1 py-1 rounded-md bg-blue-600 border border-blue-500 text-sm text-blue-300">
|
||||
{item.name}
|
||||
<span class="inline-flex max-w-full items-center gap-1 pl-2 pr-1 py-1 rounded-md bg-blue-50 dark:bg-blue-900/25 border border-blue-200 dark:border-blue-800 text-sm text-base-content">
|
||||
<span class="min-w-0 break-all">{item.name}</span>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="xs"
|
||||
class="min-h-0 p-0.5 text-blue-300 hover:bg-blue-500"
|
||||
class="min-h-0 shrink-0 p-0.5 text-base-content"
|
||||
onClick={() => removeSelected(item.id)}
|
||||
aria-label={`Remove ${item.name}`}
|
||||
title={`Remove ${item.name}`}
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ vi.mock('../ResourcePicker', () => ({
|
|||
}));
|
||||
|
||||
const baseCatalog = {
|
||||
backupHealthAvailable: true,
|
||||
id: 'advanced_reporting',
|
||||
title: 'Detailed Reporting',
|
||||
description: 'Canonical reporting surfaces',
|
||||
|
|
@ -141,6 +142,32 @@ function buildState(overrides: Record<string, unknown> = {}) {
|
|||
}
|
||||
|
||||
describe('ReportingPanel', () => {
|
||||
it('keeps incomplete backup evidence visible beside a successfully delivered report', () => {
|
||||
useReportingPanelStateMock.mockReturnValue(
|
||||
buildState({
|
||||
reportSchedules: () => [
|
||||
{
|
||||
id: 'backup',
|
||||
name: 'Client backups',
|
||||
kind: 'backup_health',
|
||||
enabled: true,
|
||||
cadence: { type: 'weekly', weekday: 'monday', time: '09:00', timezone: 'UTC' },
|
||||
scope: { resources: [], tags: [] },
|
||||
format: 'pdf',
|
||||
delivery: { method: 'email', to: [], attach: true, save_to_disk: true },
|
||||
last_run_status: 'ok',
|
||||
last_delivery_status: 'sent_with_attachment',
|
||||
last_report_data_state: 'partial',
|
||||
last_report_data_message:
|
||||
'Backup data is incomplete. The report identifies missing evidence.',
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
render(() => <ReportingPanel />);
|
||||
expect(screen.getByText('Email accepted')).toBeInTheDocument();
|
||||
expect(screen.getByText(/Last report data: Backup data is incomplete/)).toBeInTheDocument();
|
||||
});
|
||||
beforeEach(() => {
|
||||
useReportingPanelStateMock.mockReset();
|
||||
});
|
||||
|
|
@ -194,6 +221,7 @@ describe('ReportingPanel', () => {
|
|||
const select = screen.getByLabelText('Report type') as HTMLSelectElement;
|
||||
expect(Array.from(select.options).map((option) => option.textContent)).toEqual([
|
||||
'Performance report',
|
||||
'Client backup health',
|
||||
'VM inventory CSV',
|
||||
]);
|
||||
expect(screen.getByLabelText('Schedule name', { selector: 'input' })).toHaveFocus();
|
||||
|
|
@ -221,6 +249,7 @@ describe('ReportingPanel', () => {
|
|||
expect(select.value).toBe('resources');
|
||||
expect(Array.from(select.options).map((option) => option.textContent)).toEqual([
|
||||
'Performance report',
|
||||
'Client backup health',
|
||||
'VM inventory CSV',
|
||||
'Patrol weekly summary',
|
||||
]);
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ const renderPicker = (initialSelection: SelectedResource[] = []) => {
|
|||
|
||||
const renderPickerWithOptions = (
|
||||
initialSelection: SelectedResource[] = [],
|
||||
options: { maxSelection?: number } = {},
|
||||
options: { maxSelection?: number | null; resourceTypes?: Resource['type'][] } = {},
|
||||
) => {
|
||||
const onSelectionChange = vi.fn();
|
||||
render(() => {
|
||||
|
|
@ -59,6 +59,7 @@ const renderPickerWithOptions = (
|
|||
return (
|
||||
<ResourcePicker
|
||||
maxSelection={options.maxSelection}
|
||||
resourceTypes={options.resourceTypes}
|
||||
selected={selected}
|
||||
onSelectionChange={handleSelectionChange}
|
||||
/>
|
||||
|
|
@ -78,6 +79,29 @@ afterEach(() => {
|
|||
});
|
||||
|
||||
describe('ResourcePicker', () => {
|
||||
it('selects a whole client estate beyond the performance cap while excluding other resource types', async () => {
|
||||
mockResources = [
|
||||
...Array.from({ length: 61 }, (_, index) =>
|
||||
makeResource({ id: `vm-${index}`, name: `VM ${index}`, displayName: `VM ${index}` }),
|
||||
),
|
||||
makeResource({ id: 'host', type: 'agent', name: 'Host', displayName: 'Host' }),
|
||||
];
|
||||
const { onSelectionChange } = renderPickerWithOptions([], {
|
||||
maxSelection: null,
|
||||
resourceTypes: ['vm'],
|
||||
});
|
||||
await fireEvent.click(screen.getByRole('button', { name: /select all/i }));
|
||||
expect(onSelectionChange).toHaveBeenLastCalledWith(
|
||||
expect.arrayContaining([expect.objectContaining({ id: 'vm-60' })]),
|
||||
);
|
||||
expect(onSelectionChange.mock.lastCall?.[0]).toHaveLength(61);
|
||||
expect(
|
||||
onSelectionChange.mock.lastCall?.[0].some(
|
||||
(resource: SelectedResource) => resource.type === 'agent',
|
||||
),
|
||||
).toBe(false);
|
||||
expect(showWarningMock).not.toHaveBeenCalled();
|
||||
});
|
||||
it('keeps large resource inventories on the shared bounded list renderer', () => {
|
||||
expect(resourcePickerSource).toContain('PlatformWindowedList');
|
||||
expect(resourcePickerSource).toContain('estimatedItemHeight={56}');
|
||||
|
|
|
|||
|
|
@ -116,7 +116,7 @@ describe('parseReportingCatalog (entry-point branches)', () => {
|
|||
expectInvalid({ ...baseCatalog(), description: 7 });
|
||||
});
|
||||
|
||||
it('round-trips the full catalog shape when vmInventoryExport is null (null arm)', () => {
|
||||
it('normalizes the catalog when vmInventoryExport is null and backup availability is omitted', () => {
|
||||
const catalog = parseReportingCatalog(baseCatalog()) as ReportingCatalog;
|
||||
expect(catalog).toStrictEqual({
|
||||
id: 'advanced_reporting',
|
||||
|
|
@ -132,6 +132,7 @@ describe('parseReportingCatalog (entry-point branches)', () => {
|
|||
},
|
||||
performanceReport: basePerformanceReport(),
|
||||
vmInventoryExport: null,
|
||||
backupHealthAvailable: false,
|
||||
});
|
||||
// Explicit null-arm assertion.
|
||||
expect(catalog.vmInventoryExport).toBeNull();
|
||||
|
|
|
|||
|
|
@ -65,6 +65,14 @@ const baseCatalogPayload = {
|
|||
} as const;
|
||||
|
||||
describe('reporting catalog model', () => {
|
||||
it('offers backup health only when this runtime has the private renderer', () => {
|
||||
expect(parseReportingCatalog(baseCatalogPayload).backupHealthAvailable).toBe(false);
|
||||
expect(
|
||||
parseReportingCatalog({ ...baseCatalogPayload, backupHealthAvailable: true })
|
||||
.backupHealthAvailable,
|
||||
).toBe(true);
|
||||
expect(buildLegacyReportingCatalogFallback().backupHealthAvailable).toBe(false);
|
||||
});
|
||||
it('builds the canonical reporting catalog request', () => {
|
||||
expect(buildReportingCatalogRequest()).toEqual({
|
||||
url: '/api/admin/reports/catalog',
|
||||
|
|
|
|||
|
|
@ -0,0 +1,64 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
applyReportScheduleKind,
|
||||
buildReportSchedulePayload,
|
||||
DEFAULT_REPORT_SCHEDULE_FORM,
|
||||
parseReportSchedulesResponse,
|
||||
reportScheduleScopeLabel,
|
||||
scheduleToForm,
|
||||
} from '../reportingSchedulesModel';
|
||||
|
||||
describe('client backup health schedules', () => {
|
||||
it('supports an organization-wide backup report without a performance selection cap', () => {
|
||||
const form = applyReportScheduleKind(DEFAULT_REPORT_SCHEDULE_FORM(), 'backup_health');
|
||||
const payload = buildReportSchedulePayload({ ...form, name: 'Client backups' }, []);
|
||||
expect(payload.kind).toBe('backup_health');
|
||||
expect(payload.format).toBe('pdf');
|
||||
expect(payload.scope).toEqual({ resources: [], tags: [] });
|
||||
expect(reportScheduleScopeLabel(payload)).toBe('All current VMs and containers');
|
||||
const resources = Array.from({ length: 61 }, (_, i) => ({
|
||||
id: `vm-${i}`,
|
||||
type: 'vm' as const,
|
||||
name: `VM ${i}`,
|
||||
}));
|
||||
expect(buildReportSchedulePayload(form, resources).scope.resources).toHaveLength(61);
|
||||
});
|
||||
|
||||
it('retains client tags and selected format when switching from performance reporting', () => {
|
||||
const form = applyReportScheduleKind(
|
||||
{ ...DEFAULT_REPORT_SCHEDULE_FORM(), format: 'csv', tagFilter: 'client-a' },
|
||||
'backup_health',
|
||||
);
|
||||
const payload = buildReportSchedulePayload(form, [
|
||||
{ id: 'vm-client-a', type: 'vm', name: 'Client A' },
|
||||
]);
|
||||
expect(payload.scope.tags).toEqual(['client-a']);
|
||||
expect(payload.scope.resources?.[0].resourceId).toBe('vm-client-a');
|
||||
expect(payload.format).toBe('csv');
|
||||
const restored = applyReportScheduleKind(
|
||||
applyReportScheduleKind(form, 'patrol_digest'),
|
||||
'backup_health',
|
||||
);
|
||||
expect(restored.attach).toBe(true);
|
||||
expect(restored.saveToDisk).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps missing backup data separate from successful delivery after reload', () => {
|
||||
const schedule = {
|
||||
...buildReportSchedulePayload(
|
||||
{ ...DEFAULT_REPORT_SCHEDULE_FORM(), kind: 'backup_health' },
|
||||
[],
|
||||
),
|
||||
last_run_status: 'ok',
|
||||
last_delivery_status: 'sent_with_attachment',
|
||||
last_report_data_state: 'partial',
|
||||
last_report_data_message: 'Backup data is incomplete.',
|
||||
};
|
||||
const parsed = parseReportSchedulesResponse({ schedules: [schedule] });
|
||||
expect(parsed[0].last_report_data_state).toBe('partial');
|
||||
expect(parsed[0].last_run_status).toBe('ok');
|
||||
expect(scheduleToForm(parsed[0]).kind).toBe('backup_health');
|
||||
const outgoing = buildReportSchedulePayload(scheduleToForm(parsed[0]), []);
|
||||
expect(outgoing).not.toHaveProperty('last_report_data_state');
|
||||
});
|
||||
});
|
||||
|
|
@ -58,6 +58,7 @@ export interface ReportingCatalog {
|
|||
guidance: ReportingGuidanceDefinition;
|
||||
performanceReport: ReportingPerformanceReportDefinition;
|
||||
vmInventoryExport: ReportingInventoryExportDefinition | null;
|
||||
backupHealthAvailable?: boolean;
|
||||
}
|
||||
|
||||
export function buildReportingCatalogRequest(): { url: string } {
|
||||
|
|
@ -124,6 +125,7 @@ export function buildLegacyReportingCatalogFallback(): ReportingCatalog {
|
|||
supportsCustomTitle: true,
|
||||
},
|
||||
vmInventoryExport: null,
|
||||
backupHealthAvailable: false,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -284,6 +286,7 @@ export function parseReportingCatalog(input: unknown): ReportingCatalog {
|
|||
lockedState: parseReportingLockedStateDefinition(candidate.lockedState),
|
||||
guidance: parseReportingGuidanceDefinition(candidate.guidance),
|
||||
performanceReport: parseReportingPerformanceReportDefinition(candidate.performanceReport),
|
||||
backupHealthAvailable: candidate.backupHealthAvailable === true,
|
||||
vmInventoryExport:
|
||||
candidate.vmInventoryExport === null
|
||||
? null
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ export type ReportScheduleRunStatus = '' | 'ok' | 'failed' | 'running' | 'retryi
|
|||
export type ReportScheduleDeliveryMethod = 'email' | 'disk';
|
||||
// 'resources' is the multi-resource performance report; 'patrol_digest' emails
|
||||
// the last seven days of Patrol work and has no scope, format, or disk output.
|
||||
export type ReportScheduleKind = 'resources' | 'patrol_digest' | 'vm_inventory';
|
||||
export type ReportScheduleKind = 'resources' | 'patrol_digest' | 'vm_inventory' | 'backup_health';
|
||||
// Digest schedules carry the server-assigned 'email' format: the email body is
|
||||
// the report, so there is no PDF or CSV artifact.
|
||||
export type ReportScheduleFormat = ReportingFormat | 'email';
|
||||
|
|
@ -46,6 +46,8 @@ export interface ReportSchedule {
|
|||
last_error?: string;
|
||||
last_delivery_status?: string;
|
||||
last_report_path?: string;
|
||||
last_report_data_state?: 'complete' | 'partial' | 'not_observed' | 'unavailable';
|
||||
last_report_data_message?: string;
|
||||
last_success_at?: string;
|
||||
run_attempts?: number;
|
||||
retry_at?: string;
|
||||
|
|
@ -122,7 +124,9 @@ export function parseReportSchedulesResponse(value: unknown): ReportSchedule[] {
|
|||
}
|
||||
|
||||
export function normalizeReportScheduleKind(kind: unknown): ReportScheduleKind {
|
||||
return kind === 'patrol_digest' || kind === 'vm_inventory' ? kind : 'resources';
|
||||
return kind === 'patrol_digest' || kind === 'vm_inventory' || kind === 'backup_health'
|
||||
? kind
|
||||
: 'resources';
|
||||
}
|
||||
|
||||
export function isPatrolDigestSchedule(schedule: Pick<ReportSchedule, 'kind'>): boolean {
|
||||
|
|
@ -224,6 +228,7 @@ export function applyReportScheduleKind(
|
|||
}
|
||||
|
||||
export function reportScheduleKindLabel(kind: unknown): string {
|
||||
if (kind === 'backup_health') return 'Client backup health';
|
||||
if (kind === 'vm_inventory') return 'VM inventory CSV';
|
||||
return normalizeReportScheduleKind(kind) === 'patrol_digest'
|
||||
? 'Patrol weekly summary'
|
||||
|
|
@ -315,6 +320,8 @@ export function reportScheduleScopeLabel(schedule: ReportSchedule): string {
|
|||
if (schedule.kind === 'vm_inventory') return 'All VMs in this organization';
|
||||
const resources = schedule.scope.resources?.length ?? 0;
|
||||
const tags = schedule.scope.tags?.length ?? 0;
|
||||
if (schedule.kind === 'backup_health' && resources === 0 && tags === 0)
|
||||
return 'All current VMs and containers';
|
||||
const parts = [];
|
||||
if (resources > 0) parts.push(`${resources} resource${resources === 1 ? '' : 's'}`);
|
||||
if (tags > 0) parts.push(`${tags} tag${tags === 1 ? '' : 's'}`);
|
||||
|
|
|
|||
|
|
@ -298,7 +298,13 @@ export const useReportingPanelState = () => {
|
|||
|
||||
const setScheduleKind = (kind: ReportScheduleKind) => {
|
||||
setScheduleForm((current) => applyReportScheduleKind(current, kind));
|
||||
if (kind !== 'resources') {
|
||||
if (kind === 'backup_health') {
|
||||
setScheduleResources((current) =>
|
||||
current.filter(
|
||||
(resource) => resource.type === 'vm' || resource.type === 'system-container',
|
||||
),
|
||||
);
|
||||
} else if (kind !== 'resources') {
|
||||
setScheduleResources([]);
|
||||
}
|
||||
};
|
||||
|
|
|
|||
|
|
@ -226,6 +226,9 @@ func TestPaidDomainBoundaryAudit(t *testing.T) {
|
|||
// Subject identity/alert enrichment for that same default pipeline.
|
||||
// This exact helper adds no endpoint, entitlement or private import.
|
||||
"reporting_subject_alerts.go": true,
|
||||
// Shared tenant/history adapter only. New paid PDF/CSV rendering lives
|
||||
// in pulse-enterprise behind GenerateBackupHealthReportFunc.
|
||||
"reporting_backup_health.go": true,
|
||||
}
|
||||
|
||||
for name := range goFiles {
|
||||
|
|
|
|||
|
|
@ -9,8 +9,21 @@ import (
|
|||
var (
|
||||
reportingAdminBindMu sync.RWMutex
|
||||
reportingAdminEndpointsBinder extensions.BindReportingAdminEndpointsFunc
|
||||
backupHealthReportGenerator extensions.GenerateBackupHealthReportFunc
|
||||
)
|
||||
|
||||
func SetBackupHealthReportGenerator(generator extensions.GenerateBackupHealthReportFunc) {
|
||||
reportingAdminBindMu.Lock()
|
||||
defer reportingAdminBindMu.Unlock()
|
||||
backupHealthReportGenerator = generator
|
||||
}
|
||||
|
||||
func resolveBackupHealthReportGenerator() extensions.GenerateBackupHealthReportFunc {
|
||||
reportingAdminBindMu.RLock()
|
||||
defer reportingAdminBindMu.RUnlock()
|
||||
return backupHealthReportGenerator
|
||||
}
|
||||
|
||||
// SetReportingAdminEndpointsBinder registers a binder that can replace or decorate
|
||||
// default reporting admin endpoint handlers.
|
||||
func SetReportingAdminEndpointsBinder(binder extensions.BindReportingAdminEndpointsFunc) {
|
||||
|
|
|
|||
|
|
@ -1174,6 +1174,8 @@ type generatedMultiReport struct {
|
|||
Start time.Time
|
||||
End time.Time
|
||||
ResourceCount int
|
||||
DataState string
|
||||
DataMessage string
|
||||
}
|
||||
|
||||
type reportingRequestError struct {
|
||||
|
|
|
|||
42
internal/api/recovery_monitor_reload_test.go
Normal file
42
internal/api/recovery_monitor_reload_test.go
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
package api
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
|
||||
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestBackupHealthRecoveryStoreSurvivesMonitorReplacement(t *testing.T) {
|
||||
persistence := config.NewMultiTenantPersistence(t.TempDir())
|
||||
manager := recoverymanager.New(persistence)
|
||||
router := &Router{config: &config.Config{}, recoveryHandlers: NewRecoveryHandlers(manager)}
|
||||
assertBound := func(m *monitoring.Monitor) {
|
||||
t.Helper()
|
||||
// Observe the dependency without adding a production getter solely for
|
||||
// a wiring test. The live reporting proof also exercises actual writes.
|
||||
bound := reflect.ValueOf(m).Elem().FieldByName("recoveryManager").Pointer()
|
||||
require.Equal(t, reflect.ValueOf(manager).Pointer(), bound)
|
||||
}
|
||||
standalone := newReportingMonitorForTest(t, models.NewState(), nil)
|
||||
router.SetMonitor(standalone)
|
||||
assertBound(standalone)
|
||||
|
||||
firstMonitor := newReportingMonitorForTest(t, models.NewState(), nil)
|
||||
first := newReportingMTMForTest(t, firstMonitor)
|
||||
router.SetMultiTenantMonitor(first)
|
||||
m, err := first.GetMonitor("default")
|
||||
require.NoError(t, err)
|
||||
assertBound(m)
|
||||
|
||||
replacementMonitor := newReportingMonitorForTest(t, models.NewState(), nil)
|
||||
replacement := newReportingMTMForTest(t, replacementMonitor)
|
||||
router.SetMultiTenantMonitor(replacement)
|
||||
m, err = replacement.GetMonitor("default")
|
||||
require.NoError(t, err)
|
||||
assertBound(m)
|
||||
}
|
||||
|
|
@ -329,6 +329,8 @@ func copyReportScheduleRunState(dst *config.ReportSchedule, src config.ReportSch
|
|||
dst.LastError = src.LastError
|
||||
dst.LastDeliveryStatus = src.LastDeliveryStatus
|
||||
dst.LastReportPath = src.LastReportPath
|
||||
dst.LastReportDataState = src.LastReportDataState
|
||||
dst.LastReportDataMessage = src.LastReportDataMessage
|
||||
dst.LastSuccessAt = src.LastSuccessAt
|
||||
dst.RunAttempts = src.RunAttempts
|
||||
dst.RetryAt = src.RetryAt
|
||||
|
|
@ -392,6 +394,8 @@ func (h *ReportingHandlers) executeReportSchedule(ctx context.Context, persisten
|
|||
schedule.RunAttempts++
|
||||
schedule.LastRunStatus = config.ReportScheduleLastRunRunning
|
||||
schedule.LastDeliveryStatus = ""
|
||||
schedule.LastReportDataState = ""
|
||||
schedule.LastReportDataMessage = ""
|
||||
schedule.LastError = ""
|
||||
schedule.RetryAt = nil
|
||||
schedule.LastRunAt = &now
|
||||
|
|
@ -434,6 +438,13 @@ func (h *ReportingHandlers) prepareReportSchedule(ctx context.Context, schedule
|
|||
}
|
||||
switch schedule.Kind {
|
||||
case config.ReportScheduleKindResources:
|
||||
case config.ReportScheduleKindBackupHealth:
|
||||
for _, resource := range schedule.Scope.Resources {
|
||||
kind := reporting.CanonicalResourceType(resource.ResourceType)
|
||||
if kind != "vm" && kind != "system-container" {
|
||||
return schedule, reportScheduleValidationError{code: "invalid_scope", message: "Backup health reports support VMs and containers"}
|
||||
}
|
||||
}
|
||||
case config.ReportScheduleKindVMInventory:
|
||||
if schedule.Format == "" {
|
||||
schedule.Format = string(reporting.DescribeVMInventoryExport().Format)
|
||||
|
|
@ -451,7 +462,7 @@ func (h *ReportingHandlers) prepareReportSchedule(ctx context.Context, schedule
|
|||
}
|
||||
return prepared, h.validateEnabledReportDelivery(ctx, prepared)
|
||||
default:
|
||||
return schedule, reportScheduleValidationError{code: "invalid_kind", message: "Schedule kind must be resources, vm_inventory or patrol_digest"}
|
||||
return schedule, reportScheduleValidationError{code: "invalid_kind", message: "Schedule kind must be resources, backup_health, vm_inventory or patrol_digest"}
|
||||
}
|
||||
if schedule.Format == "" {
|
||||
schedule.Format = config.ReportScheduleFormatPDF
|
||||
|
|
@ -478,6 +489,10 @@ func (h *ReportingHandlers) prepareReportSchedule(ctx context.Context, schedule
|
|||
if err := h.validateReportScheduleScope(ctx, schedule.Scope); err != nil {
|
||||
return schedule, err
|
||||
}
|
||||
} else if schedule.Kind == config.ReportScheduleKindBackupHealth {
|
||||
if err := validateReportScheduleScopeFields(schedule.Scope); err != nil {
|
||||
return schedule, err
|
||||
}
|
||||
}
|
||||
next, err := nextReportScheduleRunAt(schedule, now)
|
||||
if err != nil {
|
||||
|
|
@ -516,6 +531,20 @@ func (h *ReportingHandlers) validateReportScheduleScope(ctx context.Context, sco
|
|||
if len(scope.Resources) > definition.MultiResourceMax {
|
||||
return reportScheduleValidationError{code: "too_many_resources", message: fmt.Sprintf("Maximum %d resources allowed", definition.MultiResourceMax)}
|
||||
}
|
||||
if err := validateReportScheduleScopeFields(scope); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(scope.Tags) > 0 {
|
||||
if _, err := h.resolveReportScheduleResources(ctx, GetOrgID(ctx), scope); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Field validity is shared; scope cardinality and resolution belong to the
|
||||
// report kind. Backup health intentionally supports empty and large estates.
|
||||
func validateReportScheduleScopeFields(scope config.ReportScheduleScope) error {
|
||||
for _, res := range scope.Resources {
|
||||
if !validResourceID.MatchString(res.ResourceID) || len(res.ResourceID) > 128 {
|
||||
return reportScheduleValidationError{code: "invalid_resource_id", message: "Resource IDs must match [a-zA-Z0-9._:-]+ and be 128 characters or fewer"}
|
||||
|
|
@ -529,11 +558,6 @@ func (h *ReportingHandlers) validateReportScheduleScope(ctx context.Context, sco
|
|||
return reportScheduleValidationError{code: "invalid_tag", message: "Tags must be 64 characters or fewer and cannot contain control characters"}
|
||||
}
|
||||
}
|
||||
if len(scope.Tags) > 0 {
|
||||
if _, err := h.resolveReportScheduleResources(ctx, GetOrgID(ctx), scope); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -656,6 +680,8 @@ func (h *ReportingHandlers) runReportSchedule(ctx context.Context, persistence *
|
|||
return result, markReportScheduleFailed(schedule, now, err)
|
||||
}
|
||||
result.path = path
|
||||
schedule.LastReportDataState = report.DataState
|
||||
schedule.LastReportDataMessage = report.DataMessage
|
||||
schedule.LastReportPath = path
|
||||
schedule.LastDeliveryStatus = "saved"
|
||||
|
||||
|
|
@ -676,6 +702,9 @@ func (h *ReportingHandlers) runReportSchedule(ctx context.Context, persistence *
|
|||
}
|
||||
|
||||
func (h *ReportingHandlers) generateScheduledFileReport(ctx context.Context, schedule config.ReportSchedule, now time.Time, manual bool) (generatedMultiReport, error) {
|
||||
if schedule.Kind == config.ReportScheduleKindBackupHealth {
|
||||
return h.generateBackupHealthReport(ctx, schedule, now, manual)
|
||||
}
|
||||
orgID := GetOrgID(ctx)
|
||||
if schedule.Kind == config.ReportScheduleKindVMInventory {
|
||||
snapshot, ok := h.getReportingEnrichmentSnapshot(ctx, orgID)
|
||||
|
|
|
|||
135
internal/api/reporting_backup_health.go
Normal file
135
internal/api/reporting_backup_health.go
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/reporting"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
func backupHealthWorkloads(snapshot reportingEnrichmentSnapshot, scope config.ReportScheduleScope) ([]model.BackupHealthWorkload, error) {
|
||||
selected := map[string]bool{}
|
||||
for _, resource := range scope.Resources {
|
||||
kind := reporting.CanonicalResourceType(resource.ResourceType)
|
||||
if kind != "vm" && kind != "system-container" {
|
||||
return nil, fmt.Errorf("Backup health reports support VMs and containers")
|
||||
}
|
||||
selected[resource.ResourceID] = true
|
||||
}
|
||||
tags := map[string]struct{}{}
|
||||
for _, tag := range scope.Tags {
|
||||
tags[strings.ToLower(strings.TrimSpace(tag))] = struct{}{}
|
||||
}
|
||||
all := len(selected) == 0 && len(tags) == 0
|
||||
workloads := []model.BackupHealthWorkload{}
|
||||
for i := range snapshot.Resources {
|
||||
resource := &snapshot.Resources[i]
|
||||
if resource.Type != unifiedresources.ResourceTypeVM && resource.Type != unifiedresources.ResourceTypeSystemContainer {
|
||||
continue
|
||||
}
|
||||
if !all && !selected[resource.ID] && !resourceHasAnyReportScheduleTag(*resource, tags) {
|
||||
continue
|
||||
}
|
||||
workload := model.BackupHealthWorkload{ID: resource.ID, Name: resource.Name, Type: string(resource.Type)}
|
||||
if workload.Name == "" {
|
||||
workload.Name = resource.ID
|
||||
}
|
||||
if resource.Type == unifiedresources.ResourceTypeVM {
|
||||
view := unifiedresources.NewVMView(resource)
|
||||
workload.Instance, workload.Node, workload.VMID = view.Instance(), view.Node(), view.VMID()
|
||||
} else {
|
||||
view := unifiedresources.NewContainerView(resource)
|
||||
workload.Instance, workload.Node, workload.VMID = view.Instance(), view.Node(), view.VMID()
|
||||
}
|
||||
workloads = append(workloads, workload)
|
||||
}
|
||||
return workloads, nil
|
||||
}
|
||||
|
||||
func (h *ReportingHandlers) generateBackupHealthReport(ctx context.Context, schedule config.ReportSchedule, now time.Time, manual bool) (generatedMultiReport, error) {
|
||||
snapshot, ok := h.getReportingEnrichmentSnapshot(ctx, GetOrgID(ctx))
|
||||
if !ok {
|
||||
return generatedMultiReport{}, fmt.Errorf("Backup report scope is unavailable. Check this organization's infrastructure connections.")
|
||||
}
|
||||
workloads, err := backupHealthWorkloads(snapshot, schedule.Scope)
|
||||
if err != nil {
|
||||
return generatedMultiReport{}, err
|
||||
}
|
||||
windowEnd := now
|
||||
if schedule.RunWindowEnd != nil {
|
||||
windowEnd = *schedule.RunWindowEnd
|
||||
}
|
||||
start, end, err := reportScheduleWindow(schedule, windowEnd, manual)
|
||||
if err != nil {
|
||||
return generatedMultiReport{}, err
|
||||
}
|
||||
location, err := time.LoadLocation(schedule.Cadence.Timezone)
|
||||
if err != nil {
|
||||
return generatedMultiReport{}, err
|
||||
}
|
||||
// Business backup summaries use completed calendar periods. Waiting until
|
||||
// the next day's configured delivery time lets normal collection observe
|
||||
// the cutoff rather than calling every fresh weekly report incomplete.
|
||||
if schedule.Cadence.Type == config.ReportScheduleCadenceWeekly {
|
||||
localEnd := end.In(location)
|
||||
end = time.Date(localEnd.Year(), localEnd.Month(), localEnd.Day(), 0, 0, 0, 0, location).UTC()
|
||||
start = end.In(location).AddDate(0, 0, -7).UTC()
|
||||
}
|
||||
var points []recovery.RecoveryPoint
|
||||
var coverage model.BackupHistoryCoverage
|
||||
historyAvailable := h.recoveryManager != nil
|
||||
if historyAvailable {
|
||||
store, readErr := h.recoveryManager.StoreForOrg(GetOrgID(ctx))
|
||||
if readErr == nil {
|
||||
points, coverage, readErr = store.BackupHealthHistory(ctx, start, end)
|
||||
}
|
||||
if readErr != nil {
|
||||
historyAvailable = false
|
||||
log.Warn().Err(readErr).Msg("Backup report history unavailable")
|
||||
}
|
||||
}
|
||||
summary := recovery.BuildBackupHealth(workloads, points, coverage, start, end, location)
|
||||
notice := "Counts describe recorded attempts only. Incomplete coverage and workloads without attempts need review."
|
||||
if !historyAvailable {
|
||||
summary.HistoryState = "unavailable"
|
||||
for i := range summary.Workloads {
|
||||
summary.Workloads[i].HistoryState = "unavailable"
|
||||
summary.Workloads[i].HistoryReason = "Pulse could not read retained backup execution history."
|
||||
}
|
||||
for i := range summary.Days {
|
||||
summary.Days[i].HistoryState = "unavailable"
|
||||
}
|
||||
notice = "Pulse could not read retained backup execution history. Backup outcomes are unavailable; no successful backups are assumed."
|
||||
} else if summary.HistoryState == "complete" {
|
||||
notice = "Available task history was observed throughout this reporting period. Workloads without recorded attempts still need review."
|
||||
}
|
||||
data := reporting.BackupHealthData{Title: schedule.Name, GeneratedAt: now, Brand: h.resolveReportBranding(ctx).EffectiveBrand(), Summary: summary, HistoryNotice: notice}
|
||||
if schedule.Format != "pdf" && schedule.Format != "csv" {
|
||||
return generatedMultiReport{}, fmt.Errorf("Backup health report format must be PDF or CSV")
|
||||
}
|
||||
generator := resolveBackupHealthReportGenerator()
|
||||
if generator == nil {
|
||||
return generatedMultiReport{}, fmt.Errorf("Backup health report rendering is unavailable. Update to a Pulse Pro runtime with backup health reporting.")
|
||||
}
|
||||
output, contentType, err := generator(data, reporting.ReportFormat(schedule.Format))
|
||||
if err != nil {
|
||||
return generatedMultiReport{}, err
|
||||
}
|
||||
message := "Backup data is incomplete. The report identifies missing evidence."
|
||||
switch summary.HistoryState {
|
||||
case "complete":
|
||||
message = fmt.Sprintf("Backup task history observed throughout the period; %d workloads have no recorded attempts.", summary.MissingWorkloads)
|
||||
case "not_observed":
|
||||
message = "No backup task observation history is available for this scope."
|
||||
case "unavailable":
|
||||
message = "Backup history could not be read. The report shows unavailable outcomes."
|
||||
}
|
||||
return generatedMultiReport{Data: output, ContentType: contentType, Format: reporting.ReportFormat(schedule.Format), Filename: fmt.Sprintf("backup-health-%s.%s", now.UTC().Format("20060102-150405"), schedule.Format), Start: start, End: end, ResourceCount: len(workloads), DataState: summary.HistoryState, DataMessage: message}, nil
|
||||
}
|
||||
205
internal/api/reporting_backup_health_test.go
Normal file
205
internal/api/reporting_backup_health_test.go
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/reporting"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// Public tests exercise the scoped evidence contract and shared delivery.
|
||||
// Real private renderer output is tested in pulse-enterprise and the browser.
|
||||
func installBackupHealthTestRenderer(t *testing.T) {
|
||||
t.Helper()
|
||||
previous := resolveBackupHealthReportGenerator()
|
||||
SetBackupHealthReportGenerator(func(data reporting.BackupHealthData, _ reporting.ReportFormat) ([]byte, string, error) {
|
||||
bytes, err := json.Marshal(data)
|
||||
return bytes, "application/json", err
|
||||
})
|
||||
t.Cleanup(func() { SetBackupHealthReportGenerator(previous) })
|
||||
}
|
||||
func readBackupHealthTestPayload(t *testing.T, data []byte) reporting.BackupHealthData {
|
||||
t.Helper()
|
||||
var payload reporting.BackupHealthData
|
||||
require.NoError(t, json.Unmarshal(data, &payload))
|
||||
return payload
|
||||
}
|
||||
|
||||
func TestBackupHealthRequiresPrivateRenderer(t *testing.T) {
|
||||
previous := resolveBackupHealthReportGenerator()
|
||||
SetBackupHealthReportGenerator(nil)
|
||||
t.Cleanup(func() { SetBackupHealthReportGenerator(previous) })
|
||||
h, _, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
_, err := h.generateBackupHealthReport(ctx, s, time.Now(), true)
|
||||
require.ErrorContains(t, err, "rendering is unavailable")
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleGetReportingCatalog(w, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||
var catalog reporting.ReportingCatalog
|
||||
require.NoError(t, json.Unmarshal(w.Body.Bytes(), &catalog))
|
||||
require.False(t, catalog.BackupHealthAvailable)
|
||||
require.NotContains(t, w.Body.String(), "backupHealthAvailable", "runtimes without the extension retain the existing catalog shape")
|
||||
installBackupHealthTestRenderer(t)
|
||||
w = httptest.NewRecorder()
|
||||
h.HandleGetReportingCatalog(w, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||
require.NoError(t, json.Unmarshal(w.Body.Bytes(), &catalog))
|
||||
require.True(t, catalog.BackupHealthAvailable)
|
||||
}
|
||||
|
||||
func TestBackupHealthWeeklyWindowUsesSevenCompletedLocalDaysAcrossDST(t *testing.T) {
|
||||
installBackupHealthTestRenderer(t)
|
||||
h, _, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
s.Cadence.Timezone = "Europe/London"
|
||||
now := time.Date(2026, 10, 27, 10, 0, 0, 0, time.UTC)
|
||||
report, err := h.generateBackupHealthReport(ctx, s, now, true)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC), report.End)
|
||||
require.Equal(t, 169*time.Hour, report.End.Sub(report.Start))
|
||||
require.Len(t, readBackupHealthTestPayload(t, report.Data).Summary.Days, 7)
|
||||
}
|
||||
|
||||
func TestBackupHealthScheduleDataQualityCannotBeForgedByClient(t *testing.T) {
|
||||
h, p, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
s.Enabled = false
|
||||
s.LastReportDataState = "complete"
|
||||
s.LastReportDataMessage = "untrusted completeness claim"
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleCreateReportSchedule(w, httptest.NewRequest(http.MethodPost, "/api/admin/reports/schedules", encodeScheduleBody(t, s)).WithContext(ctx))
|
||||
require.Equal(t, http.StatusCreated, w.Code, w.Body.String())
|
||||
created := storedReport(t, p)
|
||||
require.Empty(t, created.LastReportDataState)
|
||||
require.Empty(t, created.LastReportDataMessage)
|
||||
}
|
||||
|
||||
func TestBackupHealthScheduleReportsRetainedFailuresBeyondPerformanceCap(t *testing.T) {
|
||||
installBackupHealthTestRenderer(t)
|
||||
h, p, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
h.recoveryManager = recoverymanager.New(config.NewMultiTenantPersistence(t.TempDir()))
|
||||
store, err := h.recoveryManager.StoreForOrg("default")
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
at := now.Add(-48 * time.Hour)
|
||||
point := recovery.RecoveryPoint{ID: "pve-task:fixture-failed", Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther, Mode: recovery.ModeLocal, Outcome: recovery.OutcomeFailed, StartedAt: &at, CompletedAt: &at, Details: map[string]any{"instance": "fixture", "vmid": 100, "status": "ERROR: storage unavailable"}}
|
||||
observation := model.BackupHistoryCollection{Provider: recovery.ProviderProxmoxPVE, Instance: "fixture", ObservedAt: now.Add(-8 * 24 * time.Hour), Complete: true, MaxGap: 9 * 24 * time.Hour}
|
||||
require.NoError(t, store.RecordBackupHistory(ctx, []recovery.RecoveryPoint{point}, []model.BackupHistoryCollection{observation}))
|
||||
observation.ObservedAt = now
|
||||
require.NoError(t, store.RecordBackupHistory(ctx, nil, []model.BackupHistoryCollection{observation}))
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
s, err = h.prepareReportSchedule(ctx, s, now, false)
|
||||
require.NoError(t, err)
|
||||
generated, err := h.generateBackupHealthReport(ctx, s, now, true)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 61, generated.ResourceCount)
|
||||
require.Equal(t, "complete", generated.DataState)
|
||||
require.Equal(t, 0, generated.End.UTC().Hour())
|
||||
require.Equal(t, generated.End.AddDate(0, 0, -7), generated.Start)
|
||||
payload := readBackupHealthTestPayload(t, generated.Data)
|
||||
require.Equal(t, 1, payload.Summary.Counts.Failed)
|
||||
require.Len(t, payload.Summary.Workloads, 61)
|
||||
require.Len(t, payload.Summary.Failures, 1)
|
||||
require.Contains(t, payload.Summary.Failures[0].Detail, "storage unavailable")
|
||||
result, finished := h.runReportSchedule(ctx, p, s, now, true, "")
|
||||
require.Equal(t, "ok", finished.LastRunStatus, finished.LastError)
|
||||
require.Equal(t, "complete", finished.LastReportDataState)
|
||||
require.NotEmpty(t, finished.LastReportDataMessage)
|
||||
data, err := os.ReadFile(result.path)
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, string(data), "fixture-failed")
|
||||
}
|
||||
|
||||
func TestBackupHealthUnavailableHistoryDoesNotLookLikeZeroFailures(t *testing.T) {
|
||||
installBackupHealthTestRenderer(t)
|
||||
h, _, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
report, err := h.generateBackupHealthReport(ctx, s, time.Now().UTC(), true)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "unavailable", report.DataState)
|
||||
payload := readBackupHealthTestPayload(t, report.Data)
|
||||
require.Equal(t, "unavailable", payload.Summary.HistoryState)
|
||||
require.Zero(t, payload.Summary.Counts.Total())
|
||||
require.Equal(t, "unavailable", payload.Summary.Workloads[0].HistoryState)
|
||||
require.Contains(t, payload.HistoryNotice, "no successful backups are assumed")
|
||||
h.SetCommercialLicenseResolver(func(context.Context) *licenseService { return newLicenseService() })
|
||||
_, failed := h.runReportSchedule(ctx, config.NewConfigPersistence(t.TempDir()), s, time.Now(), true, "")
|
||||
require.Equal(t, "failed", failed.LastRunStatus)
|
||||
require.Contains(t, failed.LastError, "entitlement required")
|
||||
}
|
||||
|
||||
func TestBackupHealthScopeCannotIncludeAnotherClientsTasks(t *testing.T) {
|
||||
h, _, ctx, _, _ := inventoryScheduleFixture(t)
|
||||
snapshot, ok := h.getReportingEnrichmentSnapshot(ctx, "default")
|
||||
require.True(t, ok)
|
||||
require.Greater(t, len(snapshot.Resources), 1)
|
||||
snapshot.Resources[1].Tags = []string{"client-b"}
|
||||
scope := config.ReportScheduleScope{Resources: []config.ReportScheduleResource{{ResourceType: "vm", ResourceID: snapshot.Resources[0].ID}}, Tags: []string{"client-b"}}
|
||||
workloads, err := backupHealthWorkloads(snapshot, scope)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, workloads, 2)
|
||||
workloads, err = backupHealthWorkloads(snapshot, config.ReportScheduleScope{Resources: []config.ReportScheduleResource{{ResourceType: "vm", ResourceID: "other-org-resource"}}})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, workloads)
|
||||
_, err = backupHealthWorkloads(snapshot, config.ReportScheduleScope{Resources: []config.ReportScheduleResource{{ResourceType: "host", ResourceID: "host"}}})
|
||||
require.Error(t, err)
|
||||
otherOrg := context.WithValue(ctx, OrgIDContextKey, "unconnected-organization")
|
||||
_, err = h.generateBackupHealthReport(otherOrg, config.ReportSchedule{}, time.Now(), true)
|
||||
require.ErrorContains(t, err, "scope is unavailable")
|
||||
bad := validReportSchedulePayload()
|
||||
bad.ID = "backup-scope"
|
||||
bad.Kind = config.ReportScheduleKindBackupHealth
|
||||
bad.Scope = config.ReportScheduleScope{Tags: []string{"client\nforged"}}
|
||||
_, err = h.prepareReportSchedule(ctx, bad, time.Now(), false)
|
||||
require.ErrorContains(t, err, "control characters")
|
||||
container := snapshot.Resources[0]
|
||||
container.ID, container.Type = "container-1", unifiedresources.ResourceTypeSystemContainer
|
||||
snapshot.Resources = append(snapshot.Resources, container)
|
||||
workloads, err = backupHealthWorkloads(snapshot, config.ReportScheduleScope{Resources: []config.ReportScheduleResource{{ResourceID: container.ID, ResourceType: "system-container"}}})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, workloads, 1)
|
||||
require.Equal(t, "system-container", workloads[0].Type)
|
||||
}
|
||||
|
||||
func TestBackupHealthRetriesKeepWindowAndExposeDataState(t *testing.T) {
|
||||
installBackupHealthTestRenderer(t)
|
||||
h, p, ctx, s, _ := inventoryScheduleFixture(t)
|
||||
s.Kind = config.ReportScheduleKindBackupHealth
|
||||
cfg, attempts := reportSMTP(t, []int{451, 250}, nil)
|
||||
require.NoError(t, p.SaveEmailConfig(cfg))
|
||||
s.Delivery = config.ReportScheduleDelivery{Method: "email", Attach: true, SaveToDisk: true}
|
||||
require.NoError(t, p.SaveReportScheduleStore(config.ReportScheduleStore{Schedules: []config.ReportSchedule{s}}))
|
||||
now := time.Now().UTC()
|
||||
// Drive an explicit manual attempt so this test does not depend on weekday.
|
||||
store := config.ReportScheduleStore{Schedules: []config.ReportSchedule{s}}
|
||||
_, first, err := h.executeReportSchedule(ctx, p, &store, 0, now, true, "", false)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "retrying", first.LastRunStatus)
|
||||
require.Equal(t, "unavailable", first.LastReportDataState)
|
||||
require.NotNil(t, first.RetryAt)
|
||||
firstData, err := os.ReadFile(first.LastReportPath)
|
||||
require.NoError(t, err)
|
||||
restarted := NewReportingHandlers(h.reportMonitorManager(), nil)
|
||||
restarted.runDueReportSchedulesForOrg(ctx, "default", first.RetryAt.Add(time.Second))
|
||||
finished := storedReport(t, p)
|
||||
require.Equal(t, "ok", finished.LastRunStatus, finished.LastError)
|
||||
require.Equal(t, "unavailable", finished.LastReportDataState)
|
||||
require.EqualValues(t, 2, attempts.Load())
|
||||
secondData, err := os.ReadFile(finished.LastReportPath)
|
||||
require.NoError(t, err)
|
||||
firstPayload := readBackupHealthTestPayload(t, firstData)
|
||||
secondPayload := readBackupHealthTestPayload(t, secondData)
|
||||
require.Equal(t, firstPayload.Summary.Start, secondPayload.Summary.Start)
|
||||
require.Equal(t, firstPayload.Summary.End, secondPayload.Summary.End)
|
||||
}
|
||||
|
|
@ -26,6 +26,15 @@ func TestReportingSubjectBoundaryExemptionIsExact(t *testing.T) {
|
|||
name, file, content, goMarker string
|
||||
}{
|
||||
{name: "shared subject pipeline"},
|
||||
{
|
||||
name: "backup adapter exemption is exact", file: "reporting_backup_health_extra.go", content: "package api\n",
|
||||
goMarker: "untracked paid-domain file reporting_backup_health_extra.go",
|
||||
},
|
||||
{
|
||||
name: "backup adapter private import denied", file: "reporting_backup_health.go",
|
||||
content: "package api\nimport _ \"github.com/rcourtman/pulse-go-rewrite/internal/license\"\n",
|
||||
goMarker: "direct internal/license import",
|
||||
},
|
||||
{
|
||||
name: "new reporting implementation", file: "reporting_subject_alerts_extra.go", content: "package api\n",
|
||||
goMarker: "untracked paid-domain file reporting_subject_alerts_extra.go",
|
||||
|
|
|
|||
|
|
@ -16,5 +16,7 @@ func (h *ReportingHandlers) HandleGetReportingCatalog(w http.ResponseWriter, r *
|
|||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(reporting.DescribeReportingCatalog())
|
||||
catalog := reporting.DescribeReportingCatalog()
|
||||
catalog.BackupHealthAvailable = resolveBackupHealthReportGenerator() != nil
|
||||
_ = json.NewEncoder(w).Encode(catalog)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1864,6 +1864,11 @@ func (r *Router) configureMonitorDependencies(m *monitoring.Monitor) {
|
|||
// registry, not just copying the map: a delayed initializer must not reinstall
|
||||
// an old provider after a mock switch or withdrawal has completed.
|
||||
func (r *Router) configureMonitorDependenciesLocked(m *monitoring.Monitor) {
|
||||
// Full configuration reloads replace monitors. Their execution evidence
|
||||
// must still reach the same per-organization store used by recovery reads.
|
||||
if r.recoveryHandlers != nil {
|
||||
m.SetRecoveryManager(r.recoveryHandlers.manager)
|
||||
}
|
||||
// Tenant monitors must inherit the persisted instance-wide notification
|
||||
// settings. System settings are stored globally, so without this a tenant
|
||||
// monitor created after the allowlist was saved (or after a restart)
|
||||
|
|
|
|||
|
|
@ -124,6 +124,9 @@ func (p *MultiTenantStateProvider) ResourceHealthAlertsForTenant(orgID string) [
|
|||
// Used during reload.
|
||||
func (r *Router) SetMultiTenantMonitor(mtm *monitoring.MultiTenantMonitor) {
|
||||
r.mtMonitor = mtm
|
||||
if mtm != nil && r.recoveryHandlers != nil {
|
||||
mtm.SetRecoveryManager(r.recoveryHandlers.manager)
|
||||
}
|
||||
if r.reportingHandlers != nil {
|
||||
r.reportingHandlers.SetMultiTenantMonitor(mtm)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ const (
|
|||
ReportScheduleKindResources = "resources"
|
||||
ReportScheduleKindPatrolDigest = "patrol_digest"
|
||||
ReportScheduleKindVMInventory = "vm_inventory"
|
||||
ReportScheduleKindBackupHealth = "backup_health"
|
||||
|
||||
ReportScheduleDeliveryEmail = "email"
|
||||
ReportScheduleDeliveryDisk = "disk"
|
||||
|
|
@ -43,30 +44,32 @@ type ReportScheduleStore struct {
|
|||
}
|
||||
|
||||
type ReportSchedule struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Cadence ReportScheduleCadence `json:"cadence"`
|
||||
Scope ReportScheduleScope `json:"scope"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Format string `json:"format"`
|
||||
Delivery ReportScheduleDelivery `json:"delivery"`
|
||||
RetentionCount int `json:"retention_count,omitempty"`
|
||||
LastRunAt *time.Time `json:"last_run_at,omitempty"`
|
||||
LastRunStatus string `json:"last_run_status,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
LastDeliveryStatus string `json:"last_delivery_status,omitempty"`
|
||||
LastReportPath string `json:"last_report_path,omitempty"`
|
||||
LastSuccessAt *time.Time `json:"last_success_at,omitempty"`
|
||||
RunAttempts int `json:"run_attempts,omitempty"`
|
||||
RetryAt *time.Time `json:"retry_at,omitempty"`
|
||||
RunWindowEnd *time.Time `json:"run_window_end,omitempty"`
|
||||
RunManual bool `json:"run_manual,omitempty"`
|
||||
NextRunAt *time.Time `json:"next_run_at,omitempty"`
|
||||
LastOccurrenceKey string `json:"last_occurrence_key,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Cadence ReportScheduleCadence `json:"cadence"`
|
||||
Scope ReportScheduleScope `json:"scope"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Format string `json:"format"`
|
||||
Delivery ReportScheduleDelivery `json:"delivery"`
|
||||
RetentionCount int `json:"retention_count,omitempty"`
|
||||
LastRunAt *time.Time `json:"last_run_at,omitempty"`
|
||||
LastRunStatus string `json:"last_run_status,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
LastDeliveryStatus string `json:"last_delivery_status,omitempty"`
|
||||
LastReportPath string `json:"last_report_path,omitempty"`
|
||||
LastReportDataState string `json:"last_report_data_state,omitempty"`
|
||||
LastReportDataMessage string `json:"last_report_data_message,omitempty"`
|
||||
LastSuccessAt *time.Time `json:"last_success_at,omitempty"`
|
||||
RunAttempts int `json:"run_attempts,omitempty"`
|
||||
RetryAt *time.Time `json:"retry_at,omitempty"`
|
||||
RunWindowEnd *time.Time `json:"run_window_end,omitempty"`
|
||||
RunManual bool `json:"run_manual,omitempty"`
|
||||
NextRunAt *time.Time `json:"next_run_at,omitempty"`
|
||||
LastOccurrenceKey string `json:"last_occurrence_key,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
type ReportScheduleCadence struct {
|
||||
|
|
|
|||
131
internal/monitoring/backup_task_history.go
Normal file
131
internal/monitoring/backup_task_history.go
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
package monitoring
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// A short HTTP response alone does not establish that a multi-guest log was
|
||||
// complete or understood. Require contiguous lines from the start, a terminal
|
||||
// marker and at least one attributed guest before qualifying the parent job.
|
||||
func backupGuestLogComplete(job models.BackupTask, lines []proxmox.TaskLogLine, guests []models.BackupTask) bool {
|
||||
if job.EndTime.IsZero() || len(lines) == 0 || len(lines) >= 20000 || len(guests) == 0 {
|
||||
return false
|
||||
}
|
||||
for i, line := range lines {
|
||||
if line.LineNumber != int64(i+1) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
last := strings.TrimSpace(lines[len(lines)-1].Text)
|
||||
return last == "TASK OK" || strings.HasPrefix(last, "TASK ERROR:") || strings.HasPrefix(last, "TASK WARNINGS:")
|
||||
}
|
||||
|
||||
type backupTaskHistoryClient interface {
|
||||
GetBackupTaskHistory(context.Context, time.Time) (proxmox.BackupTaskHistory, error)
|
||||
}
|
||||
|
||||
func (m *Monitor) readBackupTaskHistory(ctx context.Context, instance string, client PVEClientInterface) ([]proxmox.Task, model.BackupHistoryCollection, error) {
|
||||
observation := model.BackupHistoryCollection{Provider: recovery.ProviderProxmoxPVE, Instance: instance}
|
||||
m.mu.RLock()
|
||||
interval := time.Duration(0)
|
||||
if m.config != nil {
|
||||
interval = m.config.BackupPollingInterval
|
||||
}
|
||||
if interval <= 0 && m.config != nil {
|
||||
cycles := m.config.BackupPollingCycles
|
||||
if cycles <= 0 {
|
||||
cycles = 10
|
||||
}
|
||||
interval = m.config.PVEPollingInterval * time.Duration(cycles)
|
||||
}
|
||||
m.mu.RUnlock()
|
||||
if interval < time.Minute {
|
||||
interval = time.Minute
|
||||
}
|
||||
observation.MaxGap = 2*interval + 3*time.Minute
|
||||
if source, ok := client.(backupTaskHistoryClient); ok {
|
||||
since := time.Now().UTC().Add(-90 * 24 * time.Hour)
|
||||
m.mu.RLock()
|
||||
manager, orgID := m.recoveryManager, m.orgID
|
||||
m.mu.RUnlock()
|
||||
if manager != nil {
|
||||
if orgID == "" {
|
||||
orgID = "default"
|
||||
}
|
||||
store, err := manager.StoreForOrg(orgID)
|
||||
if err == nil {
|
||||
if start, err := store.BackupHistoryCollectionStart(ctx, recovery.ProviderProxmoxPVE, instance, time.Now().UTC()); err == nil {
|
||||
since = start
|
||||
}
|
||||
}
|
||||
}
|
||||
history, err := source.GetBackupTaskHistory(ctx, since)
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
observation.Complete = err == nil && len(history.Nodes) > 0
|
||||
for _, node := range history.Nodes {
|
||||
if !node.Complete {
|
||||
observation.Complete = false
|
||||
// A capped backfill can advance the next incremental read, but
|
||||
// must never hide another node's failure or missing authority.
|
||||
if observation.Reason == "" || observation.Reason == "history_truncated" {
|
||||
observation.Reason = node.Reason
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
observation.Reason = "collection_failed"
|
||||
}
|
||||
if len(history.Nodes) == 0 && err == nil {
|
||||
observation.Reason = "no_nodes_observed"
|
||||
}
|
||||
return history.Tasks, observation, err
|
||||
}
|
||||
// Older provider adapters remain useful for live monitoring but cannot
|
||||
// establish report completeness without collection-quality metadata.
|
||||
tasks, err := client.GetBackupTasks(ctx)
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
observation.Reason = "collection_quality_unavailable"
|
||||
return tasks, observation, err
|
||||
}
|
||||
|
||||
func (m *Monitor) persistBackupTaskHistory(ctx context.Context, points []recovery.RecoveryPoint, observation model.BackupHistoryCollection) {
|
||||
m.mu.RLock()
|
||||
manager, orgID := m.recoveryManager, m.orgID
|
||||
m.mu.RUnlock()
|
||||
if manager == nil {
|
||||
return
|
||||
}
|
||||
if orgID == "" {
|
||||
orgID = "default"
|
||||
}
|
||||
store, err := manager.StoreForOrg(orgID)
|
||||
if err == nil {
|
||||
err = store.RecordBackupHistory(ctx, points, []model.BackupHistoryCollection{observation})
|
||||
}
|
||||
if err != nil {
|
||||
log.Error().Err(err).Str("instance", observation.Instance).Msg("Failed to persist backup execution history; report coverage remains unconfirmed")
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Monitor) qualifyBackupGuestTaskDetails(instance string, jobs []models.BackupTask, upids map[string]string, points []recovery.RecoveryPoint) {
|
||||
complete := make(map[string]bool, len(jobs))
|
||||
m.mu.RLock()
|
||||
for _, job := range jobs {
|
||||
entry, ok := m.vzdumpJobTaskCache[instance+"|"+upids[job.ID]]
|
||||
complete["pve-task:"+job.ID] = ok && entry.historyComplete
|
||||
}
|
||||
m.mu.RUnlock()
|
||||
for i := range points {
|
||||
if qualified, ok := complete[points[i].ID]; ok {
|
||||
points[i].Details["guestTaskDetailsComplete"] = qualified
|
||||
}
|
||||
}
|
||||
}
|
||||
100
internal/monitoring/backup_task_history_test.go
Normal file
100
internal/monitoring/backup_task_history_test.go
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
package monitoring
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type historyPVEClient struct {
|
||||
PVEClientInterface
|
||||
history proxmox.BackupTaskHistory
|
||||
}
|
||||
|
||||
func TestBackupHistoryGuestLogRequiresCompleteAttribution(t *testing.T) {
|
||||
job := models.BackupTask{EndTime: time.Now()}
|
||||
guests := []models.BackupTask{{VMID: 100}}
|
||||
lines := []proxmox.TaskLogLine{{LineNumber: 1, Text: "INFO: Starting Backup of VM 100 (qemu)"}, {LineNumber: 2, Text: "TASK OK"}}
|
||||
require.True(t, backupGuestLogComplete(job, lines, guests))
|
||||
require.False(t, backupGuestLogComplete(job, lines, nil))
|
||||
require.False(t, backupGuestLogComplete(job, lines[:1], guests))
|
||||
lines[0].LineNumber = 2
|
||||
require.False(t, backupGuestLogComplete(job, lines, guests))
|
||||
lines[0].LineNumber = 1
|
||||
job.EndTime = time.Time{}
|
||||
require.False(t, backupGuestLogComplete(job, lines, guests))
|
||||
}
|
||||
|
||||
func TestBackupHistoryIncompleteFinishedJobLogIsRetried(t *testing.T) {
|
||||
start := time.Now().Add(-time.Hour)
|
||||
client := &jobBackupTaskClient{
|
||||
task: proxmox.Task{UPID: testJobUPID, Node: "node", Type: "vzdump", Status: "OK", StartTime: start.Unix(), EndTime: start.Add(time.Minute).Unix()},
|
||||
logLines: testLogLines("INFO: Starting Backup of VM 100 (qemu)", "INFO: Finished Backup of VM 100 (00:01:00)"),
|
||||
}
|
||||
m := &Monitor{state: models.NewState()}
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
client.logLines = append(client.logLines, proxmox.TaskLogLine{LineNumber: 3, Text: "TASK OK"})
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
require.Equal(t, 2, client.logRequests)
|
||||
require.True(t, m.vzdumpJobTaskCache["site|"+testJobUPID].historyComplete)
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
require.Equal(t, 2, client.logRequests, "complete terminal logs remain cached")
|
||||
}
|
||||
|
||||
func (c *historyPVEClient) GetBackupTaskHistory(context.Context, time.Time) (proxmox.BackupTaskHistory, error) {
|
||||
return c.history, nil
|
||||
}
|
||||
|
||||
func TestBackupHistoryPollPersistsFailuresAcrossEmptySubsequentPoll(t *testing.T) {
|
||||
manager := recoverymanager.New(config.NewMultiTenantPersistence(t.TempDir()))
|
||||
m := &Monitor{state: models.NewState(), recoveryManager: manager}
|
||||
client := &historyPVEClient{history: proxmox.BackupTaskHistory{Nodes: []proxmox.BackupTaskHistoryNode{{Node: "node", Complete: true}}, Tasks: []proxmox.Task{{UPID: "UPID:node:1:2:3:vzdump:100:root@pam:", Node: "node", ID: "100", Type: "vzdump", StartTime: time.Now().Add(-time.Minute).Unix(), EndTime: time.Now().Unix(), Status: "ERROR: backup storage unavailable"}}}}
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
client.history.Tasks = nil
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
store, err := manager.StoreForOrg("default")
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
points, _, err := store.ListPoints(context.Background(), recovery.ListPointsOptions{Kind: recovery.KindOther})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, points, 1)
|
||||
require.Equal(t, recovery.OutcomeFailed, points[0].Outcome)
|
||||
coverage, err := store.BackupHistoryCoverage(context.Background(), time.Now().Add(-time.Hour), time.Now())
|
||||
require.NoError(t, err)
|
||||
require.Len(t, coverage.Sources, 1)
|
||||
require.True(t, coverage.Sources[0].Complete)
|
||||
}
|
||||
|
||||
func TestBackupHistoryBackfillKeepsLiveTaskPayloadBounded(t *testing.T) {
|
||||
manager := recoverymanager.New(config.NewMultiTenantPersistence(t.TempDir()))
|
||||
m := &Monitor{state: models.NewState(), recoveryManager: manager}
|
||||
client := &historyPVEClient{history: proxmox.BackupTaskHistory{Nodes: []proxmox.BackupTaskHistoryNode{{Node: "node", Complete: true}}}}
|
||||
for i := 0; i < 301; i++ {
|
||||
client.history.Tasks = append(client.history.Tasks, proxmox.Task{UPID: fmt.Sprintf("task-%d", i), Node: "node", ID: "100", Type: "vzdump", StartTime: time.Now().Add(-time.Duration(i+2) * time.Minute).Unix(), EndTime: time.Now().Add(-time.Duration(i+1) * time.Minute).Unix(), Status: "OK"})
|
||||
}
|
||||
m.pollBackupTasks(context.Background(), "site", client)
|
||||
require.Len(t, m.state.GetSnapshot().PVEBackups.BackupTasks, 200)
|
||||
store, err := manager.StoreForOrg("default")
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
_, total, err := store.ListPoints(context.Background(), recovery.ListPointsOptions{Kind: recovery.KindOther})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 301, total, "backfilled evidence survives outside the live API window")
|
||||
}
|
||||
|
||||
func TestBackupHistoryTruncationDoesNotHideAnotherNodesFailure(t *testing.T) {
|
||||
m := &Monitor{}
|
||||
client := &historyPVEClient{history: proxmox.BackupTaskHistory{Nodes: []proxmox.BackupTaskHistoryNode{{Node: "failed", Reason: "collection_failed"}, {Node: "capped", Reason: "history_truncated"}}}}
|
||||
_, collection, err := m.readBackupTaskHistory(context.Background(), "site", client)
|
||||
require.NoError(t, err)
|
||||
require.False(t, collection.Complete)
|
||||
require.Equal(t, "collection_failed", collection.Reason)
|
||||
}
|
||||
|
|
@ -188,6 +188,7 @@ func TestPollBackupTasksSynthesizesJobGuestTasks(t *testing.T) {
|
|||
"INFO: Finished Backup of VM 101 (00:02:00)",
|
||||
"INFO: Starting Backup of VM 102 (lxc)",
|
||||
"INFO: Finished Backup of VM 102 (00:01:00)",
|
||||
"TASK OK",
|
||||
),
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2291,14 +2291,18 @@ func appendPBSBackupsWithinLimit(backups []models.PBSBackup, additions []models.
|
|||
func (m *Monitor) pollBackupTasks(ctx context.Context, instanceName string, client PVEClientInterface) {
|
||||
log.Debug().Str("instance", instanceName).Msg("polling backup tasks")
|
||||
|
||||
tasks, err := client.GetBackupTasks(ctx)
|
||||
tasks, collection, err := m.readBackupTaskHistory(ctx, instanceName, client)
|
||||
if err != nil {
|
||||
m.persistBackupTaskHistory(ctx, nil, collection)
|
||||
monErr := errors.WrapAPIError("get_backup_tasks", instanceName, err, 0)
|
||||
log.Error().Err(monErr).Str("instance", instanceName).Msg("failed to get backup tasks")
|
||||
return
|
||||
}
|
||||
|
||||
var backupTasks []models.BackupTask
|
||||
var liveTasks []models.BackupTask
|
||||
liveByNode := make(map[string]int)
|
||||
liveJobIDs := make(map[string]bool)
|
||||
var jobTasks []models.BackupTask
|
||||
jobUPIDs := make(map[string]string)
|
||||
for _, task := range tasks {
|
||||
|
|
@ -2328,6 +2332,14 @@ func (m *Monitor) pollBackupTasks(ctx context.Context, instanceName string, clie
|
|||
}
|
||||
|
||||
backupTasks = append(backupTasks, backupTask)
|
||||
// Durable reports backfill more history than the live API historically
|
||||
// exposed. Preserve its 200 provider tasks per node rather than inflating
|
||||
// every state/WebSocket payload with the entire retained task history.
|
||||
if liveByNode[task.Node] < 200 {
|
||||
liveTasks = append(liveTasks, backupTask)
|
||||
liveByNode[task.Node]++
|
||||
liveJobIDs[taskID] = true
|
||||
}
|
||||
|
||||
// Scheduled multi-guest backup jobs run under a single UPID with an
|
||||
// empty VMID slot, so the task alone says nothing about which guests
|
||||
|
|
@ -2338,12 +2350,20 @@ func (m *Monitor) pollBackupTasks(ctx context.Context, instanceName string, clie
|
|||
}
|
||||
}
|
||||
|
||||
backupTasks = append(backupTasks, m.synthesizeVzdumpJobGuestTasks(ctx, instanceName, client, jobTasks, jobUPIDs)...)
|
||||
guestTasks := m.synthesizeVzdumpJobGuestTasks(ctx, instanceName, client, jobTasks, jobUPIDs)
|
||||
backupTasks = append(backupTasks, guestTasks...)
|
||||
for _, guest := range guestTasks {
|
||||
parentID := strings.TrimSuffix(guest.ID, fmt.Sprintf("-vm%d", guest.VMID))
|
||||
if liveJobIDs[parentID] {
|
||||
liveTasks = append(liveTasks, guest)
|
||||
}
|
||||
}
|
||||
|
||||
// Update state with new backup tasks for this instance
|
||||
m.state.UpdateBackupTasksForInstance(instanceName, backupTasks)
|
||||
m.state.UpdateBackupTasksForInstance(instanceName, liveTasks)
|
||||
|
||||
// Best-effort ingestion into recovery store (for rollups / unified backups UX).
|
||||
// Retain execution history before qualifying collection coverage. Task rows
|
||||
// are not reconciled against the provider's shrinking inventory window.
|
||||
guestInfo := buildProxmoxGuestInfoIndex(m.backupReadStateForInstance(instanceName))
|
||||
points, evidenceErr := proxmoxrecoverymapper.FromPVEBackupTasksWithEvidence(
|
||||
backupTasks,
|
||||
|
|
@ -2357,14 +2377,16 @@ func (m *Monitor) pollBackupTasks(ctx context.Context, instanceName string, clie
|
|||
Msg("Failed to attach PVE backup-task recovery evidence; preserving existing points")
|
||||
return
|
||||
}
|
||||
m.ingestRecoveryPointsAsync(points)
|
||||
m.qualifyBackupGuestTaskDetails(instanceName, jobTasks, jobUPIDs, points)
|
||||
m.persistBackupTaskHistory(ctx, points, collection)
|
||||
}
|
||||
|
||||
// vzdumpJobTaskCacheEntry caches the per-guest tasks synthesized from one
|
||||
// vzdump job run's log so finished jobs are fetched at most once.
|
||||
type vzdumpJobTaskCacheEntry struct {
|
||||
tasks []models.BackupTask
|
||||
final bool
|
||||
tasks []models.BackupTask
|
||||
final bool
|
||||
historyComplete bool
|
||||
}
|
||||
|
||||
// maxVzdumpJobLogFetchesPerPoll bounds task-log requests per poll cycle so a
|
||||
|
|
@ -2425,13 +2447,15 @@ func (m *Monitor) synthesizeVzdumpJobGuestTasks(ctx context.Context, instanceNam
|
|||
}
|
||||
|
||||
guestTasks := parseVzdumpJobLog(jobTask, upid, logLines)
|
||||
historyComplete := backupGuestLogComplete(jobTask, logLines, guestTasks)
|
||||
m.mu.Lock()
|
||||
if m.vzdumpJobTaskCache == nil {
|
||||
m.vzdumpJobTaskCache = make(map[string]vzdumpJobTaskCacheEntry)
|
||||
}
|
||||
m.vzdumpJobTaskCache[key] = vzdumpJobTaskCacheEntry{
|
||||
tasks: guestTasks,
|
||||
final: !jobTask.EndTime.IsZero(),
|
||||
tasks: guestTasks,
|
||||
final: historyComplete,
|
||||
historyComplete: historyComplete,
|
||||
}
|
||||
m.mu.Unlock()
|
||||
synthesized = append(synthesized, refreshObservedAt(guestTasks, now)...)
|
||||
|
|
|
|||
|
|
@ -34,7 +34,13 @@ func (m *Monitor) listRecoveryRollupsForAlerts(ctx context.Context, kind recover
|
|||
return recovery.BuildRollupsFromPoints(filtered), nil
|
||||
}
|
||||
|
||||
if m == nil || m.recoveryManager == nil {
|
||||
if m == nil {
|
||||
return nil, nil
|
||||
}
|
||||
m.mu.RLock()
|
||||
manager := m.recoveryManager
|
||||
m.mu.RUnlock()
|
||||
if manager == nil {
|
||||
return nil, nil
|
||||
}
|
||||
orgID := m.GetOrgID()
|
||||
|
|
@ -42,7 +48,7 @@ func (m *Monitor) listRecoveryRollupsForAlerts(ctx context.Context, kind recover
|
|||
orgID = "default"
|
||||
}
|
||||
|
||||
store, err := m.recoveryManager.StoreForOrg(orgID)
|
||||
store, err := manager.StoreForOrg(orgID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -112,7 +118,10 @@ func (m *Monitor) checkBackupAlerts(ctx context.Context) {
|
|||
m.alertManager.ClearSystemAlert(alerts.BackupEvaluationAlertType)
|
||||
return
|
||||
}
|
||||
if m.recoveryManager == nil && !mock.IsMockEnabled() {
|
||||
m.mu.RLock()
|
||||
hasRecoveryManager := m.recoveryManager != nil
|
||||
m.mu.RUnlock()
|
||||
if !hasRecoveryManager && !mock.IsMockEnabled() {
|
||||
// Missing dependencies cannot stand in for affirmative read recovery.
|
||||
return
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,6 +4,8 @@ import (
|
|||
"context"
|
||||
"database/sql"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
|
|
@ -14,6 +16,35 @@ import (
|
|||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
func TestBackupHealthRecoveryRebindingCanOverlapActiveReaders(t *testing.T) {
|
||||
m, _ := backupEvaluationMonitor(t)
|
||||
manager := m.recoveryManager
|
||||
store, err := manager.StoreForOrg("default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
var workers sync.WaitGroup
|
||||
workers.Add(2)
|
||||
go func() {
|
||||
defer workers.Done()
|
||||
for range 100 {
|
||||
m.SetRecoveryManager(manager)
|
||||
runtime.Gosched()
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
defer workers.Done()
|
||||
for range 100 {
|
||||
if _, err := m.listBackupRollupsForAlerts(context.Background()); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
}
|
||||
}()
|
||||
workers.Wait()
|
||||
}
|
||||
|
||||
func TestMonitor_ListBackupRollupsForAlerts_ToleratesMalformedPersistedMetadata(t *testing.T) {
|
||||
baseDir := t.TempDir()
|
||||
mtp := config.NewMultiTenantPersistence(baseDir)
|
||||
|
|
|
|||
247
internal/recovery/backup_health.go
Normal file
247
internal/recovery/backup_health.go
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
package recovery
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
)
|
||||
|
||||
// BuildBackupHealth summarizes execution evidence for an authoritative current
|
||||
// workload roster. It never promotes restore artifacts into execution outcomes,
|
||||
// guesses across ambiguous source identities, or treats absence as success.
|
||||
func BuildBackupHealth(workloads []model.BackupHealthWorkload, points []RecoveryPoint, coverage model.BackupHistoryCoverage, from, to time.Time, location *time.Location) model.BackupHealthSummary {
|
||||
if location == nil {
|
||||
location = time.UTC
|
||||
}
|
||||
result := model.BackupHealthSummary{Start: from, End: to, Timezone: location.String(), HistoryState: "complete", Workloads: []model.BackupHealthWorkloadResult{}, Days: []model.BackupHealthDay{}, Failures: []model.BackupHealthFailure{}}
|
||||
byID := map[string]int{}
|
||||
byNative := map[string][]int{}
|
||||
instances := map[string]bool{}
|
||||
for _, workload := range workloads {
|
||||
index := len(result.Workloads)
|
||||
state, reason := backupHistoryState(coverage, workload.Instance, from, to)
|
||||
if workload.Instance == "" || workload.VMID <= 0 {
|
||||
state, reason = "unsupported", "Backup task outcomes are not available for this workload's provider."
|
||||
}
|
||||
result.Workloads = append(result.Workloads, model.BackupHealthWorkloadResult{BackupHealthWorkload: workload, HistoryState: state, HistoryReason: reason})
|
||||
byID[workload.ID] = index
|
||||
key := fmt.Sprintf("%s\x00%d", workload.Instance, workload.VMID)
|
||||
byNative[key] = append(byNative[key], index)
|
||||
instances[workload.Instance] = true
|
||||
}
|
||||
dayIndexes := map[string]int{}
|
||||
local := from.In(location)
|
||||
day := time.Date(local.Year(), local.Month(), local.Day(), 0, 0, 0, 0, location)
|
||||
for day.Before(to) {
|
||||
next := day.AddDate(0, 0, 1)
|
||||
start, end := day, next
|
||||
if start.Before(from) {
|
||||
start = from
|
||||
}
|
||||
if end.After(to) {
|
||||
end = to
|
||||
}
|
||||
state := "not_observed"
|
||||
allComplete, observed := len(workloads) > 0, false
|
||||
for instance := range instances {
|
||||
s, _ := backupHistoryState(coverage, instance, start, end)
|
||||
if s != "not_observed" {
|
||||
observed = true
|
||||
}
|
||||
if s != "complete" {
|
||||
allComplete = false
|
||||
}
|
||||
}
|
||||
if allComplete {
|
||||
state = "complete"
|
||||
} else if observed {
|
||||
state = "partial"
|
||||
}
|
||||
label := day.Format("2006-01-02")
|
||||
dayIndexes[label] = len(result.Days)
|
||||
result.Days = append(result.Days, model.BackupHealthDay{Date: label, HistoryState: state})
|
||||
day = next
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, point := range points {
|
||||
if point.Kind != KindOther || point.Provider != ProviderProxmoxPVE || !strings.HasPrefix(point.ID, "pve-task:") || seen[point.ID] {
|
||||
continue
|
||||
}
|
||||
seen[point.ID] = true
|
||||
if point.StartedAt == nil || !point.StartedAt.Before(to) {
|
||||
continue
|
||||
}
|
||||
instance := recoveryDetailString(point, "instance")
|
||||
if !instances[instance] {
|
||||
continue
|
||||
}
|
||||
// A multi-guest job lacking a complete task log makes client-level
|
||||
// attribution incomplete without disclosing another client's job details.
|
||||
if recoveryDetailInt(point, "vmid") == 0 && (point.CompletedAt == nil || !point.CompletedAt.Before(from)) {
|
||||
qualified, _ := point.Details["guestTaskDetailsComplete"].(bool)
|
||||
if !qualified {
|
||||
at := to.Add(-time.Nanosecond)
|
||||
if point.CompletedAt != nil && point.CompletedAt.Before(to) {
|
||||
at = *point.CompletedAt
|
||||
}
|
||||
if dayIndex, ok := dayIndexes[at.In(location).Format("2006-01-02")]; ok {
|
||||
result.Days[dayIndex].HistoryState = "partial"
|
||||
}
|
||||
for i := range result.Workloads {
|
||||
if result.Workloads[i].Instance == instance {
|
||||
result.Workloads[i].HistoryState = "partial"
|
||||
result.Workloads[i].HistoryReason = "Some multi-guest backup jobs could not be fully attributed to workloads."
|
||||
}
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
index, matched := byID[point.SubjectResourceID]
|
||||
if matched && result.Workloads[index].Instance != instance {
|
||||
continue
|
||||
}
|
||||
if !matched {
|
||||
candidates := byNative[fmt.Sprintf("%s\x00%d", instance, recoveryDetailInt(point, "vmid"))]
|
||||
if len(candidates) != 1 {
|
||||
if len(candidates) > 1 && (point.CompletedAt == nil || !point.CompletedAt.Before(from)) {
|
||||
for _, candidate := range candidates {
|
||||
result.Workloads[candidate].HistoryState = "partial"
|
||||
result.Workloads[candidate].HistoryReason = "Some backup task identities could not be uniquely matched to workloads."
|
||||
}
|
||||
at := to.Add(-time.Nanosecond)
|
||||
if point.CompletedAt != nil && point.CompletedAt.Before(to) {
|
||||
at = *point.CompletedAt
|
||||
}
|
||||
if dayIndex, ok := dayIndexes[at.In(location).Format("2006-01-02")]; ok {
|
||||
result.Days[dayIndex].HistoryState = "partial"
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
index = candidates[0]
|
||||
}
|
||||
row := &result.Workloads[index]
|
||||
if point.Outcome == OutcomeSuccess && point.CompletedAt != nil && point.CompletedAt.Before(to) {
|
||||
if row.LastSuccessAt == nil || point.CompletedAt.After(*row.LastSuccessAt) {
|
||||
value := *point.CompletedAt
|
||||
row.LastSuccessAt = &value
|
||||
}
|
||||
}
|
||||
at := *point.StartedAt
|
||||
outcome := point.Outcome
|
||||
if point.CompletedAt != nil {
|
||||
at = *point.CompletedAt
|
||||
if !at.Before(to) {
|
||||
outcome = OutcomeRunning
|
||||
at = to.Add(-time.Nanosecond)
|
||||
}
|
||||
} else {
|
||||
// An unresolved task from an old poll is not evidence that it is
|
||||
// still running at the report cutoff.
|
||||
outcome = OutcomeUnknown
|
||||
if point.Outcome == OutcomeRunning && point.Evidence != nil && !point.Evidence.IngestedAt.Before(to.Add(-5*time.Minute)) {
|
||||
outcome = OutcomeRunning
|
||||
}
|
||||
if at.Before(from) {
|
||||
at = from
|
||||
}
|
||||
}
|
||||
if at.Before(from) || !at.Before(to) {
|
||||
continue
|
||||
}
|
||||
row.Counts.Add(outcome)
|
||||
result.Counts.Add(outcome)
|
||||
if dayIndex, ok := dayIndexes[at.In(location).Format("2006-01-02")]; ok {
|
||||
result.Days[dayIndex].Counts.Add(outcome)
|
||||
}
|
||||
if outcome == OutcomeFailed || outcome == OutcomeWarning {
|
||||
detail := recoveryDetailString(point, "error")
|
||||
if detail == "" {
|
||||
detail = recoveryDetailString(point, "status")
|
||||
}
|
||||
result.Failures = append(result.Failures, model.BackupHealthFailure{WorkloadID: row.ID, WorkloadName: row.Name, Instance: row.Instance, TaskID: point.ID, Detail: detail, At: at, Outcome: outcome})
|
||||
}
|
||||
}
|
||||
for _, row := range result.Workloads {
|
||||
if row.Counts.Total() == 0 {
|
||||
result.MissingWorkloads++
|
||||
}
|
||||
if row.HistoryState != "complete" {
|
||||
result.HistoryState = "partial"
|
||||
}
|
||||
}
|
||||
if len(result.Workloads) == 0 {
|
||||
result.HistoryState = "not_observed"
|
||||
}
|
||||
if result.Counts.Total() == 0 && len(result.Workloads) > 0 {
|
||||
allMissing := true
|
||||
for _, row := range result.Workloads {
|
||||
if row.HistoryState != "not_observed" && row.HistoryState != "unsupported" {
|
||||
allMissing = false
|
||||
}
|
||||
}
|
||||
if allMissing {
|
||||
result.HistoryState = "not_observed"
|
||||
}
|
||||
}
|
||||
sort.SliceStable(result.Workloads, func(i, j int) bool {
|
||||
a, b := result.Workloads[i], result.Workloads[j]
|
||||
if a.Instance != b.Instance {
|
||||
return a.Instance < b.Instance
|
||||
}
|
||||
if a.Name != b.Name {
|
||||
return a.Name < b.Name
|
||||
}
|
||||
return a.ID < b.ID
|
||||
})
|
||||
sort.SliceStable(result.Failures, func(i, j int) bool { return result.Failures[i].At.After(result.Failures[j].At) })
|
||||
return result
|
||||
}
|
||||
|
||||
func recoveryDetailInt(point RecoveryPoint, key string) int {
|
||||
switch value := point.Details[key].(type) {
|
||||
case int:
|
||||
return value
|
||||
case int64:
|
||||
return int(value)
|
||||
case float64:
|
||||
return int(value)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func backupHistoryState(coverage model.BackupHistoryCoverage, instance string, from, to time.Time) (string, string) {
|
||||
found := false
|
||||
for _, source := range coverage.Sources {
|
||||
if source.Provider != ProviderProxmoxPVE || source.Instance != instance {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
cursor := from
|
||||
for _, interval := range coverage.Intervals {
|
||||
if interval.Provider != source.Provider || interval.Instance != source.Instance || interval.Node != source.Node || interval.End.Before(cursor) {
|
||||
continue
|
||||
}
|
||||
if interval.Start.After(cursor) {
|
||||
break
|
||||
}
|
||||
if interval.End.After(cursor) {
|
||||
cursor = interval.End
|
||||
}
|
||||
}
|
||||
if cursor.Before(to) {
|
||||
reason := "Pulse did not continuously observe this source throughout the reporting period."
|
||||
if source.Reason != "" {
|
||||
reason += " Latest collection: " + strings.ReplaceAll(source.Reason, "_", " ") + "."
|
||||
}
|
||||
return "partial", reason
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return "not_observed", "No durable backup task collection has been recorded for this source."
|
||||
}
|
||||
return "complete", "Available task history was collected throughout this period. Provider retention remains external to Pulse."
|
||||
}
|
||||
99
internal/recovery/backup_health_test.go
Normal file
99
internal/recovery/backup_health_test.go
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
package recovery
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func backupHealthPoint(id, instance string, vmid int, outcome Outcome, at time.Time) RecoveryPoint {
|
||||
return RecoveryPoint{ID: "pve-task:" + id, Provider: ProviderProxmoxPVE, Kind: KindOther, Outcome: outcome, StartedAt: &at, CompletedAt: &at, Details: map[string]any{"instance": instance, "vmid": vmid, "status": "backup failed"}}
|
||||
}
|
||||
|
||||
func TestBackupHealthDailyResultsRemainClientScopedWithOverlappingVMIDs(t *testing.T) {
|
||||
from := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
|
||||
to := from.AddDate(0, 0, 2)
|
||||
coverage := model.BackupHistoryCoverage{Sources: []model.BackupHistorySource{{Provider: ProviderProxmoxPVE, Instance: "client-a"}}, Intervals: []model.BackupHistoryInterval{{Provider: ProviderProxmoxPVE, Instance: "client-a", Start: from, End: to}}}
|
||||
workloads := []model.BackupHealthWorkload{{ID: "a", Name: "Client A", Instance: "client-a", Node: "new-node", VMID: 100}}
|
||||
points := []RecoveryPoint{backupHealthPoint("a-ok", "client-a", 100, OutcomeSuccess, from.Add(time.Hour)), backupHealthPoint("a-failed", "client-a", 100, OutcomeFailed, from.Add(25*time.Hour)), backupHealthPoint("b-failed", "client-b", 100, OutcomeFailed, from.Add(2*time.Hour))}
|
||||
summary := BuildBackupHealth(workloads, points, coverage, from, to, time.UTC)
|
||||
require.Equal(t, 1, summary.Counts.Success)
|
||||
require.Equal(t, 1, summary.Counts.Failed)
|
||||
require.Len(t, summary.Failures, 1)
|
||||
require.Equal(t, "Client A", summary.Failures[0].WorkloadName)
|
||||
require.Equal(t, 1, summary.Days[0].Counts.Success)
|
||||
require.Equal(t, 1, summary.Days[1].Counts.Failed)
|
||||
require.Equal(t, "complete", summary.HistoryState)
|
||||
require.NotNil(t, summary.Workloads[0].LastSuccessAt)
|
||||
}
|
||||
|
||||
func TestBackupHealthMissingHistoryAndZeroRecordedAttemptsAreDistinct(t *testing.T) {
|
||||
from := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
|
||||
to := from.AddDate(0, 0, 1)
|
||||
workloads := []model.BackupHealthWorkload{{ID: "a", Instance: "site", VMID: 100}}
|
||||
summary := BuildBackupHealth(workloads, nil, model.BackupHistoryCoverage{}, from, to, time.UTC)
|
||||
require.Equal(t, "not_observed", summary.HistoryState)
|
||||
require.Equal(t, "not_observed", summary.Days[0].HistoryState)
|
||||
require.Equal(t, 1, summary.MissingWorkloads)
|
||||
coverage := model.BackupHistoryCoverage{Sources: []model.BackupHistorySource{{Provider: ProviderProxmoxPVE, Instance: "site"}}, Intervals: []model.BackupHistoryInterval{{Provider: ProviderProxmoxPVE, Instance: "site", Start: from, End: to}}}
|
||||
summary = BuildBackupHealth(workloads, nil, coverage, from, to, time.UTC)
|
||||
require.Equal(t, "complete", summary.HistoryState)
|
||||
require.Equal(t, 1, summary.MissingWorkloads)
|
||||
require.Zero(t, summary.Counts.Total())
|
||||
artifact := backupHealthPoint("artifact", "site", 100, OutcomeSuccess, from.Add(time.Hour))
|
||||
artifact.Kind = KindBackup
|
||||
summary = BuildBackupHealth(workloads, []RecoveryPoint{artifact}, coverage, from, to, time.UTC)
|
||||
require.Zero(t, summary.Counts.Total())
|
||||
require.Nil(t, summary.Workloads[0].LastSuccessAt)
|
||||
summary = BuildBackupHealth(nil, nil, coverage, from, to, time.UTC)
|
||||
require.Equal(t, "not_observed", summary.HistoryState)
|
||||
require.Equal(t, "not_observed", summary.Days[0].HistoryState, "an unmatched client scope is not an observed zero-failure day")
|
||||
}
|
||||
|
||||
func TestBackupHealthUnattributedJobsCannotProduceCompleteClientReport(t *testing.T) {
|
||||
from := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
|
||||
to := from.AddDate(0, 0, 1)
|
||||
coverage := model.BackupHistoryCoverage{Sources: []model.BackupHistorySource{{Provider: ProviderProxmoxPVE, Instance: "site"}}, Intervals: []model.BackupHistoryInterval{{Provider: ProviderProxmoxPVE, Instance: "site", Start: from, End: to}}}
|
||||
summary := BuildBackupHealth([]model.BackupHealthWorkload{{ID: "a", Instance: "site", VMID: 100}}, []RecoveryPoint{backupHealthPoint("job", "site", 0, OutcomeFailed, from.Add(time.Hour))}, coverage, from, to, time.UTC)
|
||||
require.Equal(t, "partial", summary.HistoryState)
|
||||
require.Equal(t, "partial", summary.Days[0].HistoryState)
|
||||
require.Empty(t, summary.Failures)
|
||||
require.Zero(t, summary.Counts.Total())
|
||||
foreignJob := backupHealthPoint("foreign-job", "another-client", 0, OutcomeFailed, from.Add(time.Hour))
|
||||
summary = BuildBackupHealth([]model.BackupHealthWorkload{{ID: "a", Instance: "site", VMID: 100}}, []RecoveryPoint{foreignJob}, coverage, from, to, time.UTC)
|
||||
require.Equal(t, "complete", summary.HistoryState)
|
||||
require.Equal(t, "complete", summary.Days[0].HistoryState)
|
||||
}
|
||||
|
||||
func TestBackupHealthCalendarDaysRespectDSTAndCutoff(t *testing.T) {
|
||||
location, err := time.LoadLocation("Europe/London")
|
||||
require.NoError(t, err)
|
||||
from := time.Date(2026, 10, 24, 0, 0, 0, 0, location)
|
||||
to := from.AddDate(0, 0, 3)
|
||||
point := backupHealthPoint("cross-cutoff", "site", 100, OutcomeSuccess, to.Add(time.Hour))
|
||||
start := to.Add(-time.Hour)
|
||||
point.StartedAt = &start
|
||||
summary := BuildBackupHealth([]model.BackupHealthWorkload{{ID: "a", Instance: "site", VMID: 100}}, []RecoveryPoint{point}, model.BackupHistoryCoverage{}, from, to, location)
|
||||
require.Len(t, summary.Days, 3)
|
||||
require.Equal(t, "2026-10-25", summary.Days[1].Date)
|
||||
require.Equal(t, 1, summary.Counts.Running)
|
||||
require.Zero(t, summary.Counts.Success)
|
||||
require.Nil(t, summary.Workloads[0].LastSuccessAt)
|
||||
}
|
||||
|
||||
func TestBackupHealthAmbiguousWorkloadIdentityIsIncomplete(t *testing.T) {
|
||||
from := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
|
||||
to := from.AddDate(0, 0, 1)
|
||||
coverage := model.BackupHistoryCoverage{Sources: []model.BackupHistorySource{{Provider: ProviderProxmoxPVE, Instance: "site"}}, Intervals: []model.BackupHistoryInterval{{Provider: ProviderProxmoxPVE, Instance: "site", Start: from, End: to}}}
|
||||
workloads := []model.BackupHealthWorkload{{ID: "a", Instance: "site", VMID: 100}, {ID: "b", Instance: "site", VMID: 100}}
|
||||
summary := BuildBackupHealth(workloads, []RecoveryPoint{backupHealthPoint("ambiguous", "site", 100, OutcomeFailed, from.Add(time.Hour))}, coverage, from, to, time.UTC)
|
||||
require.Equal(t, "partial", summary.HistoryState)
|
||||
require.Equal(t, "partial", summary.Days[0].HistoryState)
|
||||
require.Zero(t, summary.Counts.Total())
|
||||
require.Empty(t, summary.Failures)
|
||||
for _, row := range summary.Workloads {
|
||||
require.Contains(t, row.HistoryReason, "uniquely matched")
|
||||
}
|
||||
}
|
||||
|
|
@ -162,6 +162,9 @@ func outcomeFromTaskStatus(status string) recovery.Outcome {
|
|||
case "":
|
||||
return recovery.OutcomeUnknown
|
||||
default:
|
||||
if strings.HasPrefix(s, "warnings:") || strings.HasPrefix(s, "warning:") {
|
||||
return recovery.OutcomeWarning
|
||||
}
|
||||
if strings.Contains(s, "fail") || strings.Contains(s, "error") {
|
||||
return recovery.OutcomeFailed
|
||||
}
|
||||
|
|
|
|||
|
|
@ -220,6 +220,15 @@ func TestFromPVEStorageBackupsWithEvidenceAttachesCanonicalProviderScope(t *test
|
|||
}
|
||||
}
|
||||
|
||||
func TestFromPVEBackupTasksWarningExitStatus(t *testing.T) {
|
||||
for _, status := range []string{"warning", "WARNINGS: 1", "WARNING: backup completed with warnings"} {
|
||||
points := FromPVEBackupTasks([]models.BackupTask{{ID: "job", Instance: "site", Status: status}}, nil)
|
||||
if len(points) != 1 || points[0].Outcome != recovery.OutcomeWarning {
|
||||
t.Fatalf("status %q: expected warning execution evidence, got %+v", status, points)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFromPVEBackupTasks_Empty(t *testing.T) {
|
||||
result := FromPVEBackupTasks(nil, nil)
|
||||
if result != nil {
|
||||
|
|
|
|||
100
internal/recovery/model/backup_history.go
Normal file
100
internal/recovery/model/backup_history.go
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
package model
|
||||
|
||||
import "time"
|
||||
|
||||
// BackupHistoryCollection is a completed source read, including unsuccessful
|
||||
// reads. Collection quality is distinct from the outcome of a backup job.
|
||||
type BackupHistoryCollection struct {
|
||||
Provider Provider
|
||||
Instance string
|
||||
Node string
|
||||
ObservedAt time.Time
|
||||
Complete bool
|
||||
Reason string
|
||||
MaxGap time.Duration
|
||||
}
|
||||
|
||||
type BackupHistorySource struct {
|
||||
Provider Provider
|
||||
Instance string
|
||||
Node string
|
||||
FirstObservedAt time.Time
|
||||
LastObservedAt time.Time
|
||||
Complete bool
|
||||
Reason string
|
||||
}
|
||||
|
||||
type BackupHistoryInterval struct {
|
||||
Provider Provider
|
||||
Instance string
|
||||
Node string
|
||||
Start time.Time
|
||||
End time.Time
|
||||
}
|
||||
|
||||
// BackupHistoryCoverage describes Pulse's retained observation coverage. Source
|
||||
// task enumeration never backdates this coverage: provider retention is external
|
||||
// and a newly connected server cannot prove a previously unobserved period.
|
||||
type BackupHistoryCoverage struct {
|
||||
Sources []BackupHistorySource
|
||||
Intervals []BackupHistoryInterval
|
||||
}
|
||||
|
||||
type BackupHealthWorkload struct {
|
||||
ID, Name, Instance, Node, Type string
|
||||
VMID int
|
||||
}
|
||||
|
||||
type BackupOutcomeCounts struct {
|
||||
Success, Failed, Warning, Running, Unknown int
|
||||
}
|
||||
|
||||
func (c *BackupOutcomeCounts) Add(outcome Outcome) {
|
||||
switch outcome {
|
||||
case OutcomeSuccess:
|
||||
c.Success++
|
||||
case OutcomeFailed:
|
||||
c.Failed++
|
||||
case OutcomeWarning:
|
||||
c.Warning++
|
||||
case OutcomeRunning:
|
||||
c.Running++
|
||||
default:
|
||||
c.Unknown++
|
||||
}
|
||||
}
|
||||
|
||||
func (c BackupOutcomeCounts) Total() int {
|
||||
return c.Success + c.Failed + c.Warning + c.Running + c.Unknown
|
||||
}
|
||||
|
||||
type BackupHealthWorkloadResult struct {
|
||||
BackupHealthWorkload
|
||||
Counts BackupOutcomeCounts
|
||||
LastSuccessAt *time.Time
|
||||
HistoryState string
|
||||
HistoryReason string
|
||||
}
|
||||
|
||||
type BackupHealthDay struct {
|
||||
Date string
|
||||
Counts BackupOutcomeCounts
|
||||
HistoryState string
|
||||
}
|
||||
|
||||
type BackupHealthFailure struct {
|
||||
WorkloadID, WorkloadName, Instance, TaskID, Detail string
|
||||
At time.Time
|
||||
Outcome Outcome
|
||||
}
|
||||
|
||||
type BackupHealthSummary struct {
|
||||
Start, End time.Time
|
||||
Timezone string
|
||||
Counts BackupOutcomeCounts
|
||||
Workloads []BackupHealthWorkloadResult
|
||||
Days []BackupHealthDay
|
||||
Failures []BackupHealthFailure
|
||||
HistoryState string
|
||||
MissingWorkloads int
|
||||
}
|
||||
285
internal/recovery/store/backup_history.go
Normal file
285
internal/recovery/store/backup_history.go
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
)
|
||||
|
||||
const backupHistorySchema = `
|
||||
CREATE TABLE IF NOT EXISTS backup_history_sources (
|
||||
provider TEXT NOT NULL, instance TEXT NOT NULL, node TEXT NOT NULL,
|
||||
first_ms INTEGER NOT NULL, last_ms INTEGER NOT NULL,
|
||||
complete INTEGER NOT NULL, reason TEXT NOT NULL, range_start_ms INTEGER NOT NULL,
|
||||
PRIMARY KEY(provider, instance, node)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS backup_history_intervals (
|
||||
provider TEXT NOT NULL, instance TEXT NOT NULL, node TEXT NOT NULL,
|
||||
start_ms INTEGER NOT NULL, end_ms INTEGER NOT NULL,
|
||||
PRIMARY KEY(provider, instance, node, start_ms)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_backup_history_interval_end ON backup_history_intervals(end_ms);
|
||||
`
|
||||
|
||||
// RecordBackupHistory commits task evidence before recording collection success.
|
||||
// A crash or failure between these writes can only reduce declared coverage.
|
||||
// Neither an in-memory queue nor an inventory reconciliation owns this history.
|
||||
func (s *Store) RecordBackupHistory(ctx context.Context, points []recovery.RecoveryPoint, collections []model.BackupHistoryCollection) error {
|
||||
if err := s.ensureInitialized(); err != nil {
|
||||
return err
|
||||
}
|
||||
s.backupHistoryMu.Lock()
|
||||
defer s.backupHistoryMu.Unlock()
|
||||
if len(collections) != 1 {
|
||||
return fmt.Errorf("backup history requires exactly one source collection")
|
||||
}
|
||||
for _, c := range collections {
|
||||
if c.Provider == "" || strings.TrimSpace(c.Instance) == "" || c.ObservedAt.IsZero() || c.MaxGap <= 0 {
|
||||
return fmt.Errorf("invalid backup history collection")
|
||||
}
|
||||
}
|
||||
for _, c := range collections {
|
||||
var last int64
|
||||
err := s.db.QueryRowContext(ctx, "SELECT last_ms FROM backup_history_sources WHERE provider=? AND instance=? AND node=?", c.Provider, c.Instance, c.Node).Scan(&last)
|
||||
if err != nil && err != sql.ErrNoRows {
|
||||
return err
|
||||
}
|
||||
if err == nil && c.ObservedAt.UnixMilli() <= last {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
// A repeated running observation must not erase a previously committed
|
||||
// terminal outcome. Terminal corrections remain eligible on later polls.
|
||||
accepted := make([]recovery.RecoveryPoint, 0, len(points))
|
||||
for _, point := range points {
|
||||
instance, _ := point.Details["instance"].(string)
|
||||
if point.Provider != collections[0].Provider || instance != collections[0].Instance {
|
||||
return fmt.Errorf("backup task belongs to another source collection")
|
||||
}
|
||||
if !strings.HasPrefix(point.ID, "pve-task:") && !strings.HasPrefix(point.ID, "pbs-task:") {
|
||||
return fmt.Errorf("backup history accepts execution evidence only")
|
||||
}
|
||||
if point.Kind != recovery.KindOther {
|
||||
return fmt.Errorf("backup execution is not a restore artifact")
|
||||
}
|
||||
if point.CompletedAt == nil {
|
||||
var completed sql.NullInt64
|
||||
err := s.db.QueryRowContext(ctx, "SELECT completed_at_ms FROM recovery_points WHERE id = ?", point.ID).Scan(&completed)
|
||||
if err != nil && err != sql.ErrNoRows {
|
||||
return err
|
||||
}
|
||||
if completed.Valid {
|
||||
continue
|
||||
}
|
||||
}
|
||||
accepted = append(accepted, point)
|
||||
}
|
||||
if err := s.UpsertPoints(ctx, accepted); err != nil {
|
||||
return err
|
||||
}
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
for _, c := range collections {
|
||||
if c.Complete {
|
||||
c.Reason = ""
|
||||
}
|
||||
at := c.ObservedAt.UTC().UnixMilli()
|
||||
var first, last, previousRange int64
|
||||
var complete bool
|
||||
err := tx.QueryRowContext(ctx, `SELECT first_ms,last_ms,complete,range_start_ms FROM backup_history_sources WHERE provider=? AND instance=? AND node=?`, c.Provider, c.Instance, c.Node).Scan(&first, &last, &complete, &previousRange)
|
||||
if err != nil && err != sql.ErrNoRows {
|
||||
return err
|
||||
}
|
||||
if err == nil && at <= last {
|
||||
continue
|
||||
}
|
||||
if first == 0 {
|
||||
first = at
|
||||
}
|
||||
rangeStart := at
|
||||
if c.Complete {
|
||||
if complete && last > 0 && at-last <= c.MaxGap.Milliseconds() {
|
||||
rangeStart = previousRange
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO backup_history_intervals(provider,instance,node,start_ms,end_ms) VALUES(?,?,?,?,?) ON CONFLICT(provider,instance,node,start_ms) DO UPDATE SET end_ms=excluded.end_ms`, c.Provider, c.Instance, c.Node, rangeStart, at); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO backup_history_sources(provider,instance,node,first_ms,last_ms,complete,reason,range_start_ms) VALUES(?,?,?,?,?,?,?,?) ON CONFLICT(provider,instance,node) DO UPDATE SET last_ms=excluded.last_ms,complete=excluded.complete,reason=excluded.reason,range_start_ms=excluded.range_start_ms`, c.Provider, c.Instance, c.Node, first, at, c.Complete, c.Reason, rangeStart); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
cutoff := time.Now().UTC().Add(-s.retention).UnixMilli()
|
||||
if _, err := tx.ExecContext(ctx, "DELETE FROM backup_history_intervals WHERE end_ms < ?", cutoff); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func (s *Store) BackupHistoryCoverage(ctx context.Context, from, to time.Time) (model.BackupHistoryCoverage, error) {
|
||||
if err := s.ensureInitialized(); err != nil {
|
||||
return model.BackupHistoryCoverage{}, err
|
||||
}
|
||||
return readBackupHistoryCoverage(ctx, s.db, from, to)
|
||||
}
|
||||
|
||||
type backupHistoryReader interface {
|
||||
QueryContext(context.Context, string, ...any) (*sql.Rows, error)
|
||||
}
|
||||
|
||||
// BackupHistoryCollectionStart bounds later reads to the durable collection
|
||||
// watermark, with a day of overlap for delayed task publication. Known pending
|
||||
// jobs keep their original start in range so a long backup can still finish.
|
||||
// This never changes observation coverage or backdates a successful collection.
|
||||
func (s *Store) BackupHistoryCollectionStart(ctx context.Context, provider recovery.Provider, instance string, now time.Time) (time.Time, error) {
|
||||
floor := now.UTC().Add(-s.retention)
|
||||
if err := s.ensureInitialized(); err != nil {
|
||||
return floor, err
|
||||
}
|
||||
var last int64
|
||||
var complete bool
|
||||
var reason string
|
||||
err := s.db.QueryRowContext(ctx, `SELECT last_ms,complete,reason FROM backup_history_sources WHERE provider=? AND instance=? AND node=''`, provider, instance).Scan(&last, &complete, &reason)
|
||||
if err != nil && err != sql.ErrNoRows {
|
||||
return floor, err
|
||||
}
|
||||
if !complete && reason != "history_truncated" {
|
||||
var qualified sql.NullInt64
|
||||
if err := s.db.QueryRowContext(ctx, `SELECT MAX(end_ms) FROM backup_history_intervals WHERE provider=? AND instance=? AND node=''`, provider, instance).Scan(&qualified); err != nil {
|
||||
return floor, err
|
||||
}
|
||||
last = qualified.Int64
|
||||
}
|
||||
start := floor
|
||||
if last > 0 {
|
||||
candidate := time.UnixMilli(last).UTC().Add(-24 * time.Hour)
|
||||
if candidate.After(start) {
|
||||
start = candidate
|
||||
}
|
||||
}
|
||||
var pending sql.NullInt64
|
||||
err = s.db.QueryRowContext(ctx, `SELECT MIN(started_at_ms) FROM recovery_points WHERE provider=? AND kind=? AND id LIKE 'pve-task:%' AND completed_at_ms IS NULL AND started_at_ms>=? AND json_extract(details_json,'$.instance')=?`, provider, recovery.KindOther, floor.UnixMilli(), instance).Scan(&pending)
|
||||
if err != nil {
|
||||
return floor, err
|
||||
}
|
||||
if pending.Valid && pending.Int64 < start.UnixMilli() {
|
||||
start = time.UnixMilli(pending.Int64).UTC()
|
||||
}
|
||||
return start, nil
|
||||
}
|
||||
|
||||
func readBackupHistoryCoverage(ctx context.Context, reader backupHistoryReader, from, to time.Time) (model.BackupHistoryCoverage, error) {
|
||||
coverage := model.BackupHistoryCoverage{Sources: []model.BackupHistorySource{}, Intervals: []model.BackupHistoryInterval{}}
|
||||
rows, err := reader.QueryContext(ctx, `SELECT provider,instance,node,first_ms,last_ms,complete,reason FROM backup_history_sources ORDER BY provider,instance,node`)
|
||||
if err != nil {
|
||||
return coverage, err
|
||||
}
|
||||
for rows.Next() {
|
||||
var source model.BackupHistorySource
|
||||
var first, last int64
|
||||
if err := rows.Scan(&source.Provider, &source.Instance, &source.Node, &first, &last, &source.Complete, &source.Reason); err != nil {
|
||||
rows.Close()
|
||||
return coverage, err
|
||||
}
|
||||
source.FirstObservedAt, source.LastObservedAt = time.UnixMilli(first).UTC(), time.UnixMilli(last).UTC()
|
||||
coverage.Sources = append(coverage.Sources, source)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return coverage, err
|
||||
}
|
||||
rows, err = reader.QueryContext(ctx, `SELECT provider,instance,node,start_ms,end_ms FROM backup_history_intervals WHERE end_ms>=? AND start_ms<=? ORDER BY provider,instance,node,start_ms`, from.UnixMilli(), to.UnixMilli())
|
||||
if err != nil {
|
||||
return coverage, err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var interval model.BackupHistoryInterval
|
||||
var start, end int64
|
||||
if err := rows.Scan(&interval.Provider, &interval.Instance, &interval.Node, &start, &end); err != nil {
|
||||
return coverage, err
|
||||
}
|
||||
interval.Start, interval.End = time.UnixMilli(start).UTC(), time.UnixMilli(end).UTC()
|
||||
coverage.Intervals = append(coverage.Intervals, interval)
|
||||
}
|
||||
return coverage, rows.Err()
|
||||
}
|
||||
|
||||
// BackupHealthHistory reads outcomes and coverage from one SQLite snapshot.
|
||||
// A concurrent poll cannot qualify coverage over an older set of task rows.
|
||||
func (s *Store) BackupHealthHistory(ctx context.Context, from, to time.Time) ([]recovery.RecoveryPoint, model.BackupHistoryCoverage, error) {
|
||||
var coverage model.BackupHistoryCoverage
|
||||
if err := s.ensureInitialized(); err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
tx, err := s.db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
|
||||
if err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
rows, err := tx.QueryContext(ctx, `SELECT id,outcome,started_at_ms,completed_at_ms,subject_resource_id,details_json,evidence_json FROM recovery_points WHERE provider=? AND kind=? AND id LIKE 'pve-task:%' AND started_at_ms<? AND COALESCE(completed_at_ms,started_at_ms,updated_at_ms)>=? ORDER BY id`, recovery.ProviderProxmoxPVE, recovery.KindOther, to.UnixMilli(), time.Now().UTC().Add(-s.retention).UnixMilli())
|
||||
if err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
points := []recovery.RecoveryPoint{}
|
||||
for rows.Next() {
|
||||
if len(points) >= 250000 {
|
||||
rows.Close()
|
||||
return nil, coverage, fmt.Errorf("backup execution history exceeds report query limit")
|
||||
}
|
||||
point := recovery.RecoveryPoint{Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther}
|
||||
var start, end sql.NullInt64
|
||||
var subject, details, evidence sql.NullString
|
||||
if err := rows.Scan(&point.ID, &point.Outcome, &start, &end, &subject, &details, &evidence); err != nil {
|
||||
rows.Close()
|
||||
return nil, coverage, err
|
||||
}
|
||||
if start.Valid {
|
||||
value := time.UnixMilli(start.Int64).UTC()
|
||||
point.StartedAt = &value
|
||||
}
|
||||
if end.Valid {
|
||||
value := time.UnixMilli(end.Int64).UTC()
|
||||
point.CompletedAt = &value
|
||||
}
|
||||
point.SubjectResourceID = subject.String
|
||||
if details.Valid {
|
||||
if err := json.Unmarshal([]byte(details.String), &point.Details); err != nil {
|
||||
rows.Close()
|
||||
return nil, coverage, err
|
||||
}
|
||||
}
|
||||
if evidence.Valid {
|
||||
point.Evidence = &operationaltrust.EvidenceEnvelope{}
|
||||
if err := json.Unmarshal([]byte(evidence.String), point.Evidence); err != nil {
|
||||
rows.Close()
|
||||
return nil, coverage, err
|
||||
}
|
||||
}
|
||||
points = append(points, point)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
coverage, err = readBackupHistoryCoverage(ctx, tx, from, to)
|
||||
if err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return nil, coverage, err
|
||||
}
|
||||
return points, coverage, nil
|
||||
}
|
||||
128
internal/recovery/store/backup_history_test.go
Normal file
128
internal/recovery/store/backup_history_test.go
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestBackupHistorySurvivesRestartAndShrinkingSourceInventory(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
path := filepath.Join(t.TempDir(), "recovery.db")
|
||||
s, err := Open(path)
|
||||
require.NoError(t, err)
|
||||
at := time.Now().UTC().Add(-time.Hour).Truncate(time.Millisecond)
|
||||
point := recovery.RecoveryPoint{ID: "pve-task:site-job", Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther, Mode: recovery.ModeLocal, Outcome: recovery.OutcomeFailed, StartedAt: &at, CompletedAt: &at, Details: map[string]any{"instance": "site"}}
|
||||
observation := model.BackupHistoryCollection{Provider: recovery.ProviderProxmoxPVE, Instance: "site", ObservedAt: at, Complete: true, MaxGap: 5 * time.Minute}
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, []recovery.RecoveryPoint{point}, []model.BackupHistoryCollection{observation}))
|
||||
point.Outcome, point.CompletedAt = recovery.OutcomeRunning, nil
|
||||
observation.ObservedAt = at.Add(time.Minute)
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, []recovery.RecoveryPoint{point}, []model.BackupHistoryCollection{observation}))
|
||||
observation.ObservedAt = at.Add(2 * time.Minute)
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, nil, []model.BackupHistoryCollection{observation}))
|
||||
require.NoError(t, s.Close())
|
||||
s, err = Open(path)
|
||||
require.NoError(t, err)
|
||||
defer s.Close()
|
||||
points, total, err := s.ListPoints(ctx, recovery.ListPointsOptions{Kind: recovery.KindOther})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, total)
|
||||
require.Equal(t, recovery.OutcomeFailed, points[0].Outcome)
|
||||
coverage, err := s.BackupHistoryCoverage(ctx, at, at.Add(time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Len(t, coverage.Intervals, 1)
|
||||
require.Equal(t, at, coverage.Intervals[0].Start)
|
||||
require.Equal(t, at.Add(2*time.Minute), coverage.Intervals[0].End)
|
||||
}
|
||||
|
||||
func TestBackupHistoryCoverageDoesNotBridgeFailureDowntimeOrOldObservations(t *testing.T) {
|
||||
s, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
|
||||
require.NoError(t, err)
|
||||
defer s.Close()
|
||||
ctx := context.Background()
|
||||
at := time.Now().UTC().Add(-time.Hour).Truncate(time.Millisecond)
|
||||
for _, step := range []struct {
|
||||
minutes int
|
||||
complete bool
|
||||
}{{0, true}, {1, true}, {2, false}, {3, true}, {4, true}, {30, true}, {2, false}} {
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, nil, []model.BackupHistoryCollection{{Provider: recovery.ProviderProxmoxPVE, Instance: "site", ObservedAt: at.Add(time.Duration(step.minutes) * time.Minute), Complete: step.complete, MaxGap: 5 * time.Minute, Reason: "collection_failed"}}))
|
||||
}
|
||||
coverage, err := s.BackupHistoryCoverage(ctx, at, at.Add(time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Len(t, coverage.Intervals, 3)
|
||||
require.True(t, coverage.Sources[0].Complete)
|
||||
require.Equal(t, at.Add(30*time.Minute), coverage.Sources[0].LastObservedAt)
|
||||
require.Equal(t, at.Add(time.Minute), coverage.Intervals[0].End)
|
||||
require.Equal(t, at.Add(3*time.Minute), coverage.Intervals[1].Start)
|
||||
}
|
||||
|
||||
func TestBackupHistoryCoverageCannotCommitBeforeExecutionEvidence(t *testing.T) {
|
||||
s, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
|
||||
require.NoError(t, err)
|
||||
defer s.Close()
|
||||
_, err = s.db.Exec(`CREATE TRIGGER refuse_task BEFORE INSERT ON recovery_points BEGIN SELECT RAISE(ABORT,'write failed'); END`)
|
||||
require.NoError(t, err)
|
||||
at := time.Now().UTC()
|
||||
err = s.RecordBackupHistory(context.Background(), []recovery.RecoveryPoint{{ID: "pve-task:site-job", Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther, Mode: recovery.ModeLocal, Outcome: recovery.OutcomeFailed, StartedAt: &at, Details: map[string]any{"instance": "site"}}}, []model.BackupHistoryCollection{{Provider: recovery.ProviderProxmoxPVE, Instance: "site", ObservedAt: at, Complete: true, MaxGap: time.Minute}})
|
||||
require.ErrorContains(t, err, "write failed")
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
coverage, err := s.BackupHistoryCoverage(ctx, at.Add(-time.Hour), at)
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, coverage.Sources)
|
||||
require.Empty(t, coverage.Intervals)
|
||||
}
|
||||
|
||||
func TestBackupHistoryIncrementalCollectionRetainsLongRunningJobs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
path := filepath.Join(t.TempDir(), "recovery.db")
|
||||
s, err := Open(path)
|
||||
require.NoError(t, err)
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
start := now.Add(-72 * time.Hour)
|
||||
point := recovery.RecoveryPoint{ID: "pve-task:long", Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther, Mode: recovery.ModeLocal, Outcome: recovery.OutcomeRunning, StartedAt: &start, Details: map[string]any{"instance": "site"}}
|
||||
collection := model.BackupHistoryCollection{Provider: recovery.ProviderProxmoxPVE, Instance: "site", Complete: true, ObservedAt: now.Add(-time.Minute), MaxGap: 5 * time.Minute}
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, []recovery.RecoveryPoint{point}, []model.BackupHistoryCollection{collection}))
|
||||
require.NoError(t, s.Close())
|
||||
s, err = Open(path)
|
||||
require.NoError(t, err)
|
||||
defer s.Close()
|
||||
since, err := s.BackupHistoryCollectionStart(ctx, recovery.ProviderProxmoxPVE, "site", now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, start, since, "restart must still read a backup that began before the overlap window")
|
||||
finished := now.Add(-time.Second)
|
||||
point.CompletedAt, point.Outcome = &finished, recovery.OutcomeSuccess
|
||||
collection.ObservedAt = now
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, []recovery.RecoveryPoint{point}, []model.BackupHistoryCollection{collection}))
|
||||
since, err = s.BackupHistoryCollectionStart(ctx, recovery.ProviderProxmoxPVE, "site", now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, now.Add(-24*time.Hour), since)
|
||||
collection.ObservedAt, collection.Complete, collection.Reason = now.Add(time.Hour), false, "collection_failed"
|
||||
require.NoError(t, s.RecordBackupHistory(ctx, nil, []model.BackupHistoryCollection{collection}))
|
||||
since, err = s.BackupHistoryCollectionStart(ctx, recovery.ProviderProxmoxPVE, "site", now.Add(time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, now.Add(-24*time.Hour), since, "failed collections cannot advance the read watermark")
|
||||
}
|
||||
|
||||
func TestBackupHistoryCappedBackfillDoesNotBackdateCoverage(t *testing.T) {
|
||||
s, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
|
||||
require.NoError(t, err)
|
||||
defer s.Close()
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
c := model.BackupHistoryCollection{Provider: recovery.ProviderProxmoxPVE, Instance: "site", ObservedAt: now, Reason: "history_truncated", MaxGap: 5 * time.Minute}
|
||||
require.NoError(t, s.RecordBackupHistory(context.Background(), nil, []model.BackupHistoryCollection{c}))
|
||||
since, err := s.BackupHistoryCollectionStart(context.Background(), recovery.ProviderProxmoxPVE, "site", now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, now.Add(-24*time.Hour), since)
|
||||
c.Complete, c.ObservedAt = true, now.Add(time.Minute)
|
||||
require.NoError(t, s.RecordBackupHistory(context.Background(), nil, []model.BackupHistoryCollection{c}))
|
||||
coverage, err := s.BackupHistoryCoverage(context.Background(), now.Add(-30*24*time.Hour), c.ObservedAt)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, coverage.Intervals, 1)
|
||||
require.Equal(t, c.ObservedAt, coverage.Intervals[0].Start)
|
||||
}
|
||||
|
|
@ -36,8 +36,9 @@ type Store struct {
|
|||
db *sql.DB
|
||||
retention time.Duration
|
||||
|
||||
pruneMu sync.Mutex
|
||||
lastPrune time.Time
|
||||
pruneMu sync.Mutex
|
||||
lastPrune time.Time
|
||||
backupHistoryMu sync.Mutex
|
||||
}
|
||||
|
||||
type historicalSubjectIdentity struct {
|
||||
|
|
@ -586,7 +587,7 @@ func (s *Store) initSchema() error {
|
|||
return err
|
||||
}
|
||||
|
||||
schema := `
|
||||
schema := backupHistorySchema + `
|
||||
CREATE TABLE IF NOT EXISTS recovery_points (
|
||||
id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL,
|
||||
|
|
@ -1162,11 +1163,9 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
}
|
||||
}()
|
||||
// Always release the transaction on an early return, including errors
|
||||
// scoped inside the point loop. Rollback after Commit is harmless.
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
stmt, err := tx.PrepareContext(ctx, `
|
||||
INSERT INTO recovery_points (
|
||||
|
|
|
|||
|
|
@ -200,3 +200,7 @@ type ReportingAdminRuntime struct {
|
|||
// BindReportingAdminEndpointsFunc allows enterprise modules to bind replacement
|
||||
// reporting admin endpoints while retaining access to default handlers.
|
||||
type BindReportingAdminEndpointsFunc func(defaults ReportingAdminEndpoints, runtime ReportingAdminRuntime) ReportingAdminEndpoints
|
||||
|
||||
// GenerateBackupHealthReportFunc renders already-scoped evidence. The shared
|
||||
// scheduler owns entitlement, tenancy, windows, persistence and delivery.
|
||||
type GenerateBackupHealthReportFunc func(reporting.BackupHealthData, reporting.ReportFormat) ([]byte, string, error)
|
||||
|
|
|
|||
111
pkg/proxmox/backup_history.go
Normal file
111
pkg/proxmox/backup_history.go
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
// BackupTaskHistory preserves collection quality alongside execution evidence.
|
||||
// A successful HTTP response can still contain only the caller's own tasks.
|
||||
type BackupTaskHistory struct {
|
||||
Tasks []Task
|
||||
Nodes []BackupTaskHistoryNode
|
||||
}
|
||||
|
||||
type BackupTaskHistoryNode struct {
|
||||
Node string
|
||||
Complete bool
|
||||
Reason string
|
||||
}
|
||||
|
||||
// GetBackupTaskHistory lists retained vzdump tasks in a bounded, paginated
|
||||
// window. Bounds, offline nodes and missing audit authority are explicit.
|
||||
func (c *Client) GetBackupTaskHistory(ctx context.Context, since time.Time) (BackupTaskHistory, error) {
|
||||
var history BackupTaskHistory
|
||||
nodes, err := c.GetNodes(ctx)
|
||||
if err != nil {
|
||||
return history, err
|
||||
}
|
||||
until := time.Now().UTC().Unix()
|
||||
for _, node := range nodes {
|
||||
observation := BackupTaskHistoryNode{Node: node.Node}
|
||||
if node.Status != "online" {
|
||||
observation.Reason = "node_offline"
|
||||
history.Nodes = append(history.Nodes, observation)
|
||||
continue
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for page := 0; page < 30; page++ {
|
||||
params := url.Values{"typefilter": {"vzdump"}, "source": {"all"}, "limit": {"200"}, "start": {fmt.Sprint(page * 200)}, "since": {fmt.Sprint(since.Unix())}, "until": {fmt.Sprint(until)}}
|
||||
resp, readErr := c.get(ctx, "/nodes/"+url.PathEscape(node.Node)+"/tasks?"+params.Encode())
|
||||
if readErr != nil {
|
||||
observation.Reason = "collection_failed"
|
||||
break
|
||||
}
|
||||
var result struct {
|
||||
Data []Task `json:"data"`
|
||||
}
|
||||
readErr = json.NewDecoder(resp.Body).Decode(&result)
|
||||
resp.Body.Close()
|
||||
if readErr != nil {
|
||||
observation.Reason = "collection_failed"
|
||||
break
|
||||
}
|
||||
added := 0
|
||||
for _, task := range result.Data {
|
||||
if task.Type != "vzdump" || task.UPID == "" || seen[task.UPID] {
|
||||
continue
|
||||
}
|
||||
seen[task.UPID] = true
|
||||
if task.Node == "" {
|
||||
task.Node = node.Node
|
||||
}
|
||||
history.Tasks = append(history.Tasks, task)
|
||||
added++
|
||||
}
|
||||
if len(result.Data) < 200 {
|
||||
observation.Complete = true
|
||||
break
|
||||
}
|
||||
if added == 0 || page == 29 {
|
||||
observation.Reason = "history_truncated"
|
||||
break
|
||||
}
|
||||
}
|
||||
if observation.Complete && !c.backupHistoryAuditPermission(ctx, node.Node) {
|
||||
observation.Complete = false
|
||||
observation.Reason = "audit_permission_unconfirmed"
|
||||
}
|
||||
history.Nodes = append(history.Nodes, observation)
|
||||
}
|
||||
return history, nil
|
||||
}
|
||||
|
||||
func (c *Client) backupHistoryAuditPermission(ctx context.Context, node string) bool {
|
||||
path := "/nodes/" + node
|
||||
resp, err := c.get(ctx, "/access/permissions?"+url.Values{"path": {path}}.Encode())
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var result struct {
|
||||
Data map[string]map[string]int `json:"data"`
|
||||
}
|
||||
if json.NewDecoder(resp.Body).Decode(&result) != nil {
|
||||
return false
|
||||
}
|
||||
return result.Data[path]["Sys.Audit"] == 1
|
||||
}
|
||||
|
||||
func (cc *ClusterClient) GetBackupTaskHistory(ctx context.Context, since time.Time) (BackupTaskHistory, error) {
|
||||
var result BackupTaskHistory
|
||||
err := cc.executeWithFailover(ctx, func(client *Client) error {
|
||||
var err error
|
||||
result, err = client.GetBackupTaskHistory(ctx, since)
|
||||
return err
|
||||
})
|
||||
return result, err
|
||||
}
|
||||
71
pkg/proxmox/backup_history_test.go
Normal file
71
pkg/proxmox/backup_history_test.go
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestBackupHistoryPaginationAndCollectionQuality(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
audit bool
|
||||
failSecond bool
|
||||
repeat bool
|
||||
expected int
|
||||
reason string
|
||||
}{{"full", true, false, false, 201, ""}, {"permission", false, false, false, 201, "audit_permission_unconfirmed"}, {"partial", true, true, false, 200, "collection_failed"}, {"repeated-page", true, false, true, 200, "history_truncated"}} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/api2/json/nodes":
|
||||
fmt.Fprint(w, `{"data":[{"node":"node","status":"online"},{"node":"offline","status":"offline"}]}`)
|
||||
case "/api2/json/access/permissions":
|
||||
value := 0
|
||||
if test.audit {
|
||||
value = 1
|
||||
}
|
||||
fmt.Fprintf(w, `{"data":{"/nodes/node":{"Sys.Audit":%d}}}`, value)
|
||||
case "/api2/json/nodes/node/tasks":
|
||||
require.Equal(t, "vzdump", r.URL.Query().Get("typefilter"))
|
||||
require.Equal(t, "all", r.URL.Query().Get("source"))
|
||||
require.NotEmpty(t, r.URL.Query().Get("since"))
|
||||
require.NotEmpty(t, r.URL.Query().Get("until"))
|
||||
second := r.URL.Query().Get("start") == "200"
|
||||
if second && test.failSecond {
|
||||
w.WriteHeader(403)
|
||||
return
|
||||
}
|
||||
count, offset := 200, 0
|
||||
if second && !test.repeat {
|
||||
count, offset = 1, 200
|
||||
}
|
||||
tasks := make([]Task, count)
|
||||
for i := range tasks {
|
||||
tasks[i] = Task{UPID: fmt.Sprintf("task-%d", i+offset), Type: "vzdump"}
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]any{"data": tasks})
|
||||
default:
|
||||
w.WriteHeader(404)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := NewClient(ClientConfig{Host: server.URL, TokenName: "test@pam!token", TokenValue: "fixture", VerifySSL: false})
|
||||
require.NoError(t, err)
|
||||
history, err := client.GetBackupTaskHistory(context.Background(), time.Now().Add(-time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Len(t, history.Tasks, test.expected)
|
||||
require.Len(t, history.Nodes, 2)
|
||||
require.Equal(t, test.reason, history.Nodes[0].Reason)
|
||||
require.Equal(t, test.reason == "", history.Nodes[0].Complete)
|
||||
require.Equal(t, "node_offline", history.Nodes[1].Reason)
|
||||
})
|
||||
}
|
||||
}
|
||||
23
pkg/reporting/backup_health.go
Normal file
23
pkg/reporting/backup_health.go
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
package reporting
|
||||
|
||||
import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery/model"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Backup health data is a shared evidence contract. Paid rendering is supplied
|
||||
// through the enterprise extension; collection and storage remain canonical.
|
||||
type BackupHealthData struct {
|
||||
Title string
|
||||
GeneratedAt time.Time
|
||||
Brand *ReportBrand
|
||||
Summary BackupHealthSummary
|
||||
HistoryNotice string
|
||||
}
|
||||
|
||||
type BackupHealthSummary = model.BackupHealthSummary
|
||||
type BackupOutcomeCounts = model.BackupOutcomeCounts
|
||||
type BackupHealthWorkload = model.BackupHealthWorkload
|
||||
type BackupHealthWorkloadResult = model.BackupHealthWorkloadResult
|
||||
type BackupHealthDay = model.BackupHealthDay
|
||||
type BackupHealthFailure = model.BackupHealthFailure
|
||||
|
|
@ -62,13 +62,14 @@ type ReportingGuidanceDefinition struct {
|
|||
}
|
||||
|
||||
type ReportingCatalog struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
LockedState ReportingLockedStateDefinition `json:"lockedState"`
|
||||
Guidance ReportingGuidanceDefinition `json:"guidance"`
|
||||
PerformanceReport PerformanceReportDefinition `json:"performanceReport"`
|
||||
VMInventoryExport VMInventoryExportDefinition `json:"vmInventoryExport"`
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
LockedState ReportingLockedStateDefinition `json:"lockedState"`
|
||||
Guidance ReportingGuidanceDefinition `json:"guidance"`
|
||||
PerformanceReport PerformanceReportDefinition `json:"performanceReport"`
|
||||
VMInventoryExport VMInventoryExportDefinition `json:"vmInventoryExport"`
|
||||
BackupHealthAvailable bool `json:"backupHealthAvailable,omitempty"`
|
||||
}
|
||||
|
||||
// SupportsFormat reports whether the performance reporting surface allows the
|
||||
|
|
|
|||
16
pkg/reporting/pdf_brand.go
Normal file
16
pkg/reporting/pdf_brand.go
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
package reporting
|
||||
|
||||
import "github.com/go-pdf/fpdf"
|
||||
|
||||
// ReportCoverBrandName and ReportHeaderBrandName share the existing report
|
||||
// branding policy with extension renderers. Callers pass already-authorized
|
||||
// effective branding; these primitives do not grant branding entitlement.
|
||||
func ReportCoverBrandName(brand *ReportBrand) string { return reportCoverBrandName(brand) }
|
||||
func ReportHeaderBrandName(brand *ReportBrand, fallback string) string {
|
||||
return reportHeaderBrandName(brand, fallback)
|
||||
}
|
||||
|
||||
// DrawReportBrandLogo uses the same decoding and sizing as existing reports.
|
||||
func DrawReportBrandLogo(pdf *fpdf.Fpdf, brand *ReportBrand, x, y, width, height float64) {
|
||||
NewPDFGenerator().drawBrandLogo(pdf, brand, x, y, width, height)
|
||||
}
|
||||
|
|
@ -74,6 +74,10 @@ type BusinessHooks struct {
|
|||
// reporting admin endpoints without importing internal API packages.
|
||||
BindReportingAdminEndpoints extensions.BindReportingAdminEndpointsFunc
|
||||
|
||||
// GenerateBackupHealthReport supplies private backup report rendering while
|
||||
// the public scheduler retains evidence, entitlement and delivery ownership.
|
||||
GenerateBackupHealthReport extensions.GenerateBackupHealthReportFunc
|
||||
|
||||
// BindAIAutoFixEndpoints allows enterprise modules to replace or decorate
|
||||
// safe remediation endpoints (investigation, remediation, autonomy, fix execution).
|
||||
BindAIAutoFixEndpoints extensions.BindAIAutoFixEndpointsFunc
|
||||
|
|
@ -129,6 +133,7 @@ func runtimeIdentityForBusinessHooks(h BusinessHooks) pkglicensing.RuntimeIdenti
|
|||
h.BindRBACAdminEndpoints != nil ||
|
||||
h.BindSSOAdminEndpoints != nil ||
|
||||
h.BindReportingAdminEndpoints != nil ||
|
||||
h.GenerateBackupHealthReport != nil ||
|
||||
h.BindAIAutoFixEndpoints != nil ||
|
||||
h.BindAIAlertAnalysisEndpoints != nil ||
|
||||
h.AIInvestigationEnabled != nil ||
|
||||
|
|
@ -365,6 +370,7 @@ func Run(ctx context.Context, version string) (runErr error) {
|
|||
bindAuditAdminEndpoints := globalHooks.BindAuditAdminEndpoints
|
||||
bindSSOAdminEndpoints := globalHooks.BindSSOAdminEndpoints
|
||||
bindReportingAdminEndpoints := globalHooks.BindReportingAdminEndpoints
|
||||
generateBackupHealthReport := globalHooks.GenerateBackupHealthReport
|
||||
bindAIAutoFixEndpoints := globalHooks.BindAIAutoFixEndpoints
|
||||
bindAIAlertAnalysisEndpoints := globalHooks.BindAIAlertAnalysisEndpoints
|
||||
aiInvestigationEnabled := globalHooks.AIInvestigationEnabled
|
||||
|
|
@ -384,6 +390,7 @@ func Run(ctx context.Context, version string) (runErr error) {
|
|||
api.SetAuditAdminEndpointsBinder(bindAuditAdminEndpoints)
|
||||
api.SetSSOAdminEndpointsBinder(bindSSOAdminEndpoints)
|
||||
api.SetReportingAdminEndpointsBinder(bindReportingAdminEndpoints)
|
||||
api.SetBackupHealthReportGenerator(generateBackupHealthReport)
|
||||
api.SetAIAutoFixEndpointsBinder(bindAIAutoFixEndpoints)
|
||||
api.SetAIAlertAnalysisEndpointsBinder(bindAIAlertAnalysisEndpoints)
|
||||
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ import (
|
|||
"github.com/rcourtman/pulse-go-rewrite/pkg/extensions"
|
||||
pkglicensing "github.com/rcourtman/pulse-go-rewrite/pkg/licensing"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/reporting"
|
||||
)
|
||||
|
||||
func TestRunBindsPackagedVersionIdentity(t *testing.T) {
|
||||
|
|
@ -191,6 +192,12 @@ func TestRuntimeIdentityForBusinessHooks(t *testing.T) {
|
|||
if got.Build != pkglicensing.RuntimeBuildPro {
|
||||
t.Fatalf("commercial admission hook runtime build=%q, want pro", got.Build)
|
||||
}
|
||||
got = runtimeIdentityForBusinessHooks(BusinessHooks{
|
||||
GenerateBackupHealthReport: func(reporting.BackupHealthData, reporting.ReportFormat) ([]byte, string, error) { return nil, "", nil },
|
||||
})
|
||||
if got.Build != pkglicensing.RuntimeBuildPro {
|
||||
t.Fatalf("private backup renderer hook runtime build=%q, want pro", got.Build)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPerformAutoImport_Success(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -8,6 +8,9 @@ internal/api/reporting_runtime_snapshot.go
|
|||
# Shared subject enrichment behind the existing reporting admin binder/gates;
|
||||
# not a new paid endpoint or entitlement implementation. Exact path only.
|
||||
internal/api/reporting_subject_alerts.go
|
||||
# Shared scoped evidence adapter. New paid PDF/CSV renderer is implemented
|
||||
# privately through GenerateBackupHealthReportFunc. Exact path only.
|
||||
internal/api/reporting_backup_health.go
|
||||
internal/api/router.go
|
||||
internal/api/router_routes_auth_security.go
|
||||
internal/api/router_routes_cloud.go
|
||||
|
|
|
|||
|
|
@ -16,6 +16,15 @@ class ReportingSubjectBoundaryTests(unittest.TestCase):
|
|||
def test_shared_exemption_is_exact_and_preserves_import_boundaries(self):
|
||||
variants = [
|
||||
("shared pipeline", None, None, None),
|
||||
(
|
||||
"backup adapter exact path", "reporting_backup_health_extra.go",
|
||||
"package api\n", "new production paid-domain implementation files are not baselined",
|
||||
),
|
||||
(
|
||||
"backup adapter private root import", "reporting_backup_health.go",
|
||||
'package api\nimport _ "github.com/rcourtman/pulse-go-rewrite/internal/license"\n',
|
||||
"API root imports of internal/license remain",
|
||||
),
|
||||
(
|
||||
"new reporting implementation", "reporting_subject_alerts_extra.go",
|
||||
"package api\n", "new production paid-domain implementation files are not baselined",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue