Pulse/docs/METRICS_HISTORY.md

Metrics History (Persistent)

Pulse persists metrics history to disk so trend views and sparklines survive restarts.

Read a History panel

In current drawers, open the affected resource's Overview and use History range beside its charts. Proxmox guests and nodes, Docker hosts and resource drawers (including PBS) show stored charts there instead of a separate History tab. If your installed version still has a History tab, use that tab; do not upgrade or restart just to follow this guide. A resource without a supported history target may have no chart.

On a phone, drawers can show only Utilization and panels with active alerts. Use the Show … button below the charts to reveal the others, for example Show Network I/O, Disk I/O, Thermals. Its labels name the currently collapsed panels, so they vary by resource and alert. Show fewer charts collapses them again. Wider layouts show all panels and may have no button; older installed layouts may differ.

This button changes only the layout, preserving History range; it does not refresh history or collect new measurements. A collapsed panel is not missing data or measured zero. After revealing it, No stored history in this range means that panel has no plotted stored samples for the selected range, not zero usage or proof that collection stopped. Loading, an access error, a locked range and History unavailable are different states: retain the existing message and range. Do not refresh, run Diagnostics or restart just to reveal a chart.

These charts are separate from Alerts → History, table sparklines and current summary cards. A current card or a short trend preview does not establish the observations in the selected stored range. For a PBS host, use the Backup Server chart, not its datastore or the VM that happens to host it.

Start with the existing chart, not a restart or a storage change. A current reading does not prove that it was stored, and a selected time range does not prove continuous coverage. Window start and Window end bound the requested interval; they are not the first and last successful observations.

VM and LXC drawer panels can show these labels. Other charts and older Pulse versions may differ; if a time or source is not shown, keep it unknown rather than borrowing one from the current overview.

Display	How to read it	What it does not establish
Latest stored	The newest stored observation time in this panel. Each value belongs to that time; a dash means no stored observation for that metric at that time.	Not necessarily now, every metric's newest value, or another panel's time.
Stored while inspecting	Values at the selected stored observation time.	Not a live reading or an interpolated observation between points.
current, last known or freshness unknown with no stored history	A separately qualified fallback reading. Keep its label and explanation.	Not proof that the value was persisted or belongs to a past incident.
Single observation. No trend yet.	One stored time in that panel.	Not a flat trend or sustained recovery.
Missing observations	Some series lack readings at other stored times in that panel; lines stop across those gaps.	Not the cause or duration of a monitoring outage. Missing is not measured zero.
Shaded low–high band	The recorded minimum and maximum within an aggregated bucket, when supplied; the line shows its average.	Not continuous coverage, every original sample, or the duration of a spike.
peak while inspecting	That metric's maximum in the inspected bucket, when the panel shows it beside the average.	Not necessarily the highest value in the whole selected range or the value at an alert's exact time.

For example, CPU 22% at 12:02 and memory 95% at 12:01 are not one observation. At Latest stored · 12:02, memory is a dash if there is no memory point at 12:02; the earlier 95% remains inspectable. Network and disk I/O panels have their own times, not a shared snapshot of the guest.

An aggregated CPU bucket can show 22% average, 10% to 95%. The line at 22% does not rule out the recorded 95% peak, and the band does not say how long that peak lasted. No visible band may mean a single reading, equal bounds or unavailable bounds; it does not prove that usage stayed constant. Older panels may not show these bounds. Keep that limitation rather than inferring a peak from the line alone.

1. Keep the same resource, metric and range when comparing existing readings. Inspect history, when offered, steps through stored times: focus the slider and use the arrow keys, Home or End, or move it by touch. Hover is another way to inspect; it is not required. Leaving the slider returns to the latest stored display.
2. Keep the displayed observation time and qualification with the value. Stored points can be aggregated or downsampled; they are not a measurement of every instant in the selected range. A page refresh time is not the observation time. If an absolute time is unavailable, retain that limit.
3. Compare an alert or Patrol finding with the original episode, not just today's overview. Today's source cannot identify the source of a past warning. Follow guest memory interpretation when unlike readings suggest memory pressure.

Use already available observations; do not run Diagnostics or guest-agent probes, restart a guest, repeat a backup, clear History or change retention to make a comparison agree. An unresponsive guest or an uncertain freeze/thaw operation needs Backup safety, not chart inspection. When reporting a display disagreement, keep the panel, time, range, labels and relevant values with consistent private aliases; do not share raw responses, credentials or a full network export.

Retention and accessible range

Storage retention and the range your running instance can serve are separate. Community supports up to 7 days; Pro and legacy Pro+ entitlements support up to 90 days. The panel uses the active runtime's history limit, not the length of time the database has existed. A longer retention setting does not grant access, and a licence cannot recover observations that were never stored.

Relay is no longer sold. Continuing Relay subscribers receive Pro features, including 90-day history, at their existing price. Older Relay payloads can still show a 14-day compatibility limit; that is not a plan to buy or proof that a continuing subscription has received its current entitlement. See plans and legacy subscriptions.

An API request beyond the active limit returns 402 Payment Required, not an empty history or a failed collector. Use an allowed range and the existing licence/range display first; do not buy another licence, reset history or reinstall merely to diagnose it. Storage defaults below do not guarantee that every resource has data for every allowed range.

Storage Location

Metrics history is stored in a SQLite database named metrics.db under the Pulse data directory:

• systemd/LXC installs: typically /etc/pulse/metrics.db
• Docker/Kubernetes installs: typically /data/metrics.db

Retention Model (Tiered)

Pulse keeps multiple resolutions of the same data, which allows longer history without storing raw samples forever:

• Raw (high-resolution, short window)
• Minute aggregates
• Hourly aggregates
• Daily aggregates

Default retention values (subject to change) are:

• Raw: 2 hours
• Minute: 24 hours
• Hourly: 7 days
• Daily: 90 days
• Rollups: every 15 minutes by default, bounded so rollups still run well before raw samples expire

Advanced: Retention Tuning

Do not change retention or polling merely because a chart is empty or writes are high. First follow Troubleshooting and preserve the original observations. Shorter retention removes history; slower polling also changes how quickly monitoring can notice a problem.

Tiered retention is stored in system.json in the Pulse data directory:

• systemd/LXC installs: typically /etc/pulse/system.json
• Docker/Kubernetes installs: typically /data/system.json

Keys:

{
  "metricsRetentionRawHours": 2,
  "metricsRetentionMinuteHours": 24,
  "metricsRetentionHourlyDays": 7,
  "metricsRetentionDailyDays": 90
}

After changing these values, restart Pulse.

Advanced: Disk Write Tuning

This is a deliberate storage trade-off, not a repair for missing history or unexplained writes. Changing the path opens a different history store; it does not migrate the existing history. Keep the original durable data and establish the writer using bounded performance checks before changing storage.

Pulse keeps metrics history on disk by default. SSD-sensitive installs can move only the metrics SQLite database without moving secrets or general config:

PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db

Use a dedicated directory owned by the account running Pulse. Do not place the database directly in a shared parent such as /tmp/metrics.db: Pulse secures the database directory to mode 0700 and will reject a directory owned by another user. On systemd installs, PrivateTmp=true also means a path below /tmp is private to the service and will not appear in the host's /tmp.

The default systemd sandbox permits /dev/shm/pulse and Pulse can create that subdirectory itself. A dedicated mount elsewhere, such as /mnt/ramdisk, also needs an explicit writable-path grant:

sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse
sudo systemctl edit pulse

Add this drop-in (replace pulse with the installed unit name if different):

[Service]
ReadWritePaths=/mnt/ramdisk/pulse

Then set PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db in /etc/pulse/.env and restart Pulse. ProtectSystem=strict deliberately keeps paths outside the unit's writable allowlist read-only even when Unix ownership would otherwise permit writes.

For Docker, mount a tmpfs at the selected directory and keep /data on a persistent volume:

services:
  pulse:
    environment:
      PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
    tmpfs:
      - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700

Using tmpfs loses that history when the RAM-backed mount is discarded, for example on a host reboot or container recreation. A service restart alone is not guaranteed to clear it. Do not use tmpfs for /data: it also contains config, encrypted credentials, tokens, and other state that must remain durable.

The aggregation cadence can also be lengthened when an install prefers fewer, larger rollup writes over more frequent smaller writes:

PULSE_METRICS_ROLLUP_INTERVAL=30m

Values below 5 minutes are ignored. Values longer than half of the raw-retention window are capped by the metrics store so raw samples are still rolled up before retention pruning can remove them.

API Access

Pulse exposes the persistent metrics store via:

• GET /api/metrics-store/stats
• GET /api/metrics-store/history

These endpoints require authentication with the monitoring:read scope.

History Query Parameters

GET /api/metrics-store/history supports:

• resourceType (required): node, storage, agent, disk, k8s, vm, system-container, oci-container, app-container, or docker-host
• resourceId (required): the source identifier used by that chart, not its display name (for Proxmox guests this is normally instance:node:vmid)
• metric (optional): cpu, memory, disk, etc. Omit to return all metrics for the resource.
• range (optional): 1h, 6h, 12h, 24h, 1d, 7d, 30d, 90d (default 24h; duration strings also accepted)
• maxPoints (optional): Downsample to a target number of points

For a one-off check, open the affected resource's History panel in your signed-in Pulse browser. If an API read is needed, use the path and query from that panel's request on the same instance; do not extract its session cookie, use Copy as cURL, or share a full network export. /api/metrics-store/stats describes the store overall, not whether this resource has the requested metric.

For curl, follow API authentication to prepare the private header file with only monitoring:read and define its pulse_api helper in the same Bash session. It is example code, not an installed Pulse command. The default helper runs on the Pulse host; for remote access, follow that guide's HTTPS-origin and verified-CA instructions. Then make one read-only request (curl 7.76 or later):

pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'

Replace the example type and ID with the chart's values, URL-encoding each query value (: becomes %3A; an ID's & must be %26, not another query parameter). Do not substitute a hostname or strip a provider prefix to guess an ID. The helper saves each response, including an error body, in a new owner-only file and prints only the HTTP status and file location. Open that file privately; do not print or post it. HTTP 401, 402, 403 and server errors, redirects, or incomplete transport are not an empty history or zero usage. There is no automatic retry. Never put a token in a command, URL or report; share only the relevant manually redacted error and bounded observations.

With metric=cpu (or another explicit metric), the response has a points array. Omitting metric instead returns a metrics object keyed by metric name, not the same single-series shape. Inspect timestamps and source too: store is persisted history; memory or live fallback does not establish durable historical coverage. An HTTP 200 with no points proves neither zero usage nor successful collection, and a non-zero current value does not prove every historical interval is present.

History limit: Requests beyond the active runtime's allowed range return 402 Payment Required. See retention and accessible range; stored retention is not access or proof of collected observations. Older examples: container, dockerHost, dockerContainer, guest and docker are not current resourceType query values. Use the chart's current type rather than a display label or internal storage name.

Troubleshooting

• Missing or stale history: retain the affected panel, metric, range, collection time and current server/agent versions. Check the connection's last normal poll or agent report in Settings → Infrastructure, without running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw. Compare the panel's actual request, HTTP result, point timestamps and source. A valid connection test, another resource's graph or the existence of metrics.db does not establish this metric's collection or display.
• Access or query error: keep it separate from an empty result. Check the selected organisation, monitoring:read access, exact query type/ID and the requested range. A 402 is a history-range entitlement failure, not evidence that storage is empty; try an allowed range without buying another licence or resetting data.
• Storage warning: inspect a bounded existing startup/runtime error and the deployment's persistent mount and service-account access privately. Follow the safe troubleshooting guide for high writes or space pressure. Do not delete history, edit a database, shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.
• Report still needed: share the affected metric/range, versions, whether normal collection advances, and a manually redacted error; keep resource identities, response files and databases private. Follow Getting Help, not a full response dump.