Pulse/docs/TROUBLESHOOTING.md
rcourtman 2fc54a5a1f Merge pending maintenance repairs with current main
Preserve selected memory freshness and the VM/LXC interpretation.
Retain both native tool contracts and current API concurrency controls.
Qualify this source separately from release and installed recovery.

Contract-Neutral: Integrate already-governed parents without a new contract change. Incoming verification files already match main. Fresh integration tests and source-bound browser proof cover the combined memory and installer surfaces.
2026-10-18 15:11:28 +01:00

111 KiB
Raw Permalink Blame History

🔧 Troubleshooting Guide

Choose the affected workflow

Start with the symptom below, using observations you already have. A restart, reinstall or new diagnostic run is not a prerequisite for getting help.

If a guest is unresponsive, a backup is in freeze/thaw, or an earlier guest request's state is uncertain, read Backup safety first. Do not run Diagnostics, force guest-agent probes or repeat a backup to fill a report. A successful backup task or responsive console alone does not establish safe recovery.

If none fits, use Getting Help. Review evidence locally before sharing it; unknown or unavailable details are valid, and a full configuration, journal or network export is not required.

⚡ Quick Fixes

Find the running server and agent versions

Use the existing display on the affected Pulse installation; no update, restart or new diagnostic run is needed just to identify a version.

  • Pulse server: open Settings → System → Updates and read Server version, not the available update. Keep any beta/RC suffix or development build identity exactly as displayed. A newer server does not establish that a separate agent has updated.
  • Pulse Agent: under Settings → Infrastructure, open the affected connection and read Pulse Agent version alongside Last seen. An Update available value such as 6.4.5 → 6.5.0 distinguishes the reported version on the left from the target on the right. Version target alone is not a reported agent version. If the connection is stale or offline, its last report does not prove what is running now; keep that time and uncertainty. For an API-only connection with no Pulse Agent, report the agent as none, not the server's version. See agent update behaviour for the separate update path.
  • Container image: include the image tag or digest already available in your deployment manager, separately from Server version. A tag such as latest is mutable; a Compose file or selected image is not proof that a running container was replaced. Keep the exact reference, including the registry and preview suffix; do not substitute a GitHub release tag. A helm-chart-… release identifies a chart, not a Pulse server archive or the application version running in Kubernetes.

Keep the version at the failure, an interrupted update's intended version and the version observed after recovery separate. Today's display does not reconstruct the failing run. If the helper selected a release but Pulse never started, give that attempted release and the public helper name or URL instead; installer output or a verified download signature does not prove completion.

If a view is unavailable or you cannot access it with your existing permissions, write unknown; do not widen access, reinstall or re-enrol an agent, rerun an update or enable Debug to fill the gap. Do not share full container inspections, resolved Compose configuration, service environments or command lines: they can contain credentials. Existing version text or a locally reviewed, redacted excerpt is enough. Add it to the existing report; information already supplied does not need to be collected again.

I forgot my password

Pulse does not provide an email password-reset flow. Choose the path that matches how this self-hosted instance authenticates:

  • Local Pulse administrator username and password: recovery requires authorised access to the host or deployment's credential source. Follow the steps below; keep the existing username.
  • OIDC, SAML, or proxy authentication: contact the identity-provider or Pulse administrator. If the administrator deliberately kept local login as a fallback, they can open the Pulse URL with ?show_local=true; this only reveals the existing local form and does not reset credentials.
  • Temporary lockout: wait for the displayed lockout to expire. An authorised administrator can use the manual recovery API; this does not reset a password or bypass SSO.

Recover the existing local administrator login

Do not delete .env or repeat first-time setup to recover a password. The file can also hold deployment settings, and setup replaces the primary API token. Replace only the password in its active credential source instead, preserving the username, API tokens, encryption keys, monitoring configuration and data. This is login recovery, not a complete response to a compromised account; existing sessions and API tokens need separate review if credentials leaked.

  1. Find the active credential source locally. Deployment-supplied PULSE_AUTH_USER and PULSE_AUTH_PASS take precedence over Pulse's .env. Do not post full service units, container inspections or resolved Compose configuration; they can contain secrets.

    • Docker: the generated file is in the mounted data directory, normally /data/.env inside the container. If Compose, --env-file, Kubernetes or another manager supplies the password, edit that managed source instead.
    • Systemd: check the active service's unit, environment files and drop-ins privately. Root-run setup can also save credentials in /etc/systemd/system/pulse.service.d/override.conf (or the corresponding pulse-backend.service.d path on legacy installs). Editing only /etc/pulse/.env will not override those values. A custom PULSE_DATA_DIR changes the generated file's location.
    • Proxmox LXC: perform the systemd steps inside the Pulse container, not on the Proxmox host. Use its console or pct enter <ctid>.
  2. Prepare a new password hash privately. Use a trusted local bcrypt tool that prompts for a password of at least 12 characters, never one that needs the password in its command arguments. For example, if Apache's htpasswd is installed on your trusted administration machine:

    (
    set -eu
    umask 077
    recovery_dir="$(mktemp -d "${TMPDIR:-/tmp}/pulse-password.XXXXXX")"
    htpasswd -nB -C 12 pulse-recovery > "$recovery_dir/password-record"
    printf 'Private password record saved in %s/password-record\n' "$recovery_dir"
    )
    

    This prints only the file location. Open the record in a private editor and copy the complete 60-character hash after pulse-recovery:; that label is not a new Pulse username. Do not post the record or hash. If the tool fails, stop rather than saving a partial hash or putting the password in a shell command.

  3. Back up and edit only the active password setting. Keep an owner-only backup of each file you change, then replace PULSE_AUTH_PASS in a private editor. Keep PULSE_AUTH_USER and unrelated settings unchanged. In a Pulse .env, put the hash in single quotes with literal $ characters. Compose YAML has different interpolation rules; follow the authentication guide for the source you actually use. If a systemd drop-in supplies the password, update it and the generated authentication file consistently so a later restart or reload does not restore the old value. Preserve file permissions and ownership; hashes and backups are sensitive too.

  4. Apply through the existing deployment. For a generated Docker data-file change, restart the same container. Changing Docker's managed environment requires its recreate/redeploy operation, not just docker restart; preserve the same image, mounted data and other settings. For systemd, reload the service manager after a unit or drop-in change, then restart the active Pulse service during a suitable maintenance window. Do not remove a data volume, reinstall Pulse or re-enrol agents.

  5. Verify the login. Use a fresh browser session to sign in with the same local administrator username and new password, then check that monitoring and agent connections remain intact. If it still fails, stop and reconcile the effective source and any lockout; do not delete more state. Restore the private backup through the same deployment path if you need to undo the change. Remove the temporary password record after verification, and retain or dispose of the backup under your normal credential policy.

A fresh install's bootstrap token was missed

If no local password has ever been set, there is nothing to reset. Follow first login to read the existing bootstrap token for that instance, using its actual data directory. Enter it only in the Pulse setup screen, not in command arguments, URLs or issue reports.

Port change didn't take effect

The web UI and API listen on FRONTEND_PORT (default 7655). The deprecated PORT alias applies only when FRONTEND_PORT is unset; changing frontendPort in system.json has no effect. PULSE_AGENT_INGEST_PORT is a separate agent listener, not the web UI port. See Port configuration.

  • Systemd / Proxmox LXC: identify the active service with systemctl is-active pulse (legacy installs may use pulse-backend). Inspect only the port setting in its managed configuration locally. If you change a unit or drop-in, run sudo systemctl daemon-reload, then restart the affected service during a suitable maintenance window. For LXC, run these checks inside the Pulse container, not on the Proxmox host.
  • Docker / Compose: distinguish the published host port from the listener inside the container. With the default listener, 8080:7655 exposes the UI on host port 8080; changing the host port does not require FRONTEND_PORT. In the repository's Compose file, PULSE_PORT controls this host-side mapping. Save the mapping in your existing Compose project and apply it with docker compose up -d pulse, keeping the same image and mounted data volume. Restarting an existing container does not apply a new port mapping. For other container managers, use their recreate/redeploy action while preserving the data mount; do not delete the volume.
  • Reverse proxy: check its upstream port and the firewall separately. A working agent connection on a split-port deployment does not prove that the web UI is reachable.

Do not post full service environments, docker inspect output or resolved docker compose config output: they can include passwords and tokens. Share only the relevant port numbers and a redacted error if help is needed.

"Connection Refused"

Identify which connection failed before changing ports or credentials:

  • Your browser cannot open Pulse: check that the intended Pulse service or container is running, then its actual listening port, container port mapping and reverse proxy target. Pulse's default port is 7655, but an existing deployment can use a different one. Use the port-change checks and reverse proxy guide for that path; do not broadly open firewall access or expose Pulse directly to bypass a proxy.
  • Pulse opens, but a monitored platform request is refused: check the affected connection's saved hostname, scheme and port in Settings → Infrastructure, and the network path from the Pulse server. Browser access to the platform is a different path. Proxmox VE normally uses HTTPS on 8006; PBS uses HTTPS on 8007. Other platforms use their saved endpoint, not an assumed Proxmox port. Changing Pulse's listening port does not repair this connection.

A connection refusal is not an authentication response (401/403) or a certificate-validation error. Keep the original time and redacted error; do not replace credentials or disable TLS verification on a refusal alone. For a page that opens but has missing readings, use the missing-data checks instead of resetting the connection.

Proxmox or PBS certificate validation failed

A TLS error is not an invalid API token or a missing permission. Keep the original error and time, and inspect the existing connection in Settings → Infrastructure → Manage → SSL settings. Check its saved Endpoint URL, Verify SSL certificate and SSL Fingerprint (optional) without changing them just to diagnose the failure. PVE normally uses HTTPS port 8006; PBS uses 8007. A browser reaching the platform is a different client and trust store from the Pulse server or container.

Keep certificate verification enabled. Do not switch to HTTP, disable verification, clear a saved pin, rotate API tokens or delete and re-add the connection merely to get past an error.

  • Unknown authority or self-signed certificate: the Pulse runtime needs the intended trusted certificate chain, or an independently verified SHA-256 leaf fingerprint. Obtain trust through an already-trusted platform console or administration channel, not solely from the failing connection or its suggested fingerprint. See the PBS certificate check.
  • Expired, not-yet-valid or hostname error: reconcile the platform's certificate, its validity period, the clocks and the saved endpoint's hostname. An IP address is not interchangeable with a certificate's DNS name. Do not bypass the error or pin an unsuitable certificate to hide it.
  • Fingerprint mismatch: the presented leaf differs from the saved pin. Renewal is one possible reason, not proof that this is the same server. Verify the replacement independently before making a trust change.
  • TLS trust conflict: different saved fingerprints refer to equivalent endpoint URLs. This is contradictory saved trust, not a reason to choose whichever pin connects. Preserve the entries and verify which endpoint and certificate each belongs to before reconciliation.

Certificate renewal and saved fingerprints

A saved fingerprint selects leaf-certificate pinning instead of normal CA chain and hostname validation, even with Verify SSL certificate enabled. Renewing an ACME certificate can therefore invalidate a saved pin while the new certificate's chain is valid. Leaving verification enabled with no saved fingerprint uses the Pulse runtime's CA trust; disabling it with no pin removes chain and hostname checks.

Choose the intended trust method, not an error-suppression workaround. Only after independently verifying the replacement certificate and server identity, update the affected connection's pin. If deliberately moving from leaf pinning to CA validation, first establish the runtime's trusted chain and matching endpoint hostname, then remove only that connection's pin while keeping verification enabled. Preserve unrelated credentials, collection scope and saved connections. If you cannot establish trust, stop rather than accept the new certificate.

Cluster members and conflicting saved trust

Cluster members can have different certificates. A reachable primary or one healthy member does not prove trust or polling recovery for every peer; do not copy the primary's fingerprint to all members. In Cluster members, retain the affected node and existing failure reason. If Unreachable — details is available, open it with touch or keyboard rather than relying on a hover. A display-name edit does not reconcile certificate trust.

If the affected member pin or conflicting entry cannot be safely reconciled through the existing controls, leave it intact and use Getting Help. Share only the running Pulse version, PVE/PBS version, original time, redacted error class, which primary or peers are affected, and whether a certificate renewal was already observed. Use consistent aliases; keep literal endpoints, fingerprints, credentials and full configuration private. Unknown is valid. Do not edit encrypted files, run Diagnostics, force rediscovery, reinstall or restart just to fill a report. During a freeze/thaw or unresponsive-guest incident, follow Backup safety first; certificate help does not establish workload recovery.

CORS errors

CORS controls whether a browser can read a response from a different origin. It is not Pulse's connection to Proxmox, PBS or TrueNAS, and it does not replace authentication, CSRF protection or TLS.

In your browser's developer tools, compare the page's origin with the failed API request's origin and HTTP status. An origin consists of scheme, host and port: HTTP and HTTPS, or two different ports, are different origins. Review the original console error locally; a certificate failure, login redirect or 401/403 response needs its own correction, not a broader CORS allowlist.

  • Pulse UI and API share a public origin: a same-origin reverse proxy needs no CORS exception. Keep the API behind that proxy and correct its routing or trusted forwarded headers; do not expose the backend or add wildcard response headers to bypass it.
  • A separate trusted browser app needs API access: in Settings → System → Network → CORS Allowed Origins, list only that app's exact origin, for example https://app.example.com:8443. Do not include a path, trailing slash or a hostname pattern. Multiple exact origins are comma-separated. ALLOWED_ORIGINS overrides the saved setting; removing that environment override does not erase an existing saved allowlist. Re-open settings after saving and verify the effective value and the ordinary browser request.

An empty policy grants no cross-origin browser permission. * allows any origin without credentialed browser access; it cannot fix a request that needs a browser session cookie. Do not disable authentication, CSRF protection or TLS verification to make such a request work. Iframe embedding and proxy authentication have separate settings; CORS is not their trust boundary.

Keep cookies, authorization headers, API tokens and full network exports private. A report needs only the redacted error, HTTP status and relevant origins, not a credential-bearing request or HAR file.


🔍 Common Issues

Authentication

"Invalid username or password" after setup

When the local login form reports this error, stop repeated password guesses: failed attempts count against both the username and client IP. Privately check the existing username and the active credential source. Deployment-supplied PULSE_AUTH_USER and PULSE_AUTH_PASS take precedence over Pulse's generated .env, so changing only that file may leave the running password unchanged.

A bcrypt hash must be complete (60 characters). Compose YAML interpolation, Docker --env-file and Pulse's generated .env use different quoting rules; do not blindly replace every $ with $$. Follow the authentication-file guidance for your actual source, keeping hashes and resolved deployment configuration private. If the password is genuinely lost, use deployment-specific recovery, not setup or re-enrolment.

Cannot login / 401 Unauthorized

Identify which request failed before clearing cookies or changing credentials:

  • Browser session: a 401 on an ordinary UI request can mean a missing or expired session, not a wrong password. Open the same public Pulse URL in a fresh browser session and sign in through the configured method. Preserve any working administrator session; do not clear unrelated sites' cookies.
  • SSO or proxy login: use the identity-provider or Pulse administrator's recovery path. Check the proxy authentication guide when the proxy signs you in but Pulse rejects the request. A local lockout reset does not repair an identity-provider account or missing trusted proxy headers.
  • Local login form: an Account locked response gives the remaining wait. Lockout applies separately to username and client IP, normally for 15 minutes; clearing cookies does not reset it. A Too many requests response (429) is rate limiting, not proof that the password is wrong. Wait rather than retrying rapidly, restarting Pulse or disabling authentication.
  • API client: a missing or invalid API token can cause 401 even when browser login works. Check the token's status and intended permissions privately. A 403 can instead mean insufficient permissions or failed CSRF protection; do not disable those checks or reset the password to repair API access. Use the private header-file examples, never a token or session cookie pasted into a command or URL.

Manual lockout reset requires an authenticated administrator with settings:write authority; session requests also require CSRF protection. The manual recovery API resets one username or IP identifier at a time, not a password, SSO account or every lockout. Do not assume resetting the username also clears a separately locked IP.

For a report, retain the time, sign-in method, redacted error, HTTP status and request path (without query strings). Keep passwords, hashes, cookies, tokens, full headers, Copy as cURL commands and network exports private. A 401 alone does not justify deleting configuration, recreating the data volume or re-enrolling agents.

Audit Log verification shows unsigned events

Unsigned means no signature was stored for that event; it is not the same as Failed verification or a request Error. Signing can be unavailable when Pulse cannot initialise its encryption manager. Check a bounded startup log excerpt and the persistent data mount and access for the service account, without printing key contents. Do not delete or regenerate .encryption.key or an audit signing key to make the warning disappear. Restoring signing for new events cannot authenticate an old unsigned event.

See Audit verification and safe recovery for the different results and evidence to retain.

Audit Log is empty

Clear the Event, user search, Outcome and Verification filters (or use Clear all filters in the empty-state message), and check the selected organisation first. A query error is not an empty history. Pulse Pro runtime required means an active licence is running on the public community runtime; follow the panel's Download Pulse Pro link rather than buying another licence or resetting storage. Without the audit capability, reads and exports are gated, but Pulse still attempts to capture events persistently on all plans. Console Logging Only can also reflect unavailable persistent storage: inspect the bounded startup logs for audit initialisation errors. Do not change passwords or create tokens merely to populate the panel.

Audit Log verification fails for older events

A failed signature check does not by itself prove tampering. An event signed with a different key can fail even when its contents are unchanged; missing signatures and an unsupported or damaged signature format also cannot verify. Keep the failure as evidence. Do not swap an old key into the live instance, edit audit rows or re-sign old events. Preserve the current data and keys privately before any recovery; compare a matching backup only in an isolated restore, not by overwriting today's history. Follow safe audit recovery.

Monitoring Data

Unraid reports missing disks in empty slots

Compare the same host's existing Unraid status with Pulse's disk inventory, storage-topology alert and observation time. Keep the Pulse server and Unraid Unified Agent versions separate: updating the server does not establish that the agent updated. No new command, Diagnostics run or array operation is needed to report a disagreement.

Existing observation Meaning
DISK_NP, no device or size, and only fsType=auto or a transport-only identity such as ata-_ A configured-but-unassigned slot, not evidence of a missing assigned disk. A slot name such as disk6 or parity2 alone is not assignment evidence.
DISK_NP_MISSING, even without a device or identity Unraid explicitly reports an assigned-but-absent member. Do not discard it as an empty slot.
A missing/disabled slot with a real device, model/serial, filesystem or size Assignment evidence remains; do not dismiss the failure because other slots are empty. auto alone is not a real filesystem.
Single online parity disk and five data disks Pulse's numProtected counts online parity devices: 1, not five protected data disks. That count alone does not prove complete protection or healthy disks.
No matching native observation, an old reading or an unknown version The comparison is unavailable, not a healthy result. STARTED alone does not establish array health.

For the healthy partially populated topology in #1788, seventeen empty data slots wrongly contributed to numMissing. The final correction treats fsType=auto as a placeholder at agent collection, server ingestion and storage-health assessment. Pulse v6.5.0 contains this correction in the server and Unified Agent; an earlier preview retest failed before this final correction. This does not establish that a current alert has the same cause or that the reporter's installation has recovered.

If existing native status shows no missing assigned members but Pulse disagrees, preserve the mismatch rather than suppressing the check or changing the array. Do not unassign a disk, rebuild, start a parity check, restart or re-enrol the agent, or enable Debug just to diagnose the alert. A genuine missing, disabled, invalid or unavailable parity member remains a separate safety concern.

For a report, share only already available versions, the original time, relevant status/counts and redacted alert text. Keep disk serials, paths and raw agent payloads private; use consistent aliases and leave unknowns unknown. Follow Getting Help, without repeating a failed update or array operation to supply evidence.

RAID array is degraded with a spare

For a Linux MD RAID alert, compare the same array's existing native status with Pulse's reading and observation time. In mdadm output, Raid Devices is the required member count; Total Devices can include a spare. A spare does not replace an active member until it has joined the array.

Existing observation Meaning
clean; Raid Devices 4, Active 4, Total 5, Working 5, Spare 1, Failed 0 Four required members are active, with one spare. The total of five does not by itself mean a member is missing.
Raid Devices 4, Active 3, with a spare still present A required member is missing. Do not subtract the spare from the required count or treat it as restored redundancy.
Failed devices, a degraded/unavailable state, or recovery/reshape in progress These are separate health or recovery observations. A spare or the word clean alone does not establish that recovery finished.
Missing counts, an old reading or different array identities The comparison is unavailable or unlike-for-like, not a healthy result.

Pulse v6.5.0 includes the spare-count correction reported in #2369 for the clean four-member-plus-spare case, including reports from older collectors. That does not establish that every RAID alert has this cause or that an affected installation has recovered. A healthy count also does not verify data integrity, backup completeness or restore readiness.

For genuine member loss or recovery, use the storage platform's established recovery procedure and independent monitoring. Keep RAID alerts enabled; do not remove a spare, fail a member, assemble/rebuild an array, start a scrub, reboot or change collection settings just to test or silence Pulse. No new mdadm command, Diagnostics run or induced failure is required to file a report.

For help, retain the original alert/time, running Pulse and collector versions where known, collection path, and the already available state and counts for one matching array. A missing collector version or native observation can be reported as unknown. Use consistent private aliases for the host and array; keep UUIDs, serial numbers, private paths and full command output out of public reports. Do not repeat an array operation to obtain evidence.

Primary IP shows a Podman or Docker bridge address

On a Proxmox VM or LXC, Primary IP is Pulse's selected address, not proof of the guest's default route or reachability from your browser. An agentless LXC uses the Proxmox collection path. When interface associations are supplied, Pulse prefers other guest interfaces to recognised local-container/overlay interfaces such as podman0; this is a display hint, not a physical-NIC test. Useful secondary addresses are retained, including a virtual-only guest's addresses. An unknown interface association remains unknown; address order does not establish the intended route. The Proxmox API and Pulse agent use different collection paths; an agent's correct address does not establish that the API-only view is correct.

For access, use the intended guest address from your existing Proxmox network configuration rather than assuming the displayed first address is suitable. In Workloads, hover over the guest's network icon to inspect Network Interfaces: when supplied, interface names appear beside their addresses. The guest's details also show Other IPs when more than one address is available. An address list without interface names does not establish which interface it belongs to; keep that association unknown rather than guessing from its prefix or position. These views do not test routing or connectivity.

Do not remove a working bridge, renumber the guest, reinstall or add an agent, recreate the monitored connection, or run a guest-agent probe just to change this display. For a report, retain the collection path and the interface/address relationship already observed. Keep actual addresses, MACs and hostnames private, using consistent aliases to preserve which address belongs to which interface; a raw network or configuration dump is not needed.

Agent fleet update or identity issue

  • If readings became mixed after adding another Proxmox installation, start with the cross-installation identity checks, not an agent update or re-enrolment command.
  • Open an outdated-agent notice or /settings/infrastructure?agentDoctor=1 to open Agent Doctor and copy the platform-specific command for each reported host. This is a manual handoff; Pulse does not remotely execute the command.
  • Administrators can call the read-only Agent Fleet Doctor endpoint, GET /api/agents/diagnostics, to inspect liveness, version drift, profile deployment drift, expected telemetry gaps, and identity-split evidence. It does not change agent configuration or enqueue a repair.
  • A current Pulse server does not prove fleet convergence. Eligible v6 agents update asynchronously; v5, PVE, disabled, and failed updates require manual handling.

Removed Pulse server but pulse-agent still logs connection failures

Removing the Pulse server does not remove agent services installed on monitored hosts. On a systemd host, stop and disable the orphaned service to halt retries:

sudo systemctl disable --now pulse-agent.service

If the Pulse server is still reachable, use its generated uninstall command so the agent can deregister cleanly. If it is unreachable or the saved server URL is missing, stopping retries is not completed removal or credential revocation. Keep the agent's saved identity, credentials, connection files and needed logs private and intact; do not reinstall, create a new token or delete state to bypass a removal refusal. Temporary loss of connectivity does not establish that the server is gone. Follow the saved-state removal guidance before local cleanup; Unix credential-bearing removal needs confirmation from the same trusted Pulse instance. The stop command above is for systemd, not Windows or other service managers.

Filter Pulse's systemd journal by severity

Current systemd installs preserve Pulse's structured log level as the journal priority. For warnings and more important records, use this bounded Bash reader on the Pulse host, with an account authorised to read its journal. In Proxmox LXC, run it inside the Pulse container, not on the Proxmox host. Substitute the actual service name (pulse-backend on some older installs) and original incident's time window; no new action or request ID is needed.

# systemd / Proxmox LXC — warning and higher priority
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)

This requires GNU timeout and the listed local utilities, reads at most 200 records from the last 15 minutes, and limits the excerpt to 256 KiB. The eight-second hard deadline covers the reader and byte limiter together. A failed, oversized or timed-out read is unavailable, not an empty warning search; no partial excerpt is shown. Stop if a utility or journal access is unavailable; do not use an unbounded substitute. These local excerpts are not sanitised; follow log-sharing precautions before sharing any line.

The stored message remains JSON ("level":"warn", for example), while PRIORITY is available to journalctl and syslog forwarding. An empty filtered read does not prove that no warning occurred: an older generated unit may store all records at info priority. Use the bounded unfiltered systemd reader to retain the original incident evidence instead. An administrator can check locally whether the service configuration contains SyslogLevelPrefix=true and PULSE_LOG_JOURNAL_LEVEL_PREFIX=true; do not post the full unit or environment, which can contain credentials. Do not reinstall or restart Pulse just to obtain logs. Repair an older generated unit through the current signed installer at a planned maintenance window, preserving the deployment's settings and data.

VMs show "-" for disk usage

  • Start with current, retained and unavailable disk readings. A dash is not zero; a Prior number is not current free space. Clear numeric disk conditions to inspect missing readings, not to change collection or access. A filtered list or disk sort is not proof of complete monitoring.
  • Check the guest-local service, current VM Options and the configured API token's read permissions. Schedule any setup change or restart outside backups.
  • Do not run guest-agent probes during backup freeze/thaw. An OK backup task or an absent lock does not confirm thaw.
  • See VM Disk Monitoring for the passive host preflight; it does not verify a fresh disk poll. For an affected installation, follow the manual backup precaution for your actual server deployment: systemd or Docker/Compose. A planned pause is not incident recovery, and stopping Pulse does not cancel a guest-agent request already issued. If an existing operation's state is unknown, do not start a backup on the strength of a stopped server or an elapsed wait.
  • Keep Pulse stopped until the backup has ended and independent post-backup checks confirm thaw, fresh successful workload writes to every filesystem covered by the backup, and workload liveness. Use established safe checks, independent of Pulse and the QEMU Guest Agent; a console connection or a successful read alone is not enough. If any check fails or is unavailable, leave Pulse and its automatic updater paused and use the guest/platform's recovery procedure, not new probes, forced writes or another backup.
  • After all checks pass, restore only services and timers that were active before the pause, following the deployment-specific precaution. Unknown pre-pause states are not permission to start them. Pulse monitoring and alerts are unavailable while stopped; arrange independent outage coverage.

Backup-age alerts were not evaluated

The Pulse warning “Backup-age alerts were not evaluated because recovery data could not be read” means that Pulse could not complete that backup-age check. It is a monitoring failure, not evidence that a backup failed. Existing backup-age alerts are kept during the failed read; they are not a fresh assessment, and the absence of a new age alert does not establish that backups are current. A working connection or a visible backup row does not prove that age evaluation completed.

Use the owning Proxmox/PBS installation's existing backup records and your established backup checks while evaluation is unavailable. Compare the same installation, guest type/ID, datastore, namespace and backup time; a matching VMID in another installation is not the same guest. For a separate Coverage disagreement, use the backup health checks.

Keep the warning's original time and message. On the actual Pulse server, use the existing bounded log reader for the incident window, with the matching systemd or Docker deployment. The log message Failed to list recovery rollups for backup alerts can identify a failed evaluation read; a timeout does not by itself establish its cause. A failed log read is unavailable, not an empty successful search. Share only the running Pulse version, affected time window, warning and relevant manually redacted error. Keep credentials, private paths/hostnames, full logs, API responses and database files out of public reports.

A disappearing warning alone is not recovery. Disabling alerts or backup-age checks, or setting both age thresholds to zero, can clear it without a successful evaluation. Keep the intended policy unchanged and observe ordinary polling; do not disable protection just to clear the warning. A successful age evaluation still does not establish backup completeness, verification, restore readiness or guest thaw.

Do not restart or reinstall Pulse, clear History, delete/prune backup records, change database schema or increase timeouts just to diagnose this warning. Do not run a new backup, verification, restore, live diagnostics or guest-agent probe as a test. An OK backup task does not prove guest thaw; a frozen or unresponsive guest needs the separate backup safety procedure, including independent post-backup thaw, successful writes to every covered filesystem and workload liveness before restoring only previously active monitoring.

Backup health disagrees with PBS

A visible or Verified PBS backup is not the same reading as a workload's Coverage posture. Use the backup health checks to compare one affected row's explanation, Job, History and Access with the matching native PBS record. Do not run a new backup, restart or clear history just to diagnose the disagreement.

Replication jobs are Pending, stale or missing

Proxmox → Replication monitors storage replication between PVE nodes, not PBS backups. A replication result does not establish backup coverage, restore readiness or guest thaw. Use the owning Proxmox installation's existing replication status while Pulse's reading is uncertain.

Start with All and clear the table's search. No replication jobs match current filters is different from an empty inventory, a missing tab or Could not load replication jobs. Record the actual message; a missing row is not proof that Proxmox deleted the job.

Observation in Pulse What to compare What it does not establish
Pending, with current sync times The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success.
Old sync times or an overdue next sync The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler.
Empty inventory, missing tab or load error Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery.

For one affected job, expand its row using the control beside Guest; the details retain Job, Route, Last sync, Next sync, Duration and Failures even when narrow layouts hide table columns. Compare the owning installation, job/guest ID and source → target route with the existing native Proxmox replication view. Similar names or VMIDs in separate installations do not identify the same job. An unavailable native observation can be reported as unavailable; do not force a sync to obtain one.

Treat classification and stale or absent inventory after a lifecycle change as separate symptoms. If a previous restart restored data, keep that observation and the now-working setup; do not repeat a restart, reboot or update to reproduce it. One current row does not prove every job recovered.

For help, give the running Pulse/PVE versions, container image tag if relevant, the original sequence and time, which symptoms remain, and one matching job's Pulse versus native status, sync times, failures and redacted error. Include any already-observed recovery, without recreating missing original evidence. Use consistent placeholders for private installations, nodes and guest/job identities so the route remains comparable. Do not share tokens, full API responses, configuration, HAR exports or unredacted screenshots.

Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable jobs, recreate connections, widen token permissions or disable privilege separation just to diagnose the table. Do not run a new sync, backup, restore, live diagnostics or guest-agent probe as a test. A frozen or unresponsive guest needs the separate backup safety procedure, not these display checks.

Monitoring is mixed between Proxmox installations

Pulse is intended to monitor multiple Proxmox installations. Separate installations can reuse native node names and VMIDs. A distinct Pulse connection label or display name is not a repair for wrong attribution.

If adding a connection led to node errors on the wrong node, unavailable backup status for overlapping VMIDs, or lost agent-backed Docker monitoring, treat each as an unresolved symptom. One restored view does not establish that the others recovered. Use existing observations only; if you already repaired the setup, keep that working setup and the original sequence rather than undoing the repair or repeating the addition.

Affected reading Safe comparison What it does not prove
Node errors Compare the affected node and observation time with that installation's own Proxmox view. A similar node name or a different Pulse label does not establish which installation owns the error.
Guest backup status Compare one affected guest with the matching native Proxmox/PBS record: owning installation, guest type/VMID, datastore, namespace and backup time. Follow the PBS evidence checks. The same VMID or a visible backup from another installation does not establish protection for this guest.
Agent-backed Docker monitoring In Settings → Infrastructure → Agent Doctor, inspect the affected host's existing Last seen and Identity evidence, including Connection and Hostname where available. Compare with the known Docker host, not a similar guest name. Recent agent contact or a Healthy badge does not prove that the correct host's containers are visible or correctly attributed. Missing evidence is unknown, not proof that the agent stopped.

While attribution is uncertain, use each installation's own Proxmox/PBS views for node errors and backup status, and the Docker host's existing runtime view for container status. Do not make backup, restore or workload changes from a potentially misattributed Pulse reading.

For a report, give the running server/agent versions, the original sequence, which of the three symptoms remain, and one affected pair's expected versus displayed origin and observation times where available. Use consistent placeholders such as site-A, site-B, node-X and guest-100 across every view; say explicitly when the native names or VMIDs are equal. Keep actual addresses, hostnames, connection IDs and machine identities private. Do not share a full Agent Doctor report, configuration, API response, HAR export, token or unredacted screenshot. Unavailable original evidence can be reported as unavailable; do not recreate the incident to obtain it.

Do not rename production nodes, change VMIDs or machine IDs, delete/re-add connections, re-enrol agents, rotate tokens, restart services or clear History just to diagnose this disagreement. Do not run live diagnostics, guest-agent probes, a new backup or a restore as an identity test. An OK backup does not prove thaw; a frozen or unresponsive guest needs the separate backup safety procedure, not these display checks.

Temperature data missing

  • Compare the affected host's active agent version, last report, sensor and observation time. A current server or another sensor's value is not evidence that this reading is available.
  • Use the bounded local reading check on the monitored host, not the Pulse container. Linux agents can use existing recognised CPU/SoC sysfs readings without lm-sensors; unavailable is not zero.
  • Do not run automatic hardware detection, load drivers, reboot or loosen SSH restrictions merely to fill a temperature row. Retain output privately and follow the platform-specific guide for any needed setup change during a maintenance window.

Container CPU differs from Docker or Podman stats

Pulse uses the runtime host's total CPU capacity; docker stats and podman stats normally use 100% per logical CPU. Divide a per-CPU stats reading by the reported runtime host CPU count, not the physical Proxmox host's count when the engine runs inside a VM. Compare the same container and time interval in CPU History, not just a rounded table label. A dash or missing History is unavailable, not zero; an Active connection alone is not fresh CPU evidence.

See container CPU readings for worked examples, rounding and alert limits, and a passive comparison. Do not restart workloads, create CPU load or lower alert thresholds to test this.

Docker hosts appearing/disappearing

Cloned hosts can share a saved Pulse agent ID, not just an OS machine ID. The agent uses an explicit ID first, then its saved agent-id file, and derives one from the machine only when neither is available. Changing a hostname, IP or /etc/machine-id therefore does not necessarily change its Pulse identity.

Compare the affected hosts in Agent Doctor and inspect only their configured agent-id files locally. Do not delete the OS machine ID, agent state or Pulse history as a troubleshooting step. If a duplicate is confirmed, give only the clone a stable, unique ID in its managed service or container configuration; leave the original host unchanged. Follow Clone identity recovery for the configuration precedence, systemd example and checks after restart.

Guest memory is high but available memory is plentiful

A VM's hypervisor memory footprint is not the same as usable memory inside the guest. Some Proxmox API readings include reclaimable cache or allocated memory; a near-100% value alone does not establish guest pressure or a leak. Conversely, a cache-heavy reading does not prove the guest is healthy. Keep an unresponsive workload or genuine allocation failure separate from a misleading percentage.

Compare existing observations of the same guest, kernel and time, with the same units and memory allocation. The Proxmox node's available memory is not the VM's available memory, and a single application's RSS is not total guest usage. Do not sum process RSS to infer guest pressure: processes can share pages.

For History comparisons, keep each panel's stored time and qualification with the value; a range endpoint, page refresh or today's live source does not date a past reading. Use History panel interpretation to distinguish a missing observation from zero, and stored data from fallback.

Reading What it describes What it does not establish
Linux available / MemAvailable An estimate of memory usable for new allocations without swapping, including some reclaimable cache. It is not just free, a guarantee of every allocation succeeding, or a reading for another guest.
Linux free and buff/cache Unused pages and memory used for buffers/cache. Cache can be useful and partly reclaimable. Low free alone is not pressure; not all cache is immediately reclaimable.
Proxmox status memory The platform's selected footprint or estimate. The collection path matters. A cache-inclusive or allocation-based percentage is not equivalent to cache-aware guest usage.
Application RSS Resident pages for that process at that time. It is not total guest usage, Pulse's heap size, or proof that no other process is under pressure.
Missing, retained or undated reading Incomplete evidence, or an earlier observation. Missing is not zero; a prior low value or a fresh page refresh is not current recovery.

For a same-guest, same-time Linux example with 20 GiB total and 15.2 GiB available, cache-aware usage is (20 - 15.2) / 20 × 100 = 24%. Use the observed total, not a nominal configured limit, and do not mix GB with GiB or kernel kB (1,024 bytes). This illustrates the distinction; it is not a correction to apply blindly to another metric or evidence that an existing Patrol finding has cleared. Do not subtract all buff/cache from total, or substitute free for available.

Keep the running Pulse/agent and platform versions, collection path (platform API, linked Pulse agent, or both), source and original observation time where shown. A working agent reading does not prove that the API-only path recovered. Unknown source, time or counters can be reported as unknown; existing screenshots and the original sequence are enough to begin a report. Do not publish raw API responses, full /proc output, configuration or private host names; review and share only the relevant counters with consistent aliases.

Memory metric Off disables that threshold rule only: it does not hide the reading, dismiss a separate Patrol finding or establish recovery. Do not disable all monitoring or change thresholds merely to make a comparison pass.

Use the guest's ordinary status if it is already available; do not run Diagnostics, guest-agent probes or another Patrol run, install an agent, restart the guest, drop caches or change memory limits just to obtain evidence or lower the number. If the guest is frozen or unresponsive, stop these comparisons and follow backup safety, not a memory probe.

Memory use keeps growing

For a monitored VM, start with guest memory interpretation; the recipe below measures the Pulse server process, not a monitored guest.

A Proxmox LXC memory chart covers the container, not just Pulse. It can include other processes and filesystem cache; Docker's displayed memory also uses its own cache accounting. Neither is interchangeable with Pulse's resident memory (RSS), a Go heap size, or virtual address space (VSZ / VmSize). A large container reading alone does not establish a Pulse memory leak.

A higher steady baseline is also worth reporting; continuing growth is not required. First distinguish what your existing readings show:

Observation What it establishes
Comparable Pulse process RSS settles at a higher level than before with an unchanged workload A possible baseline regression, even if usage then stays flat. Keep the old and new versions, uptime and measurement scope; a plateau does not dismiss the increased requirement.
Comparable samples rise during one uninterrupted process run Growth during that interval, not by itself a leak or its cause. Keep the same PID/start time, units, workload and collection settings.
A new process uses less memory shortly after an upgrade than an older process did after days Different runs at different ages, not verified recovery or growth within one run. Startup and long-lived usage can differ.

For example, 510 MiB after a week and 420 MiB ten minutes into a new process cannot establish that steady-state memory improved. Use already available readings at comparable uptimes and workloads if you have them; otherwise keep the mismatch explicit. Do not restart, downgrade or change retention, polling or enabled features to manufacture a comparison. A startup CPU improvement is not evidence that the resident-memory baseline fell.

For a responsive Linux systemd / Proxmox LXC install, take this small, read-only sample at a normally occurring high point. Run it inside the Pulse container for LXC, not on the Proxmox host. Substitute the actual service name (pulse-backend on older installs) in both systemctl calls. Use an account already authorised to read these counters. The collector does not elevate privileges or prompt; do not add sudo inside it. A failed read is unavailable, not zero.

# systemd / Proxmox LXC: Pulse resident-memory sample
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Resident-memory sample unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if sample=$(timeout --signal=KILL 8s bash <<'PULSE_MEMORY_SAMPLE' 2>/dev/null
set -eu
date -u +'%Y-%m-%dT%H:%M:%SZ'
pid=$(systemctl show pulse --property=MainPID --value)
case "$pid" in
  ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
esac
identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
if [ -z "$identity" ]; then
  printf 'Process identity unavailable.\n' >&2; exit 1
fi
counters=$(awk '
  $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||
  $1 == "RssShmem:" || $1 == "VmSwap:" {
    if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1
    print; fields++
  }
  END { if (fields != 5) exit 1 }
' "/proc/$pid/status")
current_pid=$(systemctl show pulse --property=MainPID --value)
current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then
  printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1
fi
printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"
PULSE_MEMORY_SAMPLE
  ); then
    printf '%s\n' "$sample"
  else
    status=$?
    printf 'Resident-memory sample unavailable (exit %s); stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)

The recipe requires GNU timeout: it gives the read-only sampling commands eight seconds, then kills its own collection group, including readers that ignore TERM. Waiting for the collection shell alone can leave a reader running after it exits. This hard deadline does not stop or restart Pulse. A missing utility, timeout or reader error means the sample is unavailable; keep the failure, and do not remove the deadline or rerun it against an unresponsive installation. No timestamp, process identity, counter or raw reader error is printed unless every check succeeds; the failure reports only its exit status.

Linux labels these values kB, meaning 1,024 bytes; divide by 1,024 for MiB. VmRSS is resident process memory, split into anonymous (RssAnon), file-backed (RssFile) and shared-memory (RssShmem) pages. VmSwap is swapped private anonymous memory, not additional resident memory or all container swap. These counters are approximate and not an atomic snapshot; a nonzero RssFile is not proof of a leak, and RssAnon is not specifically the Go heap. Missing fields on an older kernel make this recipe fail rather than invent values. If it fails or Pulse stops, retain that fact; do not loosen access controls or keep retrying against an unresponsive installation.

Keep the sample time, PID/start time, running version, container memory limit, fleet size, polling interval, enabled collectors/features and whether dashboards were open. If safe, compare another naturally occurring point with the same PID and process start time; record any restart or upgrade as a different run, not evidence that the cause was fixed. For Docker, the bounded container statistics below retain the container start time and memory usage but do not measure Pulse RSS; do not assume its PID 1 is Pulse.

Do not restart Pulse, drop caches, force garbage collection, change memory limits or delete history just to obtain a lower reading. If the container is near its limit or the host is unresponsive, stop sampling and prioritise safe recovery. Do not post a heap dump, profile, full /proc status or process command line: they can contain private details. Share only these counters and the relevant context after local review; existing screenshots remain useful if collection is unsafe.

Excessive CPU, writes or database growth

First distinguish Pulse server activity, agent activity and total host or storage-device activity. A database's size is retained space, not the number of bytes written: repeated updates can produce high writes without growing the file. Keep the original running version, uptime, measurement time and workload before changing anything. A different binary can migrate persistent data; switching back to an older version does not make that data a clean older-version baseline.

  • For CPU, record the measuring tool, allocated CPUs or container quota and whether its percentage represents one core or the whole allocation. A ps %CPU value is an average since process start, not a recent sampling window.
  • Note fleet size, polling interval and whether Pulse dashboards are open. If safe, close all Pulse tabs and compare another equal-length window, without restarting the server or changing polling. A decrease narrows the workload; it does not establish the cause or prove monitoring is healthy.
  • If the host is unresponsive or storage is nearly full, do not prolong the failure to collect a benchmark. Keep existing observations and report the interruption instead. Do not enable Debug or repeatedly export diagnostics merely to measure performance; those actions add work and can change the result.

For a responsive Linux systemd / Proxmox LXC install, the following reads two process-I/O samples, waiting 60 seconds between them. Run it inside the Pulse container for LXC, not on the Proxmox host. Substitute the actual service name (pulse-backend on some older installs) in both systemctl calls. Use an account already permitted to read the process counters. The collector does not elevate privileges or prompt: an unprivileged deadline cannot reliably stop a privileged reader. If access is denied, stop rather than changing privileges to make it pass; no service restart or database access is needed.

# systemd / Proxmox LXC: bounded process-write samples
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if samples=$(timeout --signal=KILL 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null
set -eu
read_sample() {
  set -eu
  pid=$(systemctl show pulse --property=MainPID --value)
  case "$pid" in
    ''|0|*[!0-9]*) exit 1 ;;
  esac
  identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
  [ -n "$identity" ] || exit 1
  start_ticks=$(awk '
    { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
    END { if (found != 1) exit 1 }
  ' "/proc/$pid/stat")
  timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
  counters=$(awk '
    $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {
      if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1
      print; fields++
    }
    END { if (fields != 2) exit 1 }
  ' "/proc/$pid/io")
  current_pid=$(systemctl show pulse --property=MainPID --value)
  current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
  current_ticks=$(awk '
    { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
    END { if (found != 1) exit 1 }
  ' "/proc/$pid/stat")
  [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&
    [ "$start_ticks" = "$current_ticks" ] || exit 1
  printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \
    "$identity" "$start_ticks" "$timestamp" "$counters"
}
first=$(read_sample)
sleep 60
second=$(read_sample)
[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
printf '%s\n\n%s\n' "$first" "$second"
PULSE_WRITE_SAMPLES
  ); then
    printf '%s\n' "$samples"
  else
    status=$?
    printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)

Each recipe requires GNU timeout and limits the complete two-sample collection, including the 60-second wait, to 80 seconds. At the deadline it kills its own read-only collection group, including readers that ignore TERM, not Pulse or Docker. This deliberate hard stop prevents a reader surviving after the collection shell exits. A missing utility, permission failure, timeout or changed identity makes the pair unavailable; no partial identity, counter or raw reader error is printed, and there is no unbounded fallback. Stop sampling on failure rather than removing the deadline or retrying against an unresponsive installation.

The process recipe checks the same PID and process start time before and after each read and across both samples. Linux start ticks also distinguish PID reuse within one displayed second. These checks do not make the counters atomic. Compare write_bytes only when no restart occurred and the counter did not decrease. Divide the byte difference by the actual elapsed seconds. This is storage-accounted process I/O, not filesystem growth or physical SSD wear; cancelled writes and background writeback can differ from device measurements. Keep cancelled_write_bytes alongside it, not as proof of bytes reaching the drive. An extrapolated GB/day rate is a projection of that short window, not a measured day's total. Preserve the window and units with the result.

For Docker / Compose, run this on the Docker host, replacing pulse with the running container name. It reads only the start time and selected statistics, not the container environment or configuration. Each stats call targets the full container ID, not a reusable name. The name must still resolve to the same running ID, start time and restart count after the read and across both samples; a replacement, restart or stopped container makes the pair unavailable.

# Docker: bounded container statistics
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if samples=$(timeout --signal=KILL 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null
set -eu
read_sample() {
  set -eu
  identity=$(docker inspect --type container --format \
    '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
  case "$identity" in *$'\n'*) exit 1 ;; esac
  read -r id started running restarts extra <<< "$identity"
  [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1
  case "$id" in *[!0-9a-f]*) exit 1 ;; esac
  case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac
  case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac
  timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
  stats=$(docker stats --no-stream --format \
    'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")
  case "$stats" in
    *$'\n'*) exit 1 ;;
    CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;
    *) exit 1 ;;
  esac
  cpu=${stats%% *}
  [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1
  case "$stats" in *--*) exit 1 ;; esac
  current_identity=$(docker inspect --type container --format \
    '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
  [ "$identity" = "$current_identity" ] || exit 1
  printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"
}
first=$(read_sample)
sleep 60
second=$(read_sample)
[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
printf '%s\n\n%s\n' "$first" "$second"
PULSE_CONTAINER_SAMPLES
  ); then
    printf '%s\n' "$samples"
  else
    status=$?
    printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)

Docker BlockIO is cumulative read / write activity, not bytes per second; compare its write side only across the same uninterrupted container run. Its displayed units are rounded. Container, process and whole-device counters have different scopes and must not be added together or compared as interchangeable measurements. A failed, denied or empty read is unavailable, not zero activity.

Keep persistent data on durable storage. Do not delete or truncate history, incident, queue or audit files, remove database indexes, or move the data directory to tmpfs as a diagnostic workaround. These actions can lose evidence or protections without fixing the writer. Do not post databases, profiles, full /proc dumps, container configuration or raw environments. Share only the bounded measurements, workload, versions and a relevant manually redacted error; follow Getting Help for any additional evidence.

Notifications

No alert when Pulse, power or internet goes down

Pulse cannot send a notification while its host is stopped or its outbound network is unavailable. A local delivery-health warning is not an external outage detector, and a successful Send test does not prove outage coverage.

  • In Alerts → Notifications → External watchdog, configure a Healthchecks-compatible success ping URL and save the configuration. Keep the URL secret; do not include it in screenshots or support reports.
  • Run the watchdog outside the failure you want to detect. Another machine on the same power supply or internet connection does not cover a whole-site outage. Its notification destination must also remain reachable independently of that site.
  • At the watchdog, configure a one-minute period and a three-minute grace period, and enable its notification integration. Pulse sends a heartbeat every minute and a /fail signal if its monitoring loop stalls. For Healthchecks simple schedules, a missing heartbeat becomes down after period plus grace: approximately four minutes after the last success, not three. Recipient delivery can take longer. See Healthchecks timing and notification concepts.
  • Verify in an authorised test environment: confirm incoming heartbeats at the watchdog, stop the test Pulse instance, and check that the intended recipient actually receives the external alert. Restore Pulse and verify heartbeat recovery. Test loss of internet separately if you need that coverage; a stopped-process test does not prove it.
  • To retire the check, pause or remove it at the watchdog too. Clearing the URL in Pulse stops its signals but does not pause the remote check.

A healthy heartbeat indicates Pulse monitoring-loop progress, not successful delivery of every resource alert or external reachability of your services. Continue checking delivery activity for destination failures.

Test succeeds but real alerts are missing

A test sends directly to its destination: it skips the persistent delivery queue and is not listed in Recent delivery activity. It does not prove that a real alert was generated, routed or delivered to the intended recipient.

  • Open Alerts → Notifications. If Notifications are paused is shown, configured destinations and a successful test do not enable real delivery. Turn delivery on there only when you intend to send alerts.
  • Check the affected alert, the destination's Enabled state and minimum severity and tag filters. Both must match: an empty tag filter does not bypass minimum severity, and critical severity does not bypass a nonempty tag filter. Review the quiet-hours schedule and acknowledgement or snooze, plus any mute or maintenance policy, before treating an absent attempt as a transport failure. In Alerts → Overview, use Show acknowledged to find hidden incidents; Resume ends a snooze but does not remove acknowledgement.
  • For a missing repeat or all-clear, review alert reminders and recovery notifications. Cooldown off stops ordinary reminders for the same occurrence; it does not mean unlimited repeats. Recovery messages require eligible prior firing delivery to that destination. Their absence does not establish whether the workload recovered: check fresh readings and the incident state separately.
  • Use Recent delivery activity to correlate the original alert, destination and absolute timestamp, including held-notification reasons. An empty window is not proof of healthy delivery; an unavailable read is not an empty log. Do not create an outage or repeat a notification storm to populate it.

Recover retained delivery failures

Pulse shows a delivery warning for failed or dead-lettered notifications in its persistent queue, not for every recoverable retry. Authentication, configuration and rejected failures stop automatic retries as soon as they are classified, even with attempts left. A terminal failure can therefore appear without exhausting the retry budget; waiting alone will not resend it. See webhook retry behaviour for HTTP exceptions, transient failures and the normal queued attempt limit.

Recent delivery activity masks recognised credentials in URLs, not all private information. Older records and other error sources can still include email addresses, private destinations or credentials echoed outside a URL; alert identifiers and resource names can also identify your infrastructure. Withheld webhook response text is intentional, not a reason to collect it again; see withheld-response guidance. Keep full entries, copied responses and screenshots private. A REDACTED marker does not make the rest safe to share. For a public report, extract only the relevant timestamp, delivery method, failure class, HTTP status or SMTP error code and a manually redacted error. See notification log precautions.

Completed attempts remain for 7 days and dead-letter attempts for 30 days. Start with the failure class and timestamp:

Failure Check before retrying
Authentication Destination credentials and account permissions, locally; never post them.
Rate limited Provider limits and delivery volume; repeated tests or retries can make this worse.
Connectivity DNS, firewall, proxy and reachability from the Pulse server, not just your browser.
TLS Certificate trust, expiry and hostname matching; do not disable verification to diagnose it.
Configuration / rejected Enabled destination, required fields and the provider's endpoint or payload requirements.
Server error / unknown Destination service status and a relevant, bounded local error excerpt.

Save the corrected destination settings and send one test; check receipt at the intended destination. Retained deliveries keep the destination settings saved when they were queued. Editing a URL, recipient, credential, header or template does not replace that saved configuration. A test uses the edited settings; retrying an old delivery can still use the old endpoint or credential and fail again. If the old destination must no longer receive data, disable it rather than relying on a URL edit to redirect queued work. This is not a recall of requests already sent; see the pause and cancellation limits.

Retry retained deliveries gives all retained terminal failures a fresh retry budget, not just the destination you tested. Use it only when sending those original deliveries is still intended and their original settings remain appropriate, for example after a temporary provider outage. A destination that accepted an earlier attempt may receive a duplicate. Review the confirmation's delivery count and provider limits before retrying. A successful test does not itself retry retained items or prove that their saved settings now work. If uncertain, leave the failures retained and check the next normally occurring alert instead; do not use a batch retry to test a settings edit.

Use Dismiss retained failures only when those deliveries should not be sent. Dismissal clears the warning without retrying them; delivery history remains. Neither action deletes the audit trail. Do not delete notification_queue.db or audit data to clear the warning.

Pause and cancellation limits

Turning off the global Send notifications switch on the Alerts overview is not a store-and-forward hold. Monitoring continues, but pausing clears buffered alert groups and cancels pending queue deliveries for email, webhooks and Apprise, including recovery deliveries. Turning delivery back on does not replay those cancelled deliveries. Retained terminal failures are separate: pausing does not dismiss them or repair their saved destination settings.

Disabling or removing an individual destination prevents a queued delivery that sees that policy from sending. A policy-skipped item is cancelled, not a successful provider delivery, and Retry retained deliveries does not replay cancelled items. Re-enabling a destination is not proof of catch-up or receipt. For a scheduled quiet period, use Quiet hours instead, reviewing its critical-alert exceptions and later re-evaluation; it still does not guarantee that every held message will be sent.

Pause, disable and removal do not guarantee cancellation of a request already in flight. The receiver may accept it after the settings change. Neither a paused banner nor a cancelled queue row proves that the destination received no data, and Pulse cannot recall a message or delete the receiver's copy. Reconcile the original time with Recent delivery activity and the receiver's existing record, keeping both private; do not send another Test or replay a queue to check containment. If a destination credential was exposed, revoke it with the provider too: editing its URL or turning delivery off is not credential revocation and cannot undo an earlier disclosure.

Emails not sending

Follow retained-failure recovery first. Check SMTP host, port, sender, recipients, authentication and TLS settings in Alerts → Notifications against your provider's requirements (some providers require an app password). Keep passwords in the settings form, not a diagnostic command or report. If the delivery error is insufficient, inspect bounded notification logs locally.

Webhooks failing

Follow retained-failure recovery first. Use the failure class and HTTP status to check the provider's endpoint and payload requirements. Verify reachability from the Pulse server. For an intended private destination, review Settings → System → Network → Webhook Security; do not broadly weaken network or TLS controls just to make a test pass.

Prefer the redacted delivery error over raw provider response bodies. A provider can echo credentials or private content in its response; do not post it wholesale or enable debug logging just to collect it. If needed, inspect bounded notification logs and share only the consequential, manually redacted error.

Webhook response body is withheld

response body withheld, details withheld and Response body withheld are deliberate privacy messages, not the receiver's explanation of a failure. Alert-webhook receivers can echo tokens, alert content or private provider data in their replies. Pulse discards that response text instead of copying it into delivery errors, logs or a Test result. Do not enable Debug, capture raw traffic, weaken a safeguard or send another Test or queue retry to recover the text.

Use the original timestamp, destination type, failure class and HTTP status from Recent delivery activity. An HTTP rejection still matters when its body is withheld; withholding alone does not mean that delivery failed. For example, HTTP 429 is still rate limiting, not missing diagnostic data. If the error says failed to read response body, reading the reply failed: the receiver may already have accepted the request, so retrying can duplicate it. Neither HTTP 2xx nor a completed Test proves receipt at the intended destination or the success of queued alert delivery.

A Test summary such as Response body withheld (128 bytes read) records only how much Pulse read, not the response's contents or total size. read incomplete means the read failed; read limit reached means it reached the 1 MiB cap, not proof that the response exceeded that cap. A complete empty reply may have no summary. These are not byte counts of the alert Pulse sent.

If provider-specific detail is consequential, review the receiver's existing record locally using its normal authorised access. Share only the relevant error code or manually redacted explanation, not response bodies, headers, payloads or credentials. Unknown receipt remains unknown; do not replay the request to create evidence. Older logs and retained delivery errors are not retroactively scrubbed, and this safeguard does not sanitise every other error or export. Keep those private and follow the log and export precautions.

Telegram Test works but real alerts say "message text is empty"

With the built-in Telegram template (no custom template), a blank or incorrect saved Content-Type can make Telegram ignore the JSON body. The HTTP 400 error does not by itself prove that Pulse omitted the text field. Test can supply the correct header even when real delivery uses the saved one.

In Alerts → Notifications, edit that Telegram destination's Custom headers: set one static Content-Type header to application/json and remove any conflicting duplicate, including differently capitalised names. Save, leaving the bot URL, chat_id, template and grouping settings unchanged. Check whether the next normally occurring alert reaches the intended chat and compare its timestamp with Recent delivery activity; do not induce an alert or retry the old rejected batch to test this edit. Retained jobs keep their original settings as explained above.

For a custom template, check its JSON and required Telegram fields locally instead of assuming this built-in-template workaround applies. If reporting a continuing failure, share only the version, redacted delivery error, timestamp and whether a custom template is used. Keep the bot URL/token, chat ID and full notification configuration private.

TrueNAS

"TrueNAS service unavailable"

The exact truenas_unavailable error (HTTP 500 or 503) concerns Pulse's connection-management handler or configuration persistence, not proof of an appliance outage or an invalid TrueNAS API key. Keep the existing request's status/code and time, then inspect a bounded Pulse log excerpt locally. Do not rotate the key, recreate the connection or weaken TLS verification to clear it.

Follow TrueNAS error diagnosis to separate this from an explicitly disabled integration or a failed live test. For missing or stale readings, use the polling checks; a successful Test Connection does not prove collection has recovered.

TrueNAS pools/datasets not appearing

  • TrueNAS data appears in the unified resource model and may take one configured polling cycle (60 seconds by default) to appear.
  • Open TrueNAS → Overview for the appliance, TrueNAS → Storage (/truenas/storage) for pools, datasets and disks, and TrueNAS → Protection (/truenas/protection) for snapshots and replication. These are tabs within TrueNAS, not separate top-level Storage or Recovery pages.
  • For data that stops refreshing, use the TrueNAS polling checks before testing or restarting. A stale badge is not proof of an invalid key, and a successful connection test is not proof that collection has recovered.

Navigation (v6)

Old bookmarks don't work

Current Pulse uses platform navigation. /proxmox, /docker and /kubernetes are supported; do not replace them with the retired task-based routes. Open Pulse at its base URL and use the menu to find the relevant page:

Menu Entry route
Proxmox /proxmox/overview
Docker /docker/overview
Kubernetes /kubernetes/overview
TrueNAS /truenas/overview
vSphere /vmware/overview
Machines /standalone/machines

The short-lived top-level /workloads, /storage and /recovery layout is retired. /infrastructure now opens the default workspace, not the former unified host page. For old /hosts or /services bookmarks, select the current platform or Machines page instead of assuming an automatic redirect. PBS backups are under Proxmox → Backups, and TrueNAS snapshots and replication are under TrueNAS → Protection.

Platform-connected hosts can appear under their platform rather than Machines. If a menu or expected resource is missing, check its saved connection and last successful collection in Settings → Infrastructure; a missing page alone does not prove a host was deleted. Do not delete connections or re-enrol agents just to repair a bookmark.

If menu navigation works but reloading the same URL returns a proxy 404, check the proxy's route handling using Reverse Proxy Configuration. For the current layout, see FAQ. The unified-navigation migration is historical, not a guide to the current menu.

Relay / Mobile

Relay showing "Disconnected"

Pulse Mobile and Relay retire on 31 March 2027; existing paired phones keep working until then. Relay is no longer sold. Existing Relay subscribers keep Pro features at their current price for as long as their subscription continues, including after the app retires. Do not buy or switch plans to diagnose a connection failure.

For an existing paired phone, use the Relay connection checks. Keep the pairing and current configuration; a relay connection is not proof that the phone has connected. Relay connects the app, not the web UI. For phone alerts after retirement, configure ntfy, Gotify or Pushover under Alerts → Notifications and open Pulse in your phone's browser, using your own VPN or tunnel when away from home.


🛠️ Advanced Diagnostics

Collect diagnostics safely

Open Settings → Support → Diagnostics & Health in your existing authorised Pulse session (/settings/support/diagnostics on your Pulse URL). Opening this panel does not run diagnostics. If no result is displayed, the download buttons are absent: use the original errors or observations rather than running checks just to obtain an attachment. Diagnostics are optional, not a condition of reporting.

Run Diagnostics (shown as Run on phones) is not just a passive export. It can make live Proxmox/PBS API and guest-agent requests; results may also come from a short-lived cache. Do not run it during a backup, freeze/thaw or an unresponsive-host incident merely to obtain a report. Keep the original errors and existing observations instead. A successful one-off check does not prove that normal monitoring has recovered or that a guest has thawed.

After a result is displayed, the download buttons reuse that result without running the checks again. They save a local JSON file, not an upload:

  • Full (private) retains identifying diagnostic details. Keep it private; do not attach it to a public issue or discussion.
  • GitHub (review first) replaces selected infrastructure and token identifiers, private filesystem paths and raw disk-response fields. Counts, measurements, collection times and diagnostic states remain useful for triage. It is not a guarantee that every free-text error or future field is free of private information. Open the file locally and review it before sharing.

Remove credentials, cookies, secret URLs, private host/network or personal information, including details echoed in errors or notes. Do not paste a Copy as cURL command, full network export, configuration or data directory. Share only evidence relevant to the symptom; a screenshot or exact redacted error may be enough. See Getting Help.

The Support Bundle download in Settings → Support → System Logs is a different export: it can contain a complete log file or server buffer plus configuration and environment information. It is not the GitHub (review first) diagnostics export and is not limited to the lines visible in the panel. Keep the archive private and share only relevant, manually reviewed, redacted excerpts. See the log-control and export limits.

If you already posted private information

Treat a publicly posted credential as exposed, even if you later edit or delete the post. Do not paste it again to identify it or test whether it still works.

  • Revoke access at its source. Use a trusted administrator session to revoke the affected credential with its issuer. Manage Pulse API tokens in Settings → API Access; Proxmox/PBS tokens and notification-provider secrets need revocation at their respective providers. Removing a Pulse connection or stopping an agent is not credential revocation. A session cookie needs the authentication system's session-revocation procedure; clearing local browser cookies is not server-side revocation. Check the issuer's confirmation, not just a dismissed dialog. Revocation can interrupt monitoring or notifications; arrange independent coverage and configure legitimate consumers privately with a replacement carrying no wider permissions.
  • Remove the public copy too. Edit the affected text and remove the attached file or screenshot. Editing the text alone does not remove an attachment. Ask GitHub for help with content you cannot remove yourself, but do not wait for removal before revoking access. Copies, notification emails and attachment links may survive; deleting a post is not proof that the information is erased.
  • Keep the remaining evidence private. Preserve the original locally, not in another public upload. Public follow-up needs only the time, affected version, type of exposed information and relevant redacted symptoms. For sensitive security details, use the private disclosure route, not another issue or discussion. Do not send the credential itself or upload an unreviewed replacement bundle. If only private identifying details were exposed, seek GitHub's help with removal of copies you cannot remove yourself; that alone is not a reason to rotate unrelated credentials.

Inspect Notification Logs

For notification failures, prefer Recent delivery activity in Alerts → Notifications. These bounded Pulse log readers also apply to other server errors; a request ID is not required. If a local log is needed, run only the command for your deployment, on the Pulse host with an account authorised to read its logs. For Proxmox LXC, run the systemd command inside the Pulse container, not on the Proxmox host. Adjust the time window to the original incident and substitute your actual service or container name (pulse-backend on some older systemd installs). These examples read at most 200 records from the last 15 minutes; they do not follow the log or send a test. Each requires GNU timeout and the listed local utilities. A 256 KiB byte limit complements the record limit: a single log record can be large. The eight-second hard deadline kills only the read-only collection group, including the reader and byte limiter, not Pulse or Docker. If a required utility is unavailable, stop rather than running an unbounded substitute. These are Bash examples; the enclosing subshell keeps an unavailable read from exiting your interactive shell.

# systemd / Proxmox LXC
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
# Docker
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
docker logs --since 15m --tail 200 pulse 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)

Application logs can be written to stdout or stderr; these commands collect both into one owner-only temporary file, without preserving stdout/stderr attribution, and display it only after the reader completes successfully and the byte limit is checked. The file holds at most 256 KiB plus one byte used to detect overflow; it is removed when the subshell exits. A failed, oversized or timed-out read shows no partial excerpt that could be mistaken for complete evidence. If the shell is forcibly interrupted, an owner-only pulse-log.* file may remain in its temporary directory; remove that exact file privately, not unrelated logs or deployment data. Do not pipe the reader into grep email: it can miss SMTP or webhook errors and hide a failed read behind a matching partial line. A nonzero reader exit, access error or missing service/container is a failed read, not “no delivery errors”. Even a successful empty read is inconclusive: the window, retained logs or selected instance may differ.

These local excerpts are not sanitised. Do not post them wholesale. Share only the relevant timestamp, method, HTTP status or SMTP error code and a manually redacted error. Remove credentials, cookies, secret URLs, addresses and private host or personal information, including anything echoed by the provider. Never upload full environments, configuration, a queue database or audit data. See Getting Help.

Correlate Logs with Requests

For a failed HTTP API request, inspect its response in your authenticated browser's Developer tools → Network panel. Copy only the X-Request-ID response header, if present, and keep the HTTP status and time. Do not copy a session cookie, Copy as cURL command or full network export into a report. WebSocket upgrades do not pass through this request-ID middleware.

Service logs normally use JSON ("request_id":"abc123"); console logs may use request_id=abc123. Search for the literal ID value so both formats work. Replace abc123 below with the response's ID. Run only the command for your deployment, on the Pulse host using an account authorised to read its logs. These examples limit collection to the last 15 minutes and 1,000 lines; adjust the time window to the original incident rather than repeating the failed action. Like the notification-log examples above, these require GNU timeout and the listed local utilities, enforce an eight-second hard deadline and a 256 KiB byte limit, and search only a successfully completed read. They use an owner-only temporary file with the same cleanup precautions. A deadline, overflow or failure shows no partial match; a missing utility does not fall back to an unbounded reader. Keep the original response status, time and ID if the chosen window is too large; do not raise the limit or repeat the failed action to fill that gap.

# systemd / Proxmox LXC
(
umask 077
for tool in timeout bash head mktemp wc rm grep; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
# Docker
(
umask 077
for tool in timeout bash head mktemp wc rm grep; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
docker logs --since 15m --tail 1000 pulse 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)

A log-reader failure is not an empty search result: resolve any access or container/service error locally first. Exit 137 can mean the hard deadline killed the collection group; an oversized excerpt is also unavailable. The byte limiter can close the pipe before the reader finishes, producing a nonzero reader exit such as 141. These are collection results, not proof that Pulse itself is hung. Do not restart Pulse or Docker, enable Debug, widen collection or repeat the failed action merely to obtain logs. Even a successful read with no match does not prove the request succeeded. At the default log level, this middleware logs HTTP 5xx failures but not successful requests; HTTP 4xx failures are logged at debug level. The selected window, retained logs or deployment may also differ. Keep the original response status, time and ID even when there is no matching log; do not enable debug logging or retry a state-changing request just to fill that gap.

These local excerpts are not sanitised. Before sharing a relevant line, remove credentials, cookies, secret URLs and private host, network or personal information. See Getting Help for safe evidence collection.

Check Permissions (Proxmox)

If Pulse cannot see VMs, storage or guest readings, start with the user, realm and token ID configured for the affected Pulse connection, not a guessed account or your administrator login. Use the existing connection settings; do not dump the server configuration or expose the token secret.

With privilege separation enabled, the token's effective access is the intersection of the user and token permissions: both must allow the required privilege on the affected resource path. Ask the Proxmox administrator to inspect both sides, including inherited ACLs and their propagation, for that VM, storage or node. An administrator session or user-only permission listing does not prove the token has access.

First distinguish a missing privilege on an intended target from an intentional access boundary. Pulse 6.5.0 has no per-node include/exclude setting for a Proxmox API connection: cluster discovery can list nodes outside the token's authorised scope, and Pulse can still attempt to read them. Keep a department-scoped token restricted to its intended nodes; do not grant access to other departments just to silence permission errors. Table filters and alert suppression do not stop API polling. Connecting to another member of the same cluster does not limit discovery to that member. Denied readings are unavailable, not evidence that those nodes are down or unhealthy.

Inspect the existing denial and compare it with the relevant read privileges:

  • Inventory and metrics: Sys.Audit and Datastore.Audit, on the relevant node/storage scope.
  • VM filesystem usage and guest information on PVE 9+: VM.GuestAgent.Audit.
  • Guest memory through /proc/meminfo on PVE 9+: also VM.GuestAgent.FileRead.
  • On PVE 8, VM.Monitor is the legacy guest-agent fallback, not an extra requirement for PVE 9+.

The built-in PVEAuditor role cannot be modified. Where a read privilege is actually missing, use the existing narrow custom role and matching scoped user/token ACLs. Do not disable privilege separation, grant Administrator or add guest execution/write privileges to diagnose missing data. Do not rerun setup or replace a token as a permissions test: setup can rotate an existing credential. Make any necessary access repair in your normal maintenance window, outside backups, then observe normal polling without manual guest-agent probes. Sys.Modify is not a read-only monitoring privilege; see pending-update access before considering it.

Keep token secrets and full permission listings private. A report needs only the relevant missing privilege and a redacted denial, not private identities, resource paths or the complete ACL tree. Successful inventory does not prove that guest readings are fresh, or that a guest has thawed.

Rocky Linux / RHEL VMs: File-read restrictions in /etc/sysconfig/qemu-ga can explain missing guest memory; they do not by themselves establish why disk usage is absent. Review the guest's policy before changing it. Schedule any allowlist change or agent restart outside backups, following the guest OS's documentation. See VM Disk Monitoring for the distinct permissions and backup safety boundary.

Proxmox pending-update access

A successful inventory check does not establish access to the package-update list. Pulse requests GET /nodes/{node}/apt/update with its configured Proxmox credential. Proxmox's endpoint implementation requires Sys.Modify on /nodes/{node} for that GET; Sys.Audit alone is not sufficient in that implementation. Check the API requirements for your installed Proxmox version.

Sys.Modify is broader than read-only monitoring: the same privilege also authorises the separate POST that refreshes package indexes. Do not add it automatically to a monitoring role just to obtain an update badge. Keeping the narrower role and an unavailable update check is a valid choice; unavailable is not a confirmed zero. Older Pulse wording that says “Sys.Audit permission required” does not identify the actual missing privilege.

For diagnosis, compare the affected node and endpoint using the exact credential configured in Pulse, not an administrator's browser session. Keep TLS verification enabled and credentials private; report only the HTTP status and a redacted denial or whether data is empty/nonempty. Do not use POST, Refresh or Upgrade as a test. If the matching GET succeeds but Pulse remains unavailable after its next update check, report the Pulse version and displayed check time/status separately; API success alone does not confirm the Pulse display has recovered.

Recovery Mode

For a forgotten local administrator password, follow I forgot my password above to update its active credential source without deleting configuration or repeating first-time setup. For OIDC, SAML or proxy login, use the identity-provider or administrator path described there instead.

The advanced /api/security/recovery API creates a browser-bound recovery session, not a server-wide authentication bypass. Despite its legacy name, disable_auth does not disable authentication for other clients. Recovery sessions work only over direct loopback requests; remote and reverse-proxy requests cannot use them. A successful curl response does not unlock a separate browser: the session cookie belongs to the client that made the request. enable_auth clears that recovery session; it does not reset a password.

Do not transfer recovery cookies between clients or paste recovery tokens into command arguments, URLs, screenshots or GitHub threads. The password-reset steps above avoid that credential-handling detour.


🆘 Getting Help

If you're still stuck:

  1. Keep the original evidence: note what you did, when it happened and the exact error. Do not repeat an update, outage or notification storm merely to reproduce it. A failed update banner does not prove the action left the target unchanged; check its current state before another attempt.
  2. Identify the affected version: distinguish the running server, reported agent and update target, not just the version before an upgrade. For Docker, include the running image tag or digest. If installation never started Pulse, give the attempted release and public installer/helper source, or say "unknown".
  3. Choose relevant, safe evidence: use existing logs, errors or observations first. If a result is already displayed in Settings → Support → Diagnostics & Health, GitHub (review first) downloads it without running checks again; follow safe diagnostics collection. For a visual problem, a screenshot or the exact error may be enough. If logs are needed, use the bounded Pulse log readers, choosing the command for the Pulse server's deployment, not the monitored target. A record limit alone does not bound a hung reader. If the read fails or times out, retain that result; do not use an unbounded substitute or treat a partial excerpt as complete. No request ID is required; do not repeat the failed action to obtain one. Never upload a full configuration or data directory.
  4. Review before posting: even a sanitized export or screenshot can contain identifying details. Remove credentials, session cookies, webhook URLs and private host, network or personal information. Never post bootstrap/recovery tokens, .env files, private keys or an unsanitized export.
  5. Use the appropriate thread: GitHub Issues for a bug, or Discussions for a setup question. Add new evidence to an existing matching report rather than opening a duplicate. Do not refile information you have already supplied.