diff --git a/docs/PBS.md b/docs/PBS.md index ce6841246f..078e62e793 100644 --- a/docs/PBS.md +++ b/docs/PBS.md @@ -251,6 +251,12 @@ another machine does not prove that Pulse can reach PBS. openssl x509 -in /etc/proxmox-backup/proxy.pem -noout -fingerprint -sha256 ``` + For renewal, a fingerprint mismatch or contradictory saved pins, use the + [certificate-trust checks](TROUBLESHOOTING.md#proxmox-or-pbs-certificate-validation-failed) + before changing this connection. A saved leaf pin is a different trust + method from CA validation; do not disable verification or clear it simply + because a new certificate connects. + - **Authentication/permissions:** test a datastore request, not just `/version`. Prepare a private header file on the machine running curl. Stop if preparation fails; an existing file is preserved, and symlinked or non-regular credential diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 35349b4131..f6350d4fb3 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -24,6 +24,10 @@ establish safe recovery. [agent authentication and reporting evidence](UNIFIED_AGENT.md#agent-lookup-returns-http-401-or-403) before replacing a token or reinstalling. Installer success is not an accepted report. +- **Proxmox or PBS reports a certificate error:** distinguish + [certificate trust, renewal and saved-pin conflicts](TROUBLESHOOTING.md#proxmox-or-pbs-certificate-validation-failed). + Keep verification enabled; a new certificate or a working browser is not + permission to replace saved trust. - **Copied agent install or removal command failed:** distinguish its [preparation, private prompt and lifecycle action](UNIFIED_AGENT.md#copied-agent-command-fails-before-installation). A stopped new command does not diagnose the existing agent. Keep state and @@ -211,6 +215,75 @@ do not replace credentials or disable TLS verification on a refusal alone. For a page that opens but has missing readings, use the [missing-data checks](FAQ.md#no-data-showing) instead of resetting the connection. +### Proxmox or PBS certificate validation failed + +A TLS error is not an invalid API token or a missing permission. Keep the +original error and time, and inspect the **existing** connection in +**Settings → Infrastructure → Manage → SSL settings**. Check its saved +**Endpoint URL**, **Verify SSL certificate** and **SSL Fingerprint (optional)** +without changing them just to diagnose the failure. PVE normally uses HTTPS +port **8006**; PBS uses **8007**. A browser reaching the platform is a different +client and trust store from the Pulse server or container. + +**Keep certificate verification enabled.** Do not switch to HTTP, disable +verification, clear a saved pin, rotate API tokens or delete and re-add the +connection merely to get past an error. + +- **Unknown authority or self-signed certificate:** the Pulse runtime needs + the intended trusted certificate chain, or an independently verified SHA-256 + leaf fingerprint. Obtain trust through an already-trusted platform console + or administration channel, not solely from the failing connection or its + suggested fingerprint. See the [PBS certificate check](PBS.md#connection-failed-error). +- **Expired, not-yet-valid or hostname error:** reconcile the platform's + certificate, its validity period, the clocks and the saved endpoint's + hostname. An IP address is not interchangeable with a certificate's DNS + name. Do not bypass the error or pin an unsuitable certificate to hide it. +- **Fingerprint mismatch:** the presented leaf differs from the saved pin. + Renewal is one possible reason, not proof that this is the same server. + Verify the replacement independently before making a trust change. +- **TLS trust conflict:** different saved fingerprints refer to equivalent + endpoint URLs. This is contradictory saved trust, not a reason to choose + whichever pin connects. Preserve the entries and verify which endpoint and + certificate each belongs to before reconciliation. + +#### Certificate renewal and saved fingerprints + +A saved fingerprint selects leaf-certificate pinning instead of normal CA +chain and hostname validation, even with **Verify SSL certificate** enabled. +Renewing an ACME certificate can therefore invalidate a saved pin while the +new certificate's chain is valid. Leaving verification enabled with no saved +fingerprint uses the Pulse runtime's CA trust; disabling it with no pin removes +chain and hostname checks. + +Choose the intended trust method, not an error-suppression workaround. Only +after independently verifying the replacement certificate and server identity, +update the affected connection's pin. If deliberately moving from leaf pinning +to CA validation, first establish the runtime's trusted chain and matching +endpoint hostname, then remove only that connection's pin while keeping +verification enabled. Preserve unrelated credentials, collection scope and +saved connections. If you cannot establish trust, stop rather than accept the +new certificate. + +#### Cluster members and conflicting saved trust + +Cluster members can have different certificates. A reachable primary or one +healthy member does not prove trust or polling recovery for every peer; do not +copy the primary's fingerprint to all members. In **Cluster members**, retain +the affected node and existing failure reason. If **Unreachable — details** +is available, open it with touch or keyboard rather than relying on a hover. +A display-name edit does not reconcile certificate trust. + +If the affected member pin or conflicting entry cannot be safely reconciled +through the existing controls, leave it intact and use [Getting Help](#-getting-help). +Share only the running Pulse version, PVE/PBS version, original time, redacted +error class, which primary or peers are affected, and whether a certificate +renewal was already observed. Use consistent aliases; keep literal endpoints, +fingerprints, credentials and full configuration private. Unknown is valid. +Do not edit encrypted files, run Diagnostics, force rediscovery, reinstall or +restart just to fill a report. During a freeze/thaw or unresponsive-guest +incident, follow [Backup safety](VM_DISK_MONITORING.md#backup-safety) first; +certificate help does not establish workload recovery. + ### CORS errors CORS controls whether a browser can read a response from a different origin. diff --git a/frontend-modern/browser-tests/certificate-trust-help.cjs b/frontend-modern/browser-tests/certificate-trust-help.cjs new file mode 100644 index 0000000000..4e5de80ed4 --- /dev/null +++ b/frontend-modern/browser-tests/certificate-trust-help.cjs @@ -0,0 +1,163 @@ +// Exercise certificate guidance through the shipped Docs renderer, router and CSS. +// No TLS handshake, API request, credential edit or platform operation is performed. +const fs = require('node:fs'); +const path = require('node:path'); +const http = require('node:http'); +const crypto = require('node:crypto'); +const assert = require('node:assert/strict'); +const { chromium, webkit } = require('playwright'); +const root = '/workspace/frontend-modern'; +const preview = '/workspace/tmp/doc-command-help/static'; +const output = '/workspace/tmp/certificate-trust-help'; +const digest = bytes => crypto.createHash('sha256').update(bytes).digest('hex'); +const bindingBytes = fs.readFileSync(path.join(preview, 'binding.json')); +const binding = JSON.parse(bindingBytes); +const result = { result: 'incomplete', source_sha: binding.source_sha, + binding_sha256: digest(bindingBytes), playwright: require('playwright/package.json').version, + cases: [], captures: [], errors: [], served: [], cleanup: {}, + limits: 'Changed help rendering/navigation only; not native TLS, installed trust, release inclusion or workload recovery.' }; + +(async () => { + let server, browser; + try { + fs.mkdirSync(output, { recursive: true }); + assert.equal(result.playwright, '1.56.1'); + assert.equal(result.playwright, JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')) + .packages['node_modules/@playwright/test'].version); + for (const [file, hash] of Object.entries(binding.sources)) + assert.equal(digest(fs.readFileSync(path.join(root, file))), hash, file); + const assets = new Map(); + for (const [file, hash] of Object.entries(binding.artifacts)) { + const bytes = fs.readFileSync(path.join(preview, file)); + assert.equal(digest(bytes), hash, file); assets.set(file, bytes); + } + for (const doc of ['TROUBLESHOOTING', 'PBS']) { + const hash = binding.artifacts[`docs/${doc}.md`]; + assert.equal(hash, binding.sources[`public/docs/${doc}.md`]); + assert.equal(hash, digest(fs.readFileSync(`/workspace/docs/${doc}.md`))); + } + result.source_inputs = Object.keys(binding.sources).length; + server = http.createServer((req, res) => { + if (req.method !== 'GET') { res.writeHead(405); res.end(); return; } + const pathname = new URL(req.url, 'http://fixture.invalid').pathname; + const file = pathname.startsWith('/docs/') && !pathname.endsWith('.md') + ? 'browser-tests/doc-command-help.html' : pathname.slice(1); + if (!assets.has(file)) { res.writeHead(404); res.end(); return; } + const bytes = assets.get(file); + result.served.push({ pathname, file, sha256: digest(bytes) }); + const type = ({ '.md': 'text/markdown', '.css': 'text/css', '.js': 'text/javascript', '.html': 'text/html' })[path.extname(file)]; + res.writeHead(200, { 'Content-Type': `${type}; charset=utf-8`, 'Cache-Control': 'no-store' }); res.end(bytes); + }); + await new Promise((resolve, reject) => { + server.once('error', reject); server.listen(0, '127.0.0.1', resolve); + }); + const origin = `http://127.0.0.1:${server.address().port}`; + for (const [name, engine, width, dark] of [ + ['desktop-chromium', chromium, 1360, false], ['phone-webkit', webkit, 390, true], + ]) { + browser = await engine.launch(engine === chromium + ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true }); + const context = await browser.newContext({ viewport: { width, height: 900 }, + isMobile: width < 500, hasTouch: width < 500, locale: 'en-GB' }); + const page = await context.newPage(); page.setDefaultTimeout(12000); + const record = { name, width, height: 900, dark, browser: browser.version(), checks: [] }; + result.cases.push(record); + page.on('pageerror', error => result.errors.push(error.message)); + page.on('console', message => { if (message.type() === 'error') result.errors.push(message.text()); }); + page.on('response', response => { if (!response.ok()) result.errors.push(`HTTP ${response.status()}: ${response.url()}`); }); + await page.route('**/*', route => { + const url = new URL(route.request().url()); + assert.equal(url.origin, origin, 'no external requests permitted'); + assert(!url.pathname.startsWith('/api/'), 'help must not probe a backend'); + assert.equal(route.request().method(), 'GET'); + return route.continue(); + }); + await page.addInitScript(dark => document.addEventListener('DOMContentLoaded', () => + document.documentElement.classList.toggle('dark', dark)), dark); + const capture = async suffix => { + await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))); + assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), 'page overflow'); + const file = `${name}-${suffix}.png`; await page.screenshot({ path: path.join(output, file) }); + result.captures.push({ file, sha256: digest(fs.readFileSync(path.join(output, file))), name }); + }; + const activate = async link => { + if (width < 500) await link.tap(); else { await link.focus(); await page.keyboard.press('Enter'); } + }; + const atFragment = async id => page.waitForFunction(id => { + const heading = document.getElementById(id); + return heading && document.activeElement === heading && heading.getBoundingClientRect().top >= 0 + && heading.getBoundingClientRect().top < innerHeight; + }, id); + await page.goto(`${origin}/docs/TROUBLESHOOTING`, { waitUntil: 'domcontentloaded' }); + const entry = page.getByRole('link', { name: 'certificate trust, renewal and saved-pin conflicts', exact: true }); + await entry.waitFor(); + await page.evaluate(() => window.__certificateHelpShell = 'same-shell'); + await activate(entry); + await page.waitForURL('**/docs/TROUBLESHOOTING#proxmox-or-pbs-certificate-validation-failed'); + await atFragment('proxmox-or-pbs-certificate-validation-failed'); + assert.equal(await page.evaluate(() => window.__certificateHelpShell), 'same-shell'); + const text = (await page.locator('article').innerText()).replace(/\s+/g, ' '); + for (const phrase of ['A TLS error is not an invalid API token or a missing permission', + 'Endpoint URL', 'Verify SSL certificate', 'SSL Fingerprint (optional)', + 'different client and trust store', 'Keep certificate verification enabled', + 'Unknown authority or self-signed certificate', 'Expired, not-yet-valid or hostname error', + 'Renewal is one possible reason, not proof that this is the same server', + 'contradictory saved trust, not a reason to choose whichever pin connects', + 'instead of normal CA chain and hostname validation', + 'first establish the runtime’s trusted chain and matching endpoint hostname', + 'remove only that connection’s pin while keeping verification enabled', + 'do not copy the primary’s fingerprint to all members', + 'A display-name edit does not reconcile certificate trust', + 'leave it intact', 'Unknown is valid', 'Do not edit encrypted files', + 'certificate help does not establish workload recovery']) + assert(text.includes(phrase.replaceAll('’', "'")), phrase); + record.checks.push('symptom entry, four TLS error distinctions, CA versus leaf trust, verified renewal, per-member safety and private evidence'); + await capture('errors'); + for (const [id, suffix] of [ + ['certificate-renewal-and-saved-fingerprints', 'renewal'], + ['cluster-members-and-conflicting-saved-trust', 'members'], + ]) { + await page.goto(`${origin}/docs/TROUBLESHOOTING#${id}`, { waitUntil: 'domcontentloaded' }); + await atFragment(id); await capture(suffix); + await page.reload({ waitUntil: 'domcontentloaded' }); await atFragment(id); + } + record.checks.push('direct and reloaded renewal/member fragments receive visible heading focus without page overflow'); + await page.evaluate(() => window.__certificateHelpShell = 'same-shell'); + await activate(page.getByRole('link', { name: 'Getting Help', exact: true }).last()); + await page.waitForURL('**/docs/TROUBLESHOOTING#-getting-help'); + await atFragment('-getting-help'); + assert.equal(await page.evaluate(() => window.__certificateHelpShell), 'same-shell'); + await page.goBack(); await atFragment('cluster-members-and-conflicting-saved-trust'); + const backup = page.locator('article a[href="/docs/VM_DISK_MONITORING#backup-safety"]').last(); + // The preview does not include VM_DISK_MONITORING. Check the exact link + // without following it or pretending to have proved platform recovery. + assert.equal(await backup.count(), 1); + await page.goto(`${origin}/docs/TROUBLESHOOTING#proxmox-or-pbs-certificate-validation-failed`, { waitUntil: 'domcontentloaded' }); + await atFragment('proxmox-or-pbs-certificate-validation-failed'); + await page.evaluate(() => window.__certificateHelpShell = 'same-shell'); + await activate(page.getByRole('link', { name: 'PBS certificate check', exact: true })); + await page.waitForURL('**/docs/PBS#connection-failed-error'); await atFragment('connection-failed-error'); + const returnLink = page.getByRole('link', { name: 'certificate-trust checks', exact: true }); + await returnLink.waitFor(); + const pbsText = (await returnLink.locator('..').innerText()).replace(/\s+/g, ' '); + assert(pbsText.includes('A saved leaf pin is a different trust method from CA validation')); + assert(pbsText.includes('do not disable verification or clear it')); + await returnLink.evaluate(link => link.closest('p').scrollIntoView({ block: 'start' })); + await capture('pbs-entry'); + await activate(returnLink); + await page.waitForURL('**/docs/TROUBLESHOOTING#proxmox-or-pbs-certificate-validation-failed'); + await atFragment('proxmox-or-pbs-certificate-validation-failed'); + assert.equal(await page.evaluate(() => window.__certificateHelpShell), 'same-shell'); + record.checks.push('keyboard/touch Getting Help and PBS round-trip links, focused destinations and browser back'); + await context.close(); await browser.close(); browser = null; + } + assert.equal(result.errors.length, 0); result.result = 'passed'; + } catch (error) { result.failure = error.stack; throw error; } + finally { + if (browser) await browser.close(); result.cleanup.browser_closed = true; + if (server) await new Promise((resolve, reject) => { + server.close(error => error ? reject(error) : resolve()); server.closeAllConnections(); + }); result.cleanup.server_closed = true; + fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2) + '\n'); + } +})().catch(error => { console.error(error); process.exitCode = 1; }); diff --git a/frontend-modern/public/docs/PBS.md b/frontend-modern/public/docs/PBS.md index ce6841246f..078e62e793 100644 --- a/frontend-modern/public/docs/PBS.md +++ b/frontend-modern/public/docs/PBS.md @@ -251,6 +251,12 @@ another machine does not prove that Pulse can reach PBS. openssl x509 -in /etc/proxmox-backup/proxy.pem -noout -fingerprint -sha256 ``` + For renewal, a fingerprint mismatch or contradictory saved pins, use the + [certificate-trust checks](TROUBLESHOOTING.md#proxmox-or-pbs-certificate-validation-failed) + before changing this connection. A saved leaf pin is a different trust + method from CA validation; do not disable verification or clear it simply + because a new certificate connects. + - **Authentication/permissions:** test a datastore request, not just `/version`. Prepare a private header file on the machine running curl. Stop if preparation fails; an existing file is preserved, and symlinked or non-regular credential diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md index 35349b4131..f6350d4fb3 100644 --- a/frontend-modern/public/docs/TROUBLESHOOTING.md +++ b/frontend-modern/public/docs/TROUBLESHOOTING.md @@ -24,6 +24,10 @@ establish safe recovery. [agent authentication and reporting evidence](UNIFIED_AGENT.md#agent-lookup-returns-http-401-or-403) before replacing a token or reinstalling. Installer success is not an accepted report. +- **Proxmox or PBS reports a certificate error:** distinguish + [certificate trust, renewal and saved-pin conflicts](TROUBLESHOOTING.md#proxmox-or-pbs-certificate-validation-failed). + Keep verification enabled; a new certificate or a working browser is not + permission to replace saved trust. - **Copied agent install or removal command failed:** distinguish its [preparation, private prompt and lifecycle action](UNIFIED_AGENT.md#copied-agent-command-fails-before-installation). A stopped new command does not diagnose the existing agent. Keep state and @@ -211,6 +215,75 @@ do not replace credentials or disable TLS verification on a refusal alone. For a page that opens but has missing readings, use the [missing-data checks](FAQ.md#no-data-showing) instead of resetting the connection. +### Proxmox or PBS certificate validation failed + +A TLS error is not an invalid API token or a missing permission. Keep the +original error and time, and inspect the **existing** connection in +**Settings → Infrastructure → Manage → SSL settings**. Check its saved +**Endpoint URL**, **Verify SSL certificate** and **SSL Fingerprint (optional)** +without changing them just to diagnose the failure. PVE normally uses HTTPS +port **8006**; PBS uses **8007**. A browser reaching the platform is a different +client and trust store from the Pulse server or container. + +**Keep certificate verification enabled.** Do not switch to HTTP, disable +verification, clear a saved pin, rotate API tokens or delete and re-add the +connection merely to get past an error. + +- **Unknown authority or self-signed certificate:** the Pulse runtime needs + the intended trusted certificate chain, or an independently verified SHA-256 + leaf fingerprint. Obtain trust through an already-trusted platform console + or administration channel, not solely from the failing connection or its + suggested fingerprint. See the [PBS certificate check](PBS.md#connection-failed-error). +- **Expired, not-yet-valid or hostname error:** reconcile the platform's + certificate, its validity period, the clocks and the saved endpoint's + hostname. An IP address is not interchangeable with a certificate's DNS + name. Do not bypass the error or pin an unsuitable certificate to hide it. +- **Fingerprint mismatch:** the presented leaf differs from the saved pin. + Renewal is one possible reason, not proof that this is the same server. + Verify the replacement independently before making a trust change. +- **TLS trust conflict:** different saved fingerprints refer to equivalent + endpoint URLs. This is contradictory saved trust, not a reason to choose + whichever pin connects. Preserve the entries and verify which endpoint and + certificate each belongs to before reconciliation. + +#### Certificate renewal and saved fingerprints + +A saved fingerprint selects leaf-certificate pinning instead of normal CA +chain and hostname validation, even with **Verify SSL certificate** enabled. +Renewing an ACME certificate can therefore invalidate a saved pin while the +new certificate's chain is valid. Leaving verification enabled with no saved +fingerprint uses the Pulse runtime's CA trust; disabling it with no pin removes +chain and hostname checks. + +Choose the intended trust method, not an error-suppression workaround. Only +after independently verifying the replacement certificate and server identity, +update the affected connection's pin. If deliberately moving from leaf pinning +to CA validation, first establish the runtime's trusted chain and matching +endpoint hostname, then remove only that connection's pin while keeping +verification enabled. Preserve unrelated credentials, collection scope and +saved connections. If you cannot establish trust, stop rather than accept the +new certificate. + +#### Cluster members and conflicting saved trust + +Cluster members can have different certificates. A reachable primary or one +healthy member does not prove trust or polling recovery for every peer; do not +copy the primary's fingerprint to all members. In **Cluster members**, retain +the affected node and existing failure reason. If **Unreachable — details** +is available, open it with touch or keyboard rather than relying on a hover. +A display-name edit does not reconcile certificate trust. + +If the affected member pin or conflicting entry cannot be safely reconciled +through the existing controls, leave it intact and use [Getting Help](#-getting-help). +Share only the running Pulse version, PVE/PBS version, original time, redacted +error class, which primary or peers are affected, and whether a certificate +renewal was already observed. Use consistent aliases; keep literal endpoints, +fingerprints, credentials and full configuration private. Unknown is valid. +Do not edit encrypted files, run Diagnostics, force rediscovery, reinstall or +restart just to fill a report. During a freeze/thaw or unresponsive-guest +incident, follow [Backup safety](VM_DISK_MONITORING.md#backup-safety) first; +certificate help does not establish workload recovery. + ### CORS errors CORS controls whether a browser can read a response from a different origin.