Merge current notification reliability repairs into maintenance

Preserve the landed L6 source, response-verdict controls and ownership evidence
alongside the maintenance source repairs. Final protected checks and native
qualification must cover this combined source before landing.
This commit is contained in:
rcourtman 2026-10-18 17:21:17 +01:00
commit 45a82ea03c
14 changed files with 716 additions and 48 deletions

View file

@ -0,0 +1,136 @@
# Notification unclassified-failure investigation, 18 October 2026
## Signal and limits
The fixed aggregate view available during investigation was retrieved at
12:23:42 UTC, with status available and no missing columns. The clean weekly
cohort contains 8,537 rotating IDs. Terminal-failure coverage is 8,457 IDs, with
498 reporting failures. Rolling counters total 642,663 attempts, 248,005
deliveries and 115,318 terminal failures. The unclassified sum is 17,748 with
classification coverage of 8,117 IDs. Connectivity totals 38,299 and
authentication 22,462. Server-error coverage is narrower at 5,428 IDs.
The view is cached and exposes no transport, provider, endpoint or error text.
Rotating IDs are not customers, latest rolling counters are not distinct new
incidents, and retained counts may predate upgrades. Neither these source
reproductions nor changes between aggregate reads establish what proportion
of fleet failures they explain or that the fleet has recovered.
## Reproduced source defects
The unchanged baseline is public main
`98817ebce4e02545ca8870519ae929cee9c724c8`, including the preceding SMTP defaults,
connectivity and LOGIN repairs. Evidence is retained under the owned worker
directory `/opt/pulse-release-worker/task-proofs/notification-unclassified-20261018/`.
`baseline.log` records real loopback HTTP connections closed before headers
and SMTP connections closed before greeting or during AUTH. Plain EOF and
opaque failed network operations classify as unknown, while a close during
AUTH is misclassified as authentication through the sender's stage text.
This can stop a transient SMTP disconnect on the first attempt. Explicit
HTTP/SMTP verdicts and certificate failures must still outrank network evidence.
Go documents [network operation errors](https://pkg.go.dev/net#OpError) as
structured operation/network evidence, and [HTTP response bodies](https://pkg.go.dev/net/http#Response)
as streams that can fail after headers have arrived. The real receivers here
establish the concrete client failures; a bare error message is not evidence
of a particular destination or provider fault.
HTTP Apprise discarded the error from its bounded response-body drain.
All firing, recovery and Test controls accept a truncated HTTP 200 as success.
The exhausted-budget queue control consequently persists sent status, a
successful audit and a delivery receipt despite the interrupted response.
The unchanged complete-response control succeeds legitimately.
Invalid custom payload templates are another deterministic unknown path:
parse, execution and generated-JSON errors enter the ordinary queue's retry
ladder before any request can be sent. `baseline-templates.log` additionally
shows enhanced ntfy using a duplicate renderer with the same untyped verdict,
private template text in errors and rendered invalid payload text in logs.
All test credentials, messages and templates are synthetic.
The initial production Notifications browser pass also showed retained
configuration failures labelled "Failed, retries exhausted". Dead-letter
status does not establish exhaustion when a permanent verdict stops the first
attempt. The shared alerts presentation helper now says "Failed, retries
stopped" and describes retention using that same boundary. Existing outcome
assertions are updated at the helper and activity card, without changing
queue policy or retention.
`baseline-retained.log` records malformed retained email, webhook and Apprise
configuration, plus an unsupported retained notification type, each classified
unknown and rescheduled. These are invalid retained-job controls, not evidence
that Pulse normally writes malformed configuration or a fleet corruption claim.
## Canonical repair and proof boundaries
The shared failure classifier consumes propagated EOF/truncation and failed
network-operation evidence before prose, retaining declared classes, structured
SMTP replies and certificate/TLS precedence. Explicit unconfirmed SMTP
acceptance after DATA retains unknown classification and its existing transport
retry stop. Truly opaque failures stay unknown.
This supplies a diagnosis and preserves transient retry authority; it does not
repair the remote connection itself.
HTTP Apprise delegates its bounded response verdict to the existing webhook
reader, retains authoritative HTTP rejection classes even for interrupted
bodies, and preserves the read cause through safe diagnostics. A 2xx body-read
failure observed within the existing read cap cannot create a successful
receipt or clear recovery receipts. Reaching the cap retains the existing
bounded-reader semantics and does not prove whole-response completion.
Template and retained-configuration failures are declared configuration errors
at their owners. Enhanced ntfy delegates to the canonical renderer and retains
plain-text semantics. Fixed stage diagnostics preserve errors.Is/As without
copying template, payload or configuration contents into errors or logs. Valid
templates, receiver authentication verdicts, original queue snapshots, explicit
operator retry and dismissal semantics remain governed by their existing owners.
The tests exercise real HTTP/SMTP receivers and the actual persistent queue,
reopening its audit, status, receipt and telemetry records. The Apprise
terminal-verdict control deliberately admits one retained job with a
one-attempt budget; it does not simulate a natural retry schedule or establish
provider acceptance. Template admission uses ordinary SendAlert and verifies
that the receiver gets no request. Retained-config controls directly enqueue
invalid retained jobs and exercise the real decoder and processor.
The positive SMTP retry control loses the first connection during AUTH and
forwards the next connection at the same configured endpoint to the real
credential-checking receiver. It requires two connections and exactly one
accepted message. Existing plain/TLS/STARTTLS acceptance controls retain the
unknown verdict and single transport attempt after a missing DATA reply.
A real TLS handshake-refusal control preserves the TLS diagnosis of native
TCP alerts. [Go's pinned TLS implementation](https://github.com/golang/go/blob/go1.26.9/src/crypto/tls/conn.go)
wraps its unexported alert cause in a network-operation error, so generic
network classification must retain that native cause's TLS meaning.
Current-source focused controls pass in 0.293s, the full notifications package
passes in 136.861s, and the targeted notification race controls pass in 9.520s.
Those include the real TLS-alert and SMTP retry controls and unchanged
plain/TLS/STARTTLS missing-acceptance tests. Backend source is unchanged after
that pass.
Browser verification uses the existing production Notifications tab and API
readers with explicitly synthetic response data. It verifies display of the
reproduced verdicts, retained unknowns and recovery controls, separately from
the backend wire and persistence proofs. It is not native provider acceptance.
The affected matrix is the production tab fixture at
`/browser-tests/notification-evidence.html`, at 1440x1000 and 390x1100:
retained configuration/connectivity/opaque failures, safe diagnostics, shared
terminal label and retention description, refresh, cancelled retry
confirmation, focus return and reload. The fixture supplies read responses
only and requires zero API mutations. Pixel review checks wrapping, clipping,
control visibility and readable failure details at both widths.
The final helper/card test run passes 51 tests across three files. The final
browser pass succeeds after the copy change, with zero page errors, zero API
mutations and no horizontal overflow. All four desktop/narrow page and activity
captures were inspected. The frontend receipt binds the final helper content;
the retained worker proof also hashes all three changed backend source files.
## Remaining scope
The existing notification-delivery-terminal-failure-diagnosis gap stays triaged.
Destination-level repeated failures, destination recovery evidence and precise
delivery-health guidance remain active. Opaque CLI/provider failures and
unrecognised verdicts retain unknown classification when available evidence
does not support a more precise class. Historical audit rows and aggregate
counters are not rewritten. This slice does not publish or deploy a release.

View file

@ -10036,7 +10036,7 @@
},
{
"id": "notification-delivery-terminal-failure-diagnosis",
"summary": "The 18 October 2026 production summary retrieved at 10:01 UTC has 498 of 8,455 covered rotating IDs reporting terminal notification failures, with rolling totals of 120,040 failures and 246,419 deliveries. Connectivity (38,088) and authentication (27,959) are the largest classified sums; classification coverage is 8,113 IDs. These rolling aggregates do not identify a transport/provider or prove a release regression or recovery. Typed classification, deterministic-failure short circuiting and explicit retained-queue recovery already exist. Prior source slices repair SMTP provider-default authentication and concurrent envelope mutation, pinned HTTP fallback starvation, and webhook DNS continuing after cancellation. This authentication slice independently reproduces and repairs literal LOGIN challenge matching, substring AUTH mechanism selection, and port-465 connection tests attempting STARTTLS inside implicit TLS. Shared exact-name negotiation, a bounded prompt-independent LOGIN exchange and a canonical TLS-mode decision retain real 535 authentication verdicts, 454 retries and persisted queue/audit/telemetry outcomes. Unsupported mechanism lists now have a typed configuration verdict. Remaining work is destination-level repeated-failure handling, evidence of destination recovery, and precise delivery-health guidance. These source proofs do not establish which fleet failures they explain or that the production fleet has recovered.",
"summary": "The 18 October 2026 fixed production summary retrieved at 12:23 UTC has 498 of 8,457 covered rotating IDs reporting terminal notification failures, with rolling totals of 115,318 failures and 248,005 deliveries. Unclassified failures total 17,748 with classification coverage of 8,117 IDs. Connectivity (38,299) and authentication (22,462) remain the largest classified sums. These cached rolling aggregates cannot identify providers, destinations, distinct new incidents or recovery. Previous source slices repair SMTP provider defaults, concurrent envelopes, pinned HTTP fallback, bounded DNS, LOGIN exchange order, AUTH selection and port-465 connection-test parity. This slice independently reproduces and repairs unknown or misclassified pre-acceptance connection closes, deterministic template and invalid retained-job failures spending retry budgets, and HTTP Apprise suppressing truncated response errors and inventing sent receipts. Shared structural classification, canonical template rendering and bounded HTTP response verdicts retain authoritative receiver replies, safe wrapped diagnostics, and explicit unconfirmed SMTP acceptance as unknown. Real wire and reopened queue/audit/receipt/telemetry controls cover the source boundaries. The shared delivery activity label describes retries stopped without inferring budget exhaustion from terminal status. Remaining work is destination-level repeated-failure handling, destination recovery evidence and precise delivery-health guidance. Truly opaque failures remain unknown, and source proofs do not establish which fleet counters these defects explain or that the production fleet has recovered.",
"owner": "project-owner",
"status": "triaged",
"recorded_at": "2026-09-03",
@ -10064,6 +10064,11 @@
"path": "docs/release-control/v6/internal/records/notification-smtp-delivery-reliability-2026-10-17.md",
"kind": "file"
},
{
"repo": "pulse",
"path": "docs/release-control/v6/internal/records/notification-unclassified-reliability-2026-10-18.md",
"kind": "file"
},
{
"repo": "pulse",
"path": "frontend-modern/src/utils/alertDestinationsPresentation.ts",

View file

@ -4142,7 +4142,11 @@ newest-first record of real alert delivery attempts with plain-language
outcome labels, destination names resolved from the loaded webhook configs,
failure classes, and secret-redacted error text. The card names its retention
windows separately: completed attempts remain for seven days while
dead-letter attempts can remain for 30 days. It requests the bounded maximum
dead-letter attempts can remain for 30 days. Retained terminal failures say
that retries stopped, because a permanent verdict can stop the first attempt
before its retry budget is exhausted. The description uses the same boundary
and never infers attempt exhaustion from dead-letter status alone.
It requests the bounded maximum
of 200 rows, displays an absolute semantic timestamp for timeline correlation
with relative time as secondary hover context, and says that test sends skip
the queue. An unreadable log renders as unavailable rather than empty. While

View file

@ -1136,13 +1136,20 @@ The failure class is authoritative where the sender knows it and heuristic only
where nobody does. `internal/notifications/failure_class.go` owns that order:
a class declared by the sender through `NotificationFailureError` wins, then
Go's own error types decide (`*textproto.Error` carries the SMTP reply code,
x509 and TLS errors mean `tls`, `net.DNSError` and timeouts mean
`connectivity`), and only then is the error message text consulted.
x509 and TLS errors mean `tls`, while DNS errors, timeouts, failed network
operations and propagated EOF/truncation mean `connectivity`), and only then
is the error message text consulted. A connection closing during AUTH or TLS
does not prove a credential or certificate verdict. The structured SMTP/TLS
verdicts still outrank generic network evidence, and opaque failures remain
`unknown` when no declared or structured evidence identifies them.
Explicit `EmailAcceptanceUnconfirmedError` also remains `unknown`: loss of
the final DATA acknowledgement must not be presented as a definitive delivery
or rejection, nor as an ordinary pre-acceptance connection failure.
Two rules follow and are not optional. A destination's response body must never
determine its own failure class: every site that builds an error from an HTTP
status classifies from the status code, so the body is retained for the
operator's audit row but cannot make a 500 that says "rate limit" record as
status classifies from the status code, so arbitrary body text is withheld and
cannot make a 500 that says "rate limit" record as
`rate_limited`. And an SMTP reply code is not read like an HTTP status: a 5xx
reply is the destination refusing the message (`rejected`, or `authentication`
for 530/534/535/538 and `configuration` for the 500-504 syntax replies), while
@ -1168,6 +1175,18 @@ them proves a later attempt fails. TLS is deliberately on the retrying side: a
handshake can fail transiently during rotation, and the cost of one wasted
ladder is lower than the cost of dropping a recoverable notification.
Payload template parse, execution and JSON-validation failures are declared
configuration failures at the renderer. Ordinary and enhanced delivery,
including plain-text ntfy, share that renderer; ntfy retains plain-text payloads
without JSON validation. Malformed retained email, webhook and Apprise config,
and unsupported retained notification types, are also configuration failures.
These local failures stop on the first attempt rather than repeatedly trying a
job that cannot be rendered or decoded. Diagnostics retain a fixed stage
message and the wrapped cause for errors.Is/As, without copying configured
template text, rendered payloads or private configuration into logs and audits.
Retained configuration snapshots and explicit operator retry authority remain
unchanged. This does not rewrite historical audit classes or private jobs.
`NotificationFailureClass.Retryable` is the single owner of that split. Neither
the queue nor any destination type may keep its own list. This generalises to
every destination the decision webhook delivery already made for HTTP 4xx in
@ -1372,15 +1391,20 @@ HTTP or browser save path. It does not start a new operating-system process,
drive an alert lifecycle or establish recipient delivery. Those installed
acceptance obligations remain separate from constructor restoration coverage.
### Interrupted webhook rejection bodies retain the HTTP verdict
### Interrupted HTTP response bodies retain the delivery verdict
When webhook response headers establish a non-2xx status, a subsequent body
Webhooks and HTTP Apprise share the bounded `readWebhookResponse` verdict.
When response headers establish a non-2xx status, a subsequent body
read failure retains that HTTP status in the error and its authoritative
notification failure class. Terminal rejections must not become connectivity
retries merely because the diagnostic body is truncated. Response headers,
including Retry-After, and the underlying read error remain available; partial
body text is not added to the error. Existing handling of 2xx body read errors
is unchanged: it remains a read failure, not proof of recipient receipt.
body text is not added to the error. A 2xx response whose bounded body read
fails remains a failure, not proof of recipient receipt. HTTP Apprise firing,
recovery and Test must not suppress that error or mark the queued attempt sent.
Only a successful bounded response verdict can create or consume a delivery
receipt. Reaching the cap retains the reader's existing limit semantics. The
response-byte cap, payloads, TLS/SSRF policy and retry budgets remain.
Transport-only `TestWebhookTruncatedResponseClassification` covers 200, 401,
403, 422, 429 and 503 with interrupted bodies. `TestWebhookRetryTruncatedResponse`

View file

@ -0,0 +1,38 @@
{
"version": 1,
"result": "passed",
"base_sha": "98817ebce4e02545ca8870519ae929cee9c724c8",
"changed_paths": [
"frontend-modern/src/utils/alertDestinationsPresentation.ts"
],
"content_sha256": {
"frontend-modern/src/utils/alertDestinationsPresentation.ts": "f7d9ba6b5fe5a698a14e966a68539de247b38d14781472e7a1d379a3cf765cd5"
},
"routes": [
"/browser-tests/notification-evidence.html"
],
"viewports": [
{
"width": 1440,
"height": 1000
},
{
"width": 390,
"height": 1100
}
],
"states": [
"Production Notifications tab with synthetic read responses for retained configuration, connectivity and opaque failures",
"Retries stopped label and matching retention description, safe diagnostics and preserved unclassified verdict",
"Desktop and narrow pixels inspected for readable wrapping, unclipped controls and all three activity rows",
"Retained retry confirmation cancelled, focus restored and reloaded verdicts preserved"
],
"interactions": [
"Refresh delivery status and observe all retained verdicts",
"Open retained retry confirmation and dismiss it without an API mutation",
"Verify focus returns to the retry button",
"Reload and verify all three retries-stopped labels"
],
"verified_at": "2026-10-18T14:02:03.002Z",
"evidence_scope": "Production components and API readers with synthetic local read responses. Separate real HTTP/SMTP and persistent-queue tests establish backend source verdicts. No native provider or installed-release acceptance is claimed."
}

View file

@ -67,7 +67,7 @@ describe('AlertDeliveryLogCard', () => {
expect(screen.getByText('Ops Discord')).toBeInTheDocument();
expect(screen.getByText('Email')).toBeInTheDocument();
expect(screen.getByText('Failed, retries exhausted')).toBeInTheDocument();
expect(screen.getByText('Failed, retries stopped')).toBeInTheDocument();
expect(screen.getByText('Delivered')).toBeInTheDocument();
expect(screen.getByText('vm-offline-101 +1 more')).toBeInTheDocument();
expect(screen.getByText(/Authentication failure/)).toBeInTheDocument();
@ -91,7 +91,7 @@ describe('AlertDeliveryLogCard', () => {
it.each([
['retry', 'Retrying'],
['failed', 'Failed'],
['dead_letter', 'Failed, retries exhausted'],
['dead_letter', 'Failed, retries stopped'],
['cancelled', 'Cancelled'],
] as const)(
'replaces a previous successful row with %s evidence on refresh',
@ -156,7 +156,7 @@ describe('AlertDeliveryLogCard', () => {
expect(screen.getByText(/Completed attempts are retained for 7 days/)).toBeInTheDocument();
expect(
screen.getByText(/Failures that exhausted retries remain available for 30 days/),
screen.getByText(/Deliveries Pulse has stopped retrying remain available for 30 days/),
).toBeInTheDocument();
const timestamps = Array.from(container.querySelectorAll('time'));
expect(timestamps[0]).toHaveAttribute('datetime', log.entries[0].timestamp);

View file

@ -237,7 +237,7 @@ describe('alert destinations delivery log copy', () => {
expect(getAlertDeliveryLogOutcomeLabel('retry')).toBe('Retrying');
expect(getAlertDeliveryLogOutcomeLabel('failed')).toBe('Failed');
// "Dead letter" is queue vocabulary; a user needs to know retries stopped.
expect(getAlertDeliveryLogOutcomeLabel('dead_letter')).toBe('Failed, retries exhausted');
expect(getAlertDeliveryLogOutcomeLabel('dead_letter')).toBe('Failed, retries stopped');
expect(getAlertDeliveryLogOutcomeLabel('cancelled')).toBe('Cancelled');
});

View file

@ -307,7 +307,7 @@ export function getAlertDestinationsDeliveryLogDescription(
completedRetentionDays: number,
deadLetterRetentionDays: number,
) {
return `Delivery attempts and held notifications for real alerts. Completed attempts are retained for ${completedRetentionDays} days. Failures that exhausted retries remain available for ${deadLetterRetentionDays} days. Test sends skip the queue and are not listed here.`;
return `Delivery attempts and held notifications for real alerts. Completed attempts are retained for ${completedRetentionDays} days. Deliveries Pulse has stopped retrying remain available for ${deadLetterRetentionDays} days. Test sends skip the queue and are not listed here.`;
}
export function getAlertDestinationsDeliveryLogEmpty() {
@ -329,7 +329,7 @@ export function getAlertDeliveryLogOutcomeLabel(outcome: AlertDeliveryLogOutcome
case 'retry':
return 'Retrying';
case 'dead_letter':
return 'Failed, retries exhausted';
return 'Failed, retries stopped';
case 'cancelled':
return 'Cancelled';
default:

View file

@ -0,0 +1,103 @@
package notifications
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/stretchr/testify/require"
)
func TestAppriseInterruptedResponseRetainsVerdict(t *testing.T) {
for _, kind := range []string{"firing", "resolved", "test"} {
for _, code := range []int{200, 401, 429, 503} {
t.Run(kind+"/"+strconv.Itoa(code), func(t *testing.T) {
captured := captureAppriseLogs(t)
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "1000")
w.WriteHeader(code)
_, _ = fmt.Fprint(w, "provider-private-content ", appriseSecret)
}))
defer server.Close()
manager := &NotificationManager{}
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
err := sendAppriseForConfidentiality(manager, AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL}, kind)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
want := ClassFromHTTPStatus(code)
if code == 200 {
want = NotificationFailureConnectivity
}
require.Equal(t, want, ClassifyNotificationFailureError(err))
assertAppriseConfidential(t, err.Error())
assertAppriseConfidential(t, captured.String())
})
}
}
}
func TestQueuedAppriseInterruptedResponseDoesNotInventReceipt(t *testing.T) {
for _, complete := range []bool{false, true} {
t.Run(strconv.FormatBool(complete), func(t *testing.T) {
var requests atomic.Int32
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
if !complete {
w.Header().Set("Content-Length", "1000")
}
_, _ = fmt.Fprint(w, "provider-private-content")
}))
defer server.Close()
dir := t.TempDir()
manager := NewNotificationManagerWithDeferredQueue("", dir)
t.Cleanup(manager.Stop)
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
cfg := NormalizeAppriseConfig(AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL})
manager.SetAppriseConfig(cfg)
config, err := json.Marshal(cfg)
require.NoError(t, err)
alert := &alerts.Alert{ID: "response-verdict", ResourceName: "synthetic-node", Message: "synthetic alert", Level: alerts.AlertLevelWarning, StartTime: time.Now()}
// Admit one retained job with a one-attempt budget to observe an
// exhausted verdict through the actual processor and durable queue.
require.NoError(t, manager.queue.Enqueue(&QueuedNotification{ID: "response-verdict", Type: "apprise", Config: config, MaxAttempts: 1, Alerts: []*alerts.Alert{alert}}))
manager.StartQueueProcessing()
waitVerdictAudit(t, manager.queue, "response-verdict")
manager.Stop()
reopened := NewNotificationManagerWithDeferredQueue("", dir)
t.Cleanup(reopened.Stop)
var status, class string
var attempts, success int
require.NoError(t, reopened.queue.db.QueryRow(`SELECT status, attempts FROM notification_queue WHERE id = 'response-verdict'`).Scan(&status, &attempts))
require.NoError(t, reopened.queue.db.QueryRow(`SELECT success, failure_class FROM notification_audit WHERE notification_id = 'response-verdict'`).Scan(&success, &class))
var receipts int
require.NoError(t, reopened.queue.db.QueryRow(`SELECT count(*) FROM notification_delivery_receipts`).Scan(&receipts))
stats, err := reopened.GetTelemetryStats(time.Now().Add(-time.Hour))
require.NoError(t, err)
require.Equal(t, int32(1), requests.Load())
require.Equal(t, 1, attempts)
require.Equal(t, 1, stats.Attempts)
if complete {
require.Equal(t, string(QueueStatusSent), status)
require.Equal(t, 1, success)
require.Empty(t, class)
require.Equal(t, 1, receipts)
require.Equal(t, 1, stats.Deliveries)
require.Zero(t, stats.Failures)
} else {
require.Equal(t, string(QueueStatusDLQ), status)
require.Zero(t, success)
require.Equal(t, string(NotificationFailureConnectivity), class)
require.Zero(t, receipts)
require.Zero(t, stats.Deliveries)
require.Equal(t, 1, stats.Failures)
require.Equal(t, 1, stats.FailureClasses.Connectivity)
}
require.Zero(t, stats.FailureClasses.Unknown)
})
}
}

View file

@ -6,8 +6,10 @@ import (
"crypto/x509"
"errors"
"fmt"
"io"
"net"
"net/textproto"
"strings"
"syscall"
)
@ -62,6 +64,22 @@ func FailfWithClass(class NotificationFailureClass, format string, args ...any)
return &NotificationFailureError{Class: class, Err: fmt.Errorf(format, args...)}
}
// Local decoding/rendering errors can include private configuration or alert
// content. Retain their causes without putting that prose in diagnostics.
type notificationConfigurationError struct {
message string
cause error
}
func (e *notificationConfigurationError) Error() string {
return e.message + " (configuration details withheld)"
}
func (e *notificationConfigurationError) Unwrap() error { return e.cause }
func failNotificationConfiguration(message string, cause error) error {
return FailWithClass(NotificationFailureConfiguration, &notificationConfigurationError{message: message, cause: cause})
}
// ClassFromHTTPStatus maps an HTTP response status to its delivery class.
// Status codes are the destination's own structured verdict, so they outrank
// anything the response body happens to say.
@ -131,6 +149,14 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
return declared.Class
}
// Losing the final DATA acknowledgement leaves SMTP acceptance unknown.
// That explicit uncertainty must not become an ordinary connection verdict
// merely because the retained cause is EOF or another network error.
var unconfirmed *EmailAcceptanceUnconfirmedError
if errors.As(err, &unconfirmed) {
return NotificationFailureUnknown
}
// An SMTP reply code is the mail server's structured verdict. net/smtp
// surfaces it as *textproto.Error however deeply it is wrapped.
var protoErr *textproto.Error
@ -159,6 +185,8 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
}
if errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, io.EOF) ||
errors.Is(err, io.ErrUnexpectedEOF) ||
errors.Is(err, syscall.ECONNREFUSED) ||
errors.Is(err, syscall.ECONNRESET) ||
errors.Is(err, syscall.EHOSTUNREACH) ||
@ -168,6 +196,22 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
return NotificationFailureConnectivity
}
// A failed network operation remains connectivity evidence even when its
// cause has no recognised prose or the sender's stage mentions AUTH/TLS.
// Structured SMTP replies and certificate failures have already won above.
var operationErr *net.OpError
if errors.As(err, &operationErr) {
// Go's TCP TLS alerts have an unexported cause inside net.OpError.
// Retain their native TLS diagnosis, inspecting the cause rather than
// a sender's AUTH/TLS stage text around an ordinary connection error.
for cause := operationErr.Err; cause != nil; cause = errors.Unwrap(cause) {
if strings.HasPrefix(cause.Error(), "tls:") {
return NotificationFailureTLS
}
}
return NotificationFailureConnectivity
}
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
return NotificationFailureConnectivity

View file

@ -10,7 +10,6 @@ import (
"encoding/json"
"fmt"
"html"
"io"
"math"
"net"
"net/http"
@ -2513,18 +2512,22 @@ func (n *NotificationManager) sendAppriseViaHTTP(cfg AppriseConfig, title, body,
}
defer resp.Body.Close()
limited := io.LimitReader(resp.Body, WebhookMaxResponseSize)
responseBytes, _ := io.Copy(io.Discard, limited)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return FailfWithClass(ClassFromHTTPStatus(resp.StatusCode), "apprise server returned HTTP %d", resp.StatusCode)
// Share the bounded response verdict with webhooks. Receiving 2xx headers
// does not make a failed body read successful delivery or a durable receipt.
result, err := readWebhookResponse(resp)
if err != nil {
operation := "read Apprise server response"
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
operation = fmt.Sprintf("apprise server returned HTTP %d", resp.StatusCode)
}
return safeAppriseError(operation, err)
}
if responseBytes > 0 {
if result.responseBytes > 0 {
log.Debug().
Str("mode", string(cfg.Mode)).
Int("statusCode", resp.StatusCode).
Int64("responseBytes", responseBytes).
Int64("responseBytes", result.responseBytes).
Msg("apprise API response")
}
@ -3557,12 +3560,12 @@ func templateFuncMap() template.FuncMap {
func (n *NotificationManager) generatePayloadFromTemplateWithService(templateStr string, data WebhookPayloadData, service string) ([]byte, error) {
tmpl, err := template.New("webhook").Funcs(templateFuncMap()).Parse(templateStr)
if err != nil {
return nil, fmt.Errorf("invalid template: %w", err)
return nil, failNotificationConfiguration("invalid template", err)
}
var buf bytes.Buffer
if err := tmpl.Execute(&buf, data); err != nil {
return nil, fmt.Errorf("template execution failed: %w", err)
return nil, failNotificationConfiguration("template execution failed", err)
}
// Skip JSON validation for services that use plain text payloads
@ -3575,10 +3578,9 @@ func (n *NotificationManager) generatePayloadFromTemplateWithService(templateStr
var jsonCheck interface{}
if err := json.Unmarshal(buf.Bytes(), &jsonCheck); err != nil {
log.Error().
Err(err).
Str("payload", buf.String()).
Int("payloadBytes", buf.Len()).
Msg("generated webhook payload is invalid JSON")
return nil, fmt.Errorf("template produced invalid JSON: %w", err)
return nil, failNotificationConfiguration("template produced invalid JSON", err)
}
return buf.Bytes(), nil
@ -4243,7 +4245,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
case "email":
var emailConfig EmailConfig
if err = json.Unmarshal(notif.Config, &emailConfig); err != nil {
return fmt.Errorf("failed to unmarshal email config: %w", err)
return failNotificationConfiguration("failed to unmarshal email config", err)
}
currentEmailConfig := n.GetEmailConfig()
if !n.IsEnabled() || !currentEmailConfig.Enabled {
@ -4266,7 +4268,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
case "webhook":
var webhookConfig WebhookConfig
if err = json.Unmarshal(notif.Config, &webhookConfig); err != nil {
return fmt.Errorf("failed to unmarshal webhook config: %w", err)
return failNotificationConfiguration("failed to unmarshal webhook config", err)
}
if !n.IsEnabled() || !n.isQueuedWebhookStillEnabled(webhookConfig) {
log.Info().
@ -4289,7 +4291,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
case "apprise":
var appriseConfig AppriseConfig
if err = json.Unmarshal(notif.Config, &appriseConfig); err != nil {
return fmt.Errorf("failed to unmarshal apprise config: %w", err)
return failNotificationConfiguration("failed to unmarshal apprise config", err)
}
currentAppriseConfig := n.GetAppriseConfig()
if !n.IsEnabled() || !currentAppriseConfig.Enabled {
@ -4310,7 +4312,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
err = n.deliverNotificationJob(deliveredJob)
default:
return fmt.Errorf("unknown notification type: %s", baseType)
return FailfWithClass(NotificationFailureConfiguration, "unknown notification type")
}
if err == nil {

View file

@ -0,0 +1,326 @@
package notifications
import (
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/textproto"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"text/template"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestUnclassifiedTLSAlertRetainsTLSVerdict(t *testing.T) {
listener, err := tls.Listen("tcp4", "127.0.0.1:0", &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
return nil, errors.New("synthetic handshake refusal")
}})
require.NoError(t, err)
done := make(chan struct{})
go func() {
defer close(done)
conn, err := listener.Accept()
if err != nil {
return
}
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(3 * time.Second))
_ = conn.(*tls.Conn).Handshake()
}()
t.Cleanup(func() { _ = listener.Close(); <-done })
client, err := tls.DialWithDialer(&net.Dialer{Timeout: time.Second}, "tcp4", listener.Addr().String(), &tls.Config{InsecureSkipVerify: true})
if client != nil {
defer client.Close()
}
require.Error(t, err)
var operation *net.OpError
require.ErrorAs(t, err, &operation)
require.Equal(t, NotificationFailureTLS, ClassifyNotificationFailureError(err))
}
func TestUnclassifiedWireErrorsUseStructuralEvidence(t *testing.T) {
for _, tc := range []struct {
name string
err error
want NotificationFailureClass
}{
{"plain EOF", fmt.Errorf("send failed: %w", io.EOF), NotificationFailureConnectivity},
{"authentication stage EOF", fmt.Errorf("SMTP auth failed: %w", io.EOF), NotificationFailureConnectivity},
{"TLS stage EOF", fmt.Errorf("TLS dial failed: %w", io.EOF), NotificationFailureConnectivity},
{"truncation with misleading context", fmt.Errorf("authentication failed: %w", io.ErrUnexpectedEOF), NotificationFailureConnectivity},
{"opaque network operation", &net.OpError{Op: "read", Net: "tcp", Err: errors.New("opaque wire interruption")}, NotificationFailureConnectivity},
{"declared rejection wins", FailWithClass(NotificationFailureRejected, io.EOF), NotificationFailureRejected},
{"unconfirmed SMTP acceptance remains unknown", &EmailAcceptanceUnconfirmedError{Err: io.EOF}, NotificationFailureUnknown},
{"SMTP reply wins", &textproto.Error{Code: 535, Msg: "receiver disconnected"}, NotificationFailureAuthentication},
{"certificate wins", &net.OpError{Op: "dial", Net: "tcp", Err: x509.UnknownAuthorityError{}}, NotificationFailureTLS},
{"opaque error remains unknown", errors.New("opaque delivery failure"), NotificationFailureUnknown},
} {
t.Run(tc.name, func(t *testing.T) {
require.Equal(t, tc.want, ClassifyNotificationFailureError(tc.err))
})
}
}
func TestUnclassifiedClosedHTTPConnection(t *testing.T) {
for _, kind := range []string{"webhook", "apprise"} {
t.Run(kind, func(t *testing.T) {
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Error(err)
return
}
_ = conn.Close()
}))
defer server.Close()
manager := &NotificationManager{webhookClient: server.Client(), webhookRateLimits: make(map[string]*webhookRateLimit)}
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
var err error
if kind == "webhook" {
err = manager.sendWebhookRequest(WebhookConfig{URL: server.URL}, []byte(`{"synthetic":true}`), "test", "synthetic-event")
} else {
err = manager.SendTestAppriseWithConfig(AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL})
}
require.ErrorIs(t, err, io.EOF)
require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))
})
}
}
func TestUnclassifiedClosedSMTPConnection(t *testing.T) {
for _, stage := range []string{"greeting", "authentication"} {
t.Run(stage, func(t *testing.T) {
listener, err := net.Listen("tcp4", "127.0.0.1:0")
require.NoError(t, err)
var sessions sync.WaitGroup
done := make(chan struct{})
go func() {
defer close(done)
for {
conn, err := listener.Accept()
if err != nil {
return
}
sessions.Add(1)
go func() {
defer sessions.Done()
defer conn.Close()
if stage == "greeting" {
return
}
_ = conn.SetDeadline(time.Now().Add(3 * time.Second))
wire := textproto.NewConn(conn)
if wire.PrintfLine("220 localhost ESMTP") != nil {
return
}
if _, err := wire.ReadLine(); err != nil {
return
}
if wire.PrintfLine("250-localhost\r\n250 AUTH LOGIN") != nil {
return
}
_, _ = wire.ReadLine() // Close before answering AUTH, with no credential verdict.
}()
}
}()
t.Cleanup(func() { _ = listener.Close(); <-done; sessions.Wait() })
_, port, err := net.SplitHostPort(listener.Addr().String())
require.NoError(t, err)
p, err := strconv.Atoi(port)
require.NoError(t, err)
manager := NewEnhancedEmailManager(EmailProviderConfig{EmailConfig: EmailConfig{
SMTPHost: "127.0.0.1", SMTPPort: p, From: "pulse@example.test", To: []string{"ops@example.test"},
Username: "synthetic-user", Password: "synthetic-password",
}})
err = manager.SendEmailWithRetry("synthetic", "<p>synthetic</p>", "synthetic")
require.ErrorIs(t, err, io.EOF)
require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))
})
}
}
func TestUnclassifiedSMTPAuthDisconnectRecoversWithTransportRetry(t *testing.T) {
config, receiver := smtpAuthenticationReceiver(t, "plain", "LOGIN", []string{"User Name", "Password"}, 235)
upstreamAddress := net.JoinHostPort(config.SMTPHost, strconv.Itoa(config.SMTPPort))
listener, err := net.Listen("tcp4", "127.0.0.1:0")
require.NoError(t, err)
var connections atomic.Int32
var sessions sync.WaitGroup
done := make(chan struct{})
go func() {
defer close(done)
for {
conn, err := listener.Accept()
if err != nil {
return
}
attempt := connections.Add(1)
sessions.Add(1)
go func() {
defer sessions.Done()
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
if attempt == 1 {
wire := textproto.NewConn(conn)
if wire.PrintfLine("220 localhost ESMTP") != nil {
return
}
if _, err := wire.ReadLine(); err != nil {
return
}
if wire.PrintfLine("250-localhost\r\n250 AUTH LOGIN") != nil {
return
}
_, _ = wire.ReadLine() // First AUTH exchange loses its connection.
return
}
// Later connections reach the real credential-checking receiver
// behind the same configured endpoint, without replacing SMTP.
upstream, err := net.DialTimeout("tcp4", upstreamAddress, time.Second)
if err != nil {
t.Error(err)
return
}
defer upstream.Close()
_ = upstream.SetDeadline(time.Now().Add(5 * time.Second))
copied := make(chan struct{})
go func() { _, _ = io.Copy(upstream, conn); close(copied) }()
_, _ = io.Copy(conn, upstream)
_ = conn.Close()
_ = upstream.Close()
<-copied
}()
}
}()
t.Cleanup(func() { _ = listener.Close(); <-done; sessions.Wait() })
_, port, err := net.SplitHostPort(listener.Addr().String())
require.NoError(t, err)
config.SMTPPort, err = strconv.Atoi(port)
require.NoError(t, err)
config.MaxRetries = 1
require.NoError(t, NewEnhancedEmailManager(config).SendEmailWithRetry("synthetic", "<p>synthetic</p>", "synthetic"))
require.Equal(t, int32(2), connections.Load())
require.Equal(t, int32(1), receiver.accepted.Load())
require.Equal(t, int32(2), receiver.credentials.Load())
}
func TestUnclassifiedPayloadTemplateFailuresAreConfiguration(t *testing.T) {
for _, tc := range []struct{ name, source string }{
{"parse", `{"message":"{{synthetic_private_function}}"}`},
{"execution", `{"message":"{{.Absent}}"}`},
{"JSON", `{"message":"{{.Message}}"}`},
} {
t.Run(tc.name, func(t *testing.T) {
captured := captureAppriseLogs(t)
_, err := (&NotificationManager{}).generatePayloadFromTemplateWithService(tc.source,
WebhookPayloadData{Message: "synthetic-private-message\"invalid"}, "generic")
require.Error(t, err)
assert.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
require.NotContains(t, err.Error(), "synthetic_private_function")
require.NotContains(t, captured.String(), "synthetic-private-message")
})
}
}
func TestUnclassifiedEnhancedNtfyUsesCanonicalTemplateVerdict(t *testing.T) {
for _, action := range []string{"send", "test"} {
t.Run(action, func(t *testing.T) {
manager := &NotificationManager{}
config := EnhancedWebhookConfig{WebhookConfig: WebhookConfig{URL: "http://127.0.0.1:1"}, Service: "ntfy", PayloadTemplate: `{{.Absent}}`}
var err error
if action == "send" {
err = manager.SendEnhancedWebhook(config, &alerts.Alert{ID: "synthetic", StartTime: time.Now()})
} else {
_, _, err = manager.TestEnhancedWebhook(config)
}
require.Error(t, err)
require.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
var execution template.ExecError
require.ErrorAs(t, err, &execution)
require.NotContains(t, err.Error(), "Absent")
})
}
}
func TestUnclassifiedQueuedTemplateStopsWithoutSending(t *testing.T) {
var requests atomic.Int32
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { requests.Add(1); w.WriteHeader(204) }))
defer server.Close()
dir := t.TempDir()
manager := NewNotificationManagerWithDeferredQueue("", dir)
t.Cleanup(manager.Stop)
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
manager.SetGroupingConfig(false, 0, false, false)
manager.AddWebhook(WebhookConfig{ID: "synthetic-template", Enabled: true, URL: server.URL, Template: `{"message":"{{.Absent}}"}`})
manager.SendAlert(&alerts.Alert{ID: "template-failure", ResourceName: "synthetic-node", Level: alerts.AlertLevelWarning, StartTime: time.Now()})
pending, err := manager.queue.GetPending(10)
require.NoError(t, err)
require.Len(t, pending, 1)
manager.StartQueueProcessing()
waitVerdictAudit(t, manager.queue, pending[0].ID)
manager.Stop()
queue, err := NewNotificationQueue(dir)
require.NoError(t, err)
t.Cleanup(func() { _ = queue.Stop() })
var status, class, message string
var attempts int
require.NoError(t, queue.db.QueryRow(`SELECT status, attempts, failure_class, error_message FROM notification_audit WHERE notification_id = ?`, pending[0].ID).Scan(&status, &attempts, &class, &message))
require.Equal(t, string(QueueStatusDLQ), status)
require.Equal(t, 1, attempts)
require.Equal(t, string(NotificationFailureConfiguration), class)
require.True(t, strings.Contains(message, "template execution failed"))
stats, err := queue.GetTelemetryStats(time.Now().Add(-time.Hour))
require.NoError(t, err)
require.Equal(t, 1, stats.Failures)
require.Equal(t, 1, stats.FailureClasses.Configuration)
require.Zero(t, stats.FailureClasses.Unknown)
require.Zero(t, stats.Deliveries)
require.Zero(t, requests.Load())
}
func TestUnclassifiedRetainedConfigurationStopsWithoutRetry(t *testing.T) {
for _, kind := range []string{"email", "webhook", "apprise", "unsupported"} {
t.Run(kind, func(t *testing.T) {
dir := t.TempDir()
manager := NewNotificationManagerWithDeferredQueue("", dir)
t.Cleanup(manager.Stop)
config := []byte(`{"synthetic-private-field":`)
if kind == "unsupported" {
config = []byte(`{}`)
}
notif := &QueuedNotification{ID: "invalid-retained-config", Type: kind, Config: config, MaxAttempts: 3}
err := manager.ProcessQueuedNotification(notif)
require.Error(t, err)
assert.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
if kind != "unsupported" {
var syntax *json.SyntaxError
require.ErrorAs(t, err, &syntax)
}
require.NoError(t, manager.queue.Enqueue(notif))
manager.StartQueueProcessing()
waitVerdictAudit(t, manager.queue, notif.ID)
manager.Stop()
queue, err := NewNotificationQueue(dir)
require.NoError(t, err)
t.Cleanup(func() { _ = queue.Stop() })
var status, class string
var attempts int
require.NoError(t, queue.db.QueryRow(`SELECT status, attempts, failure_class FROM notification_audit WHERE notification_id = ?`, notif.ID).Scan(&status, &attempts, &class))
require.Equal(t, string(QueueStatusDLQ), status)
require.Equal(t, 1, attempts)
require.Equal(t, string(NotificationFailureConfiguration), class)
})
}
}

View file

@ -1,14 +1,12 @@
package notifications
import (
"bytes"
"errors"
"fmt"
"net/http"
"regexp"
"strconv"
"strings"
"text/template"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
@ -169,18 +167,6 @@ func (n *NotificationManager) prepareEnhancedWebhookExecution(webhook EnhancedWe
}
webhook.WebhookConfig = canonical
if webhook.Service == "ntfy" && strings.TrimSpace(webhook.PayloadTemplate) != "" {
tmpl, err := template.New("enhanced-webhook-ntfy").Funcs(templateFuncMap()).Parse(webhook.PayloadTemplate)
if err != nil {
return webhook, nil, fmt.Errorf("parse ntfy payload template: %w", err)
}
var rendered bytes.Buffer
if err := tmpl.Execute(&rendered, data); err != nil {
return webhook, nil, fmt.Errorf("render ntfy payload template: %w", err)
}
return webhook, rendered.Bytes(), nil
}
payload, err := n.renderWebhookPayloadJSON(canonical, data, webhookRenderModeSingle, func() ([]byte, error) {
if strings.TrimSpace(webhook.PayloadTemplate) == "" {
return nil, fmt.Errorf("enhanced webhook requires payload template or known service template")

View file

@ -407,7 +407,7 @@ test.describe('Alert operator qualification', () => {
).toBeVisible();
await expect(
page.getByText(
'Completed attempts are retained for 7 days. Failures that exhausted retries remain available for 30 days.',
'Completed attempts are retained for 7 days. Deliveries Pulse has stopped retrying remain available for 30 days.',
{ exact: false },
),
).toBeVisible();