mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-21 14:20:06 +00:00
Merge current notification reliability repairs into maintenance
Preserve the landed L6 source, response-verdict controls and ownership evidence alongside the maintenance source repairs. Final protected checks and native qualification must cover this combined source before landing.
This commit is contained in:
commit
45a82ea03c
14 changed files with 716 additions and 48 deletions
|
|
@ -0,0 +1,136 @@
|
|||
# Notification unclassified-failure investigation, 18 October 2026
|
||||
|
||||
## Signal and limits
|
||||
|
||||
The fixed aggregate view available during investigation was retrieved at
|
||||
12:23:42 UTC, with status available and no missing columns. The clean weekly
|
||||
cohort contains 8,537 rotating IDs. Terminal-failure coverage is 8,457 IDs, with
|
||||
498 reporting failures. Rolling counters total 642,663 attempts, 248,005
|
||||
deliveries and 115,318 terminal failures. The unclassified sum is 17,748 with
|
||||
classification coverage of 8,117 IDs. Connectivity totals 38,299 and
|
||||
authentication 22,462. Server-error coverage is narrower at 5,428 IDs.
|
||||
|
||||
The view is cached and exposes no transport, provider, endpoint or error text.
|
||||
Rotating IDs are not customers, latest rolling counters are not distinct new
|
||||
incidents, and retained counts may predate upgrades. Neither these source
|
||||
reproductions nor changes between aggregate reads establish what proportion
|
||||
of fleet failures they explain or that the fleet has recovered.
|
||||
|
||||
## Reproduced source defects
|
||||
|
||||
The unchanged baseline is public main
|
||||
`98817ebce4e02545ca8870519ae929cee9c724c8`, including the preceding SMTP defaults,
|
||||
connectivity and LOGIN repairs. Evidence is retained under the owned worker
|
||||
directory `/opt/pulse-release-worker/task-proofs/notification-unclassified-20261018/`.
|
||||
|
||||
`baseline.log` records real loopback HTTP connections closed before headers
|
||||
and SMTP connections closed before greeting or during AUTH. Plain EOF and
|
||||
opaque failed network operations classify as unknown, while a close during
|
||||
AUTH is misclassified as authentication through the sender's stage text.
|
||||
This can stop a transient SMTP disconnect on the first attempt. Explicit
|
||||
HTTP/SMTP verdicts and certificate failures must still outrank network evidence.
|
||||
|
||||
Go documents [network operation errors](https://pkg.go.dev/net#OpError) as
|
||||
structured operation/network evidence, and [HTTP response bodies](https://pkg.go.dev/net/http#Response)
|
||||
as streams that can fail after headers have arrived. The real receivers here
|
||||
establish the concrete client failures; a bare error message is not evidence
|
||||
of a particular destination or provider fault.
|
||||
|
||||
HTTP Apprise discarded the error from its bounded response-body drain.
|
||||
All firing, recovery and Test controls accept a truncated HTTP 200 as success.
|
||||
The exhausted-budget queue control consequently persists sent status, a
|
||||
successful audit and a delivery receipt despite the interrupted response.
|
||||
The unchanged complete-response control succeeds legitimately.
|
||||
|
||||
Invalid custom payload templates are another deterministic unknown path:
|
||||
parse, execution and generated-JSON errors enter the ordinary queue's retry
|
||||
ladder before any request can be sent. `baseline-templates.log` additionally
|
||||
shows enhanced ntfy using a duplicate renderer with the same untyped verdict,
|
||||
private template text in errors and rendered invalid payload text in logs.
|
||||
All test credentials, messages and templates are synthetic.
|
||||
|
||||
The initial production Notifications browser pass also showed retained
|
||||
configuration failures labelled "Failed, retries exhausted". Dead-letter
|
||||
status does not establish exhaustion when a permanent verdict stops the first
|
||||
attempt. The shared alerts presentation helper now says "Failed, retries
|
||||
stopped" and describes retention using that same boundary. Existing outcome
|
||||
assertions are updated at the helper and activity card, without changing
|
||||
queue policy or retention.
|
||||
|
||||
`baseline-retained.log` records malformed retained email, webhook and Apprise
|
||||
configuration, plus an unsupported retained notification type, each classified
|
||||
unknown and rescheduled. These are invalid retained-job controls, not evidence
|
||||
that Pulse normally writes malformed configuration or a fleet corruption claim.
|
||||
|
||||
## Canonical repair and proof boundaries
|
||||
|
||||
The shared failure classifier consumes propagated EOF/truncation and failed
|
||||
network-operation evidence before prose, retaining declared classes, structured
|
||||
SMTP replies and certificate/TLS precedence. Explicit unconfirmed SMTP
|
||||
acceptance after DATA retains unknown classification and its existing transport
|
||||
retry stop. Truly opaque failures stay unknown.
|
||||
This supplies a diagnosis and preserves transient retry authority; it does not
|
||||
repair the remote connection itself.
|
||||
|
||||
HTTP Apprise delegates its bounded response verdict to the existing webhook
|
||||
reader, retains authoritative HTTP rejection classes even for interrupted
|
||||
bodies, and preserves the read cause through safe diagnostics. A 2xx body-read
|
||||
failure observed within the existing read cap cannot create a successful
|
||||
receipt or clear recovery receipts. Reaching the cap retains the existing
|
||||
bounded-reader semantics and does not prove whole-response completion.
|
||||
|
||||
Template and retained-configuration failures are declared configuration errors
|
||||
at their owners. Enhanced ntfy delegates to the canonical renderer and retains
|
||||
plain-text semantics. Fixed stage diagnostics preserve errors.Is/As without
|
||||
copying template, payload or configuration contents into errors or logs. Valid
|
||||
templates, receiver authentication verdicts, original queue snapshots, explicit
|
||||
operator retry and dismissal semantics remain governed by their existing owners.
|
||||
|
||||
The tests exercise real HTTP/SMTP receivers and the actual persistent queue,
|
||||
reopening its audit, status, receipt and telemetry records. The Apprise
|
||||
terminal-verdict control deliberately admits one retained job with a
|
||||
one-attempt budget; it does not simulate a natural retry schedule or establish
|
||||
provider acceptance. Template admission uses ordinary SendAlert and verifies
|
||||
that the receiver gets no request. Retained-config controls directly enqueue
|
||||
invalid retained jobs and exercise the real decoder and processor.
|
||||
|
||||
The positive SMTP retry control loses the first connection during AUTH and
|
||||
forwards the next connection at the same configured endpoint to the real
|
||||
credential-checking receiver. It requires two connections and exactly one
|
||||
accepted message. Existing plain/TLS/STARTTLS acceptance controls retain the
|
||||
unknown verdict and single transport attempt after a missing DATA reply.
|
||||
A real TLS handshake-refusal control preserves the TLS diagnosis of native
|
||||
TCP alerts. [Go's pinned TLS implementation](https://github.com/golang/go/blob/go1.26.9/src/crypto/tls/conn.go)
|
||||
wraps its unexported alert cause in a network-operation error, so generic
|
||||
network classification must retain that native cause's TLS meaning.
|
||||
|
||||
Current-source focused controls pass in 0.293s, the full notifications package
|
||||
passes in 136.861s, and the targeted notification race controls pass in 9.520s.
|
||||
Those include the real TLS-alert and SMTP retry controls and unchanged
|
||||
plain/TLS/STARTTLS missing-acceptance tests. Backend source is unchanged after
|
||||
that pass.
|
||||
Browser verification uses the existing production Notifications tab and API
|
||||
readers with explicitly synthetic response data. It verifies display of the
|
||||
reproduced verdicts, retained unknowns and recovery controls, separately from
|
||||
the backend wire and persistence proofs. It is not native provider acceptance.
|
||||
The affected matrix is the production tab fixture at
|
||||
`/browser-tests/notification-evidence.html`, at 1440x1000 and 390x1100:
|
||||
retained configuration/connectivity/opaque failures, safe diagnostics, shared
|
||||
terminal label and retention description, refresh, cancelled retry
|
||||
confirmation, focus return and reload. The fixture supplies read responses
|
||||
only and requires zero API mutations. Pixel review checks wrapping, clipping,
|
||||
control visibility and readable failure details at both widths.
|
||||
The final helper/card test run passes 51 tests across three files. The final
|
||||
browser pass succeeds after the copy change, with zero page errors, zero API
|
||||
mutations and no horizontal overflow. All four desktop/narrow page and activity
|
||||
captures were inspected. The frontend receipt binds the final helper content;
|
||||
the retained worker proof also hashes all three changed backend source files.
|
||||
|
||||
## Remaining scope
|
||||
|
||||
The existing notification-delivery-terminal-failure-diagnosis gap stays triaged.
|
||||
Destination-level repeated failures, destination recovery evidence and precise
|
||||
delivery-health guidance remain active. Opaque CLI/provider failures and
|
||||
unrecognised verdicts retain unknown classification when available evidence
|
||||
does not support a more precise class. Historical audit rows and aggregate
|
||||
counters are not rewritten. This slice does not publish or deploy a release.
|
||||
|
|
@ -10036,7 +10036,7 @@
|
|||
},
|
||||
{
|
||||
"id": "notification-delivery-terminal-failure-diagnosis",
|
||||
"summary": "The 18 October 2026 production summary retrieved at 10:01 UTC has 498 of 8,455 covered rotating IDs reporting terminal notification failures, with rolling totals of 120,040 failures and 246,419 deliveries. Connectivity (38,088) and authentication (27,959) are the largest classified sums; classification coverage is 8,113 IDs. These rolling aggregates do not identify a transport/provider or prove a release regression or recovery. Typed classification, deterministic-failure short circuiting and explicit retained-queue recovery already exist. Prior source slices repair SMTP provider-default authentication and concurrent envelope mutation, pinned HTTP fallback starvation, and webhook DNS continuing after cancellation. This authentication slice independently reproduces and repairs literal LOGIN challenge matching, substring AUTH mechanism selection, and port-465 connection tests attempting STARTTLS inside implicit TLS. Shared exact-name negotiation, a bounded prompt-independent LOGIN exchange and a canonical TLS-mode decision retain real 535 authentication verdicts, 454 retries and persisted queue/audit/telemetry outcomes. Unsupported mechanism lists now have a typed configuration verdict. Remaining work is destination-level repeated-failure handling, evidence of destination recovery, and precise delivery-health guidance. These source proofs do not establish which fleet failures they explain or that the production fleet has recovered.",
|
||||
"summary": "The 18 October 2026 fixed production summary retrieved at 12:23 UTC has 498 of 8,457 covered rotating IDs reporting terminal notification failures, with rolling totals of 115,318 failures and 248,005 deliveries. Unclassified failures total 17,748 with classification coverage of 8,117 IDs. Connectivity (38,299) and authentication (22,462) remain the largest classified sums. These cached rolling aggregates cannot identify providers, destinations, distinct new incidents or recovery. Previous source slices repair SMTP provider defaults, concurrent envelopes, pinned HTTP fallback, bounded DNS, LOGIN exchange order, AUTH selection and port-465 connection-test parity. This slice independently reproduces and repairs unknown or misclassified pre-acceptance connection closes, deterministic template and invalid retained-job failures spending retry budgets, and HTTP Apprise suppressing truncated response errors and inventing sent receipts. Shared structural classification, canonical template rendering and bounded HTTP response verdicts retain authoritative receiver replies, safe wrapped diagnostics, and explicit unconfirmed SMTP acceptance as unknown. Real wire and reopened queue/audit/receipt/telemetry controls cover the source boundaries. The shared delivery activity label describes retries stopped without inferring budget exhaustion from terminal status. Remaining work is destination-level repeated-failure handling, destination recovery evidence and precise delivery-health guidance. Truly opaque failures remain unknown, and source proofs do not establish which fleet counters these defects explain or that the production fleet has recovered.",
|
||||
"owner": "project-owner",
|
||||
"status": "triaged",
|
||||
"recorded_at": "2026-09-03",
|
||||
|
|
@ -10064,6 +10064,11 @@
|
|||
"path": "docs/release-control/v6/internal/records/notification-smtp-delivery-reliability-2026-10-17.md",
|
||||
"kind": "file"
|
||||
},
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "docs/release-control/v6/internal/records/notification-unclassified-reliability-2026-10-18.md",
|
||||
"kind": "file"
|
||||
},
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "frontend-modern/src/utils/alertDestinationsPresentation.ts",
|
||||
|
|
|
|||
|
|
@ -4142,7 +4142,11 @@ newest-first record of real alert delivery attempts with plain-language
|
|||
outcome labels, destination names resolved from the loaded webhook configs,
|
||||
failure classes, and secret-redacted error text. The card names its retention
|
||||
windows separately: completed attempts remain for seven days while
|
||||
dead-letter attempts can remain for 30 days. It requests the bounded maximum
|
||||
dead-letter attempts can remain for 30 days. Retained terminal failures say
|
||||
that retries stopped, because a permanent verdict can stop the first attempt
|
||||
before its retry budget is exhausted. The description uses the same boundary
|
||||
and never infers attempt exhaustion from dead-letter status alone.
|
||||
It requests the bounded maximum
|
||||
of 200 rows, displays an absolute semantic timestamp for timeline correlation
|
||||
with relative time as secondary hover context, and says that test sends skip
|
||||
the queue. An unreadable log renders as unavailable rather than empty. While
|
||||
|
|
|
|||
|
|
@ -1136,13 +1136,20 @@ The failure class is authoritative where the sender knows it and heuristic only
|
|||
where nobody does. `internal/notifications/failure_class.go` owns that order:
|
||||
a class declared by the sender through `NotificationFailureError` wins, then
|
||||
Go's own error types decide (`*textproto.Error` carries the SMTP reply code,
|
||||
x509 and TLS errors mean `tls`, `net.DNSError` and timeouts mean
|
||||
`connectivity`), and only then is the error message text consulted.
|
||||
x509 and TLS errors mean `tls`, while DNS errors, timeouts, failed network
|
||||
operations and propagated EOF/truncation mean `connectivity`), and only then
|
||||
is the error message text consulted. A connection closing during AUTH or TLS
|
||||
does not prove a credential or certificate verdict. The structured SMTP/TLS
|
||||
verdicts still outrank generic network evidence, and opaque failures remain
|
||||
`unknown` when no declared or structured evidence identifies them.
|
||||
Explicit `EmailAcceptanceUnconfirmedError` also remains `unknown`: loss of
|
||||
the final DATA acknowledgement must not be presented as a definitive delivery
|
||||
or rejection, nor as an ordinary pre-acceptance connection failure.
|
||||
|
||||
Two rules follow and are not optional. A destination's response body must never
|
||||
determine its own failure class: every site that builds an error from an HTTP
|
||||
status classifies from the status code, so the body is retained for the
|
||||
operator's audit row but cannot make a 500 that says "rate limit" record as
|
||||
status classifies from the status code, so arbitrary body text is withheld and
|
||||
cannot make a 500 that says "rate limit" record as
|
||||
`rate_limited`. And an SMTP reply code is not read like an HTTP status: a 5xx
|
||||
reply is the destination refusing the message (`rejected`, or `authentication`
|
||||
for 530/534/535/538 and `configuration` for the 500-504 syntax replies), while
|
||||
|
|
@ -1168,6 +1175,18 @@ them proves a later attempt fails. TLS is deliberately on the retrying side: a
|
|||
handshake can fail transiently during rotation, and the cost of one wasted
|
||||
ladder is lower than the cost of dropping a recoverable notification.
|
||||
|
||||
Payload template parse, execution and JSON-validation failures are declared
|
||||
configuration failures at the renderer. Ordinary and enhanced delivery,
|
||||
including plain-text ntfy, share that renderer; ntfy retains plain-text payloads
|
||||
without JSON validation. Malformed retained email, webhook and Apprise config,
|
||||
and unsupported retained notification types, are also configuration failures.
|
||||
These local failures stop on the first attempt rather than repeatedly trying a
|
||||
job that cannot be rendered or decoded. Diagnostics retain a fixed stage
|
||||
message and the wrapped cause for errors.Is/As, without copying configured
|
||||
template text, rendered payloads or private configuration into logs and audits.
|
||||
Retained configuration snapshots and explicit operator retry authority remain
|
||||
unchanged. This does not rewrite historical audit classes or private jobs.
|
||||
|
||||
`NotificationFailureClass.Retryable` is the single owner of that split. Neither
|
||||
the queue nor any destination type may keep its own list. This generalises to
|
||||
every destination the decision webhook delivery already made for HTTP 4xx in
|
||||
|
|
@ -1372,15 +1391,20 @@ HTTP or browser save path. It does not start a new operating-system process,
|
|||
drive an alert lifecycle or establish recipient delivery. Those installed
|
||||
acceptance obligations remain separate from constructor restoration coverage.
|
||||
|
||||
### Interrupted webhook rejection bodies retain the HTTP verdict
|
||||
### Interrupted HTTP response bodies retain the delivery verdict
|
||||
|
||||
When webhook response headers establish a non-2xx status, a subsequent body
|
||||
Webhooks and HTTP Apprise share the bounded `readWebhookResponse` verdict.
|
||||
When response headers establish a non-2xx status, a subsequent body
|
||||
read failure retains that HTTP status in the error and its authoritative
|
||||
notification failure class. Terminal rejections must not become connectivity
|
||||
retries merely because the diagnostic body is truncated. Response headers,
|
||||
including Retry-After, and the underlying read error remain available; partial
|
||||
body text is not added to the error. Existing handling of 2xx body read errors
|
||||
is unchanged: it remains a read failure, not proof of recipient receipt.
|
||||
body text is not added to the error. A 2xx response whose bounded body read
|
||||
fails remains a failure, not proof of recipient receipt. HTTP Apprise firing,
|
||||
recovery and Test must not suppress that error or mark the queued attempt sent.
|
||||
Only a successful bounded response verdict can create or consume a delivery
|
||||
receipt. Reaching the cap retains the reader's existing limit semantics. The
|
||||
response-byte cap, payloads, TLS/SSRF policy and retry budgets remain.
|
||||
|
||||
Transport-only `TestWebhookTruncatedResponseClassification` covers 200, 401,
|
||||
403, 422, 429 and 503 with interrupted bodies. `TestWebhookRetryTruncatedResponse`
|
||||
|
|
|
|||
38
frontend-modern/browser-verification/772a87707f4029d0.json
Normal file
38
frontend-modern/browser-verification/772a87707f4029d0.json
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
{
|
||||
"version": 1,
|
||||
"result": "passed",
|
||||
"base_sha": "98817ebce4e02545ca8870519ae929cee9c724c8",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/utils/alertDestinationsPresentation.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/utils/alertDestinationsPresentation.ts": "f7d9ba6b5fe5a698a14e966a68539de247b38d14781472e7a1d379a3cf765cd5"
|
||||
},
|
||||
"routes": [
|
||||
"/browser-tests/notification-evidence.html"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 1100
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Production Notifications tab with synthetic read responses for retained configuration, connectivity and opaque failures",
|
||||
"Retries stopped label and matching retention description, safe diagnostics and preserved unclassified verdict",
|
||||
"Desktop and narrow pixels inspected for readable wrapping, unclipped controls and all three activity rows",
|
||||
"Retained retry confirmation cancelled, focus restored and reloaded verdicts preserved"
|
||||
],
|
||||
"interactions": [
|
||||
"Refresh delivery status and observe all retained verdicts",
|
||||
"Open retained retry confirmation and dismiss it without an API mutation",
|
||||
"Verify focus returns to the retry button",
|
||||
"Reload and verify all three retries-stopped labels"
|
||||
],
|
||||
"verified_at": "2026-10-18T14:02:03.002Z",
|
||||
"evidence_scope": "Production components and API readers with synthetic local read responses. Separate real HTTP/SMTP and persistent-queue tests establish backend source verdicts. No native provider or installed-release acceptance is claimed."
|
||||
}
|
||||
|
|
@ -67,7 +67,7 @@ describe('AlertDeliveryLogCard', () => {
|
|||
|
||||
expect(screen.getByText('Ops Discord')).toBeInTheDocument();
|
||||
expect(screen.getByText('Email')).toBeInTheDocument();
|
||||
expect(screen.getByText('Failed, retries exhausted')).toBeInTheDocument();
|
||||
expect(screen.getByText('Failed, retries stopped')).toBeInTheDocument();
|
||||
expect(screen.getByText('Delivered')).toBeInTheDocument();
|
||||
expect(screen.getByText('vm-offline-101 +1 more')).toBeInTheDocument();
|
||||
expect(screen.getByText(/Authentication failure/)).toBeInTheDocument();
|
||||
|
|
@ -91,7 +91,7 @@ describe('AlertDeliveryLogCard', () => {
|
|||
it.each([
|
||||
['retry', 'Retrying'],
|
||||
['failed', 'Failed'],
|
||||
['dead_letter', 'Failed, retries exhausted'],
|
||||
['dead_letter', 'Failed, retries stopped'],
|
||||
['cancelled', 'Cancelled'],
|
||||
] as const)(
|
||||
'replaces a previous successful row with %s evidence on refresh',
|
||||
|
|
@ -156,7 +156,7 @@ describe('AlertDeliveryLogCard', () => {
|
|||
|
||||
expect(screen.getByText(/Completed attempts are retained for 7 days/)).toBeInTheDocument();
|
||||
expect(
|
||||
screen.getByText(/Failures that exhausted retries remain available for 30 days/),
|
||||
screen.getByText(/Deliveries Pulse has stopped retrying remain available for 30 days/),
|
||||
).toBeInTheDocument();
|
||||
const timestamps = Array.from(container.querySelectorAll('time'));
|
||||
expect(timestamps[0]).toHaveAttribute('datetime', log.entries[0].timestamp);
|
||||
|
|
|
|||
|
|
@ -237,7 +237,7 @@ describe('alert destinations delivery log copy', () => {
|
|||
expect(getAlertDeliveryLogOutcomeLabel('retry')).toBe('Retrying');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('failed')).toBe('Failed');
|
||||
// "Dead letter" is queue vocabulary; a user needs to know retries stopped.
|
||||
expect(getAlertDeliveryLogOutcomeLabel('dead_letter')).toBe('Failed, retries exhausted');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('dead_letter')).toBe('Failed, retries stopped');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('cancelled')).toBe('Cancelled');
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -307,7 +307,7 @@ export function getAlertDestinationsDeliveryLogDescription(
|
|||
completedRetentionDays: number,
|
||||
deadLetterRetentionDays: number,
|
||||
) {
|
||||
return `Delivery attempts and held notifications for real alerts. Completed attempts are retained for ${completedRetentionDays} days. Failures that exhausted retries remain available for ${deadLetterRetentionDays} days. Test sends skip the queue and are not listed here.`;
|
||||
return `Delivery attempts and held notifications for real alerts. Completed attempts are retained for ${completedRetentionDays} days. Deliveries Pulse has stopped retrying remain available for ${deadLetterRetentionDays} days. Test sends skip the queue and are not listed here.`;
|
||||
}
|
||||
|
||||
export function getAlertDestinationsDeliveryLogEmpty() {
|
||||
|
|
@ -329,7 +329,7 @@ export function getAlertDeliveryLogOutcomeLabel(outcome: AlertDeliveryLogOutcome
|
|||
case 'retry':
|
||||
return 'Retrying';
|
||||
case 'dead_letter':
|
||||
return 'Failed, retries exhausted';
|
||||
return 'Failed, retries stopped';
|
||||
case 'cancelled':
|
||||
return 'Cancelled';
|
||||
default:
|
||||
|
|
|
|||
103
internal/notifications/apprise_response_verdict_test.go
Normal file
103
internal/notifications/apprise_response_verdict_test.go
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
package notifications
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestAppriseInterruptedResponseRetainsVerdict(t *testing.T) {
|
||||
for _, kind := range []string{"firing", "resolved", "test"} {
|
||||
for _, code := range []int{200, 401, 429, 503} {
|
||||
t.Run(kind+"/"+strconv.Itoa(code), func(t *testing.T) {
|
||||
captured := captureAppriseLogs(t)
|
||||
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Length", "1000")
|
||||
w.WriteHeader(code)
|
||||
_, _ = fmt.Fprint(w, "provider-private-content ", appriseSecret)
|
||||
}))
|
||||
defer server.Close()
|
||||
manager := &NotificationManager{}
|
||||
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
|
||||
err := sendAppriseForConfidentiality(manager, AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL}, kind)
|
||||
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
|
||||
want := ClassFromHTTPStatus(code)
|
||||
if code == 200 {
|
||||
want = NotificationFailureConnectivity
|
||||
}
|
||||
require.Equal(t, want, ClassifyNotificationFailureError(err))
|
||||
assertAppriseConfidential(t, err.Error())
|
||||
assertAppriseConfidential(t, captured.String())
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueuedAppriseInterruptedResponseDoesNotInventReceipt(t *testing.T) {
|
||||
for _, complete := range []bool{false, true} {
|
||||
t.Run(strconv.FormatBool(complete), func(t *testing.T) {
|
||||
var requests atomic.Int32
|
||||
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
requests.Add(1)
|
||||
if !complete {
|
||||
w.Header().Set("Content-Length", "1000")
|
||||
}
|
||||
_, _ = fmt.Fprint(w, "provider-private-content")
|
||||
}))
|
||||
defer server.Close()
|
||||
dir := t.TempDir()
|
||||
manager := NewNotificationManagerWithDeferredQueue("", dir)
|
||||
t.Cleanup(manager.Stop)
|
||||
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
|
||||
cfg := NormalizeAppriseConfig(AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL})
|
||||
manager.SetAppriseConfig(cfg)
|
||||
config, err := json.Marshal(cfg)
|
||||
require.NoError(t, err)
|
||||
alert := &alerts.Alert{ID: "response-verdict", ResourceName: "synthetic-node", Message: "synthetic alert", Level: alerts.AlertLevelWarning, StartTime: time.Now()}
|
||||
// Admit one retained job with a one-attempt budget to observe an
|
||||
// exhausted verdict through the actual processor and durable queue.
|
||||
require.NoError(t, manager.queue.Enqueue(&QueuedNotification{ID: "response-verdict", Type: "apprise", Config: config, MaxAttempts: 1, Alerts: []*alerts.Alert{alert}}))
|
||||
manager.StartQueueProcessing()
|
||||
waitVerdictAudit(t, manager.queue, "response-verdict")
|
||||
manager.Stop()
|
||||
reopened := NewNotificationManagerWithDeferredQueue("", dir)
|
||||
t.Cleanup(reopened.Stop)
|
||||
var status, class string
|
||||
var attempts, success int
|
||||
require.NoError(t, reopened.queue.db.QueryRow(`SELECT status, attempts FROM notification_queue WHERE id = 'response-verdict'`).Scan(&status, &attempts))
|
||||
require.NoError(t, reopened.queue.db.QueryRow(`SELECT success, failure_class FROM notification_audit WHERE notification_id = 'response-verdict'`).Scan(&success, &class))
|
||||
var receipts int
|
||||
require.NoError(t, reopened.queue.db.QueryRow(`SELECT count(*) FROM notification_delivery_receipts`).Scan(&receipts))
|
||||
stats, err := reopened.GetTelemetryStats(time.Now().Add(-time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, int32(1), requests.Load())
|
||||
require.Equal(t, 1, attempts)
|
||||
require.Equal(t, 1, stats.Attempts)
|
||||
if complete {
|
||||
require.Equal(t, string(QueueStatusSent), status)
|
||||
require.Equal(t, 1, success)
|
||||
require.Empty(t, class)
|
||||
require.Equal(t, 1, receipts)
|
||||
require.Equal(t, 1, stats.Deliveries)
|
||||
require.Zero(t, stats.Failures)
|
||||
} else {
|
||||
require.Equal(t, string(QueueStatusDLQ), status)
|
||||
require.Zero(t, success)
|
||||
require.Equal(t, string(NotificationFailureConnectivity), class)
|
||||
require.Zero(t, receipts)
|
||||
require.Zero(t, stats.Deliveries)
|
||||
require.Equal(t, 1, stats.Failures)
|
||||
require.Equal(t, 1, stats.FailureClasses.Connectivity)
|
||||
}
|
||||
require.Zero(t, stats.FailureClasses.Unknown)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -6,8 +6,10 @@ import (
|
|||
"crypto/x509"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/textproto"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
|
|
@ -62,6 +64,22 @@ func FailfWithClass(class NotificationFailureClass, format string, args ...any)
|
|||
return &NotificationFailureError{Class: class, Err: fmt.Errorf(format, args...)}
|
||||
}
|
||||
|
||||
// Local decoding/rendering errors can include private configuration or alert
|
||||
// content. Retain their causes without putting that prose in diagnostics.
|
||||
type notificationConfigurationError struct {
|
||||
message string
|
||||
cause error
|
||||
}
|
||||
|
||||
func (e *notificationConfigurationError) Error() string {
|
||||
return e.message + " (configuration details withheld)"
|
||||
}
|
||||
func (e *notificationConfigurationError) Unwrap() error { return e.cause }
|
||||
|
||||
func failNotificationConfiguration(message string, cause error) error {
|
||||
return FailWithClass(NotificationFailureConfiguration, ¬ificationConfigurationError{message: message, cause: cause})
|
||||
}
|
||||
|
||||
// ClassFromHTTPStatus maps an HTTP response status to its delivery class.
|
||||
// Status codes are the destination's own structured verdict, so they outrank
|
||||
// anything the response body happens to say.
|
||||
|
|
@ -131,6 +149,14 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
|
|||
return declared.Class
|
||||
}
|
||||
|
||||
// Losing the final DATA acknowledgement leaves SMTP acceptance unknown.
|
||||
// That explicit uncertainty must not become an ordinary connection verdict
|
||||
// merely because the retained cause is EOF or another network error.
|
||||
var unconfirmed *EmailAcceptanceUnconfirmedError
|
||||
if errors.As(err, &unconfirmed) {
|
||||
return NotificationFailureUnknown
|
||||
}
|
||||
|
||||
// An SMTP reply code is the mail server's structured verdict. net/smtp
|
||||
// surfaces it as *textproto.Error however deeply it is wrapped.
|
||||
var protoErr *textproto.Error
|
||||
|
|
@ -159,6 +185,8 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
|
|||
}
|
||||
|
||||
if errors.Is(err, context.DeadlineExceeded) ||
|
||||
errors.Is(err, io.EOF) ||
|
||||
errors.Is(err, io.ErrUnexpectedEOF) ||
|
||||
errors.Is(err, syscall.ECONNREFUSED) ||
|
||||
errors.Is(err, syscall.ECONNRESET) ||
|
||||
errors.Is(err, syscall.EHOSTUNREACH) ||
|
||||
|
|
@ -168,6 +196,22 @@ func ClassifyNotificationFailureError(err error) NotificationFailureClass {
|
|||
return NotificationFailureConnectivity
|
||||
}
|
||||
|
||||
// A failed network operation remains connectivity evidence even when its
|
||||
// cause has no recognised prose or the sender's stage mentions AUTH/TLS.
|
||||
// Structured SMTP replies and certificate failures have already won above.
|
||||
var operationErr *net.OpError
|
||||
if errors.As(err, &operationErr) {
|
||||
// Go's TCP TLS alerts have an unexported cause inside net.OpError.
|
||||
// Retain their native TLS diagnosis, inspecting the cause rather than
|
||||
// a sender's AUTH/TLS stage text around an ordinary connection error.
|
||||
for cause := operationErr.Err; cause != nil; cause = errors.Unwrap(cause) {
|
||||
if strings.HasPrefix(cause.Error(), "tls:") {
|
||||
return NotificationFailureTLS
|
||||
}
|
||||
}
|
||||
return NotificationFailureConnectivity
|
||||
}
|
||||
|
||||
var netErr net.Error
|
||||
if errors.As(err, &netErr) && netErr.Timeout() {
|
||||
return NotificationFailureConnectivity
|
||||
|
|
|
|||
|
|
@ -10,7 +10,6 @@ import (
|
|||
"encoding/json"
|
||||
"fmt"
|
||||
"html"
|
||||
"io"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
|
|
@ -2513,18 +2512,22 @@ func (n *NotificationManager) sendAppriseViaHTTP(cfg AppriseConfig, title, body,
|
|||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
limited := io.LimitReader(resp.Body, WebhookMaxResponseSize)
|
||||
responseBytes, _ := io.Copy(io.Discard, limited)
|
||||
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return FailfWithClass(ClassFromHTTPStatus(resp.StatusCode), "apprise server returned HTTP %d", resp.StatusCode)
|
||||
// Share the bounded response verdict with webhooks. Receiving 2xx headers
|
||||
// does not make a failed body read successful delivery or a durable receipt.
|
||||
result, err := readWebhookResponse(resp)
|
||||
if err != nil {
|
||||
operation := "read Apprise server response"
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
operation = fmt.Sprintf("apprise server returned HTTP %d", resp.StatusCode)
|
||||
}
|
||||
return safeAppriseError(operation, err)
|
||||
}
|
||||
|
||||
if responseBytes > 0 {
|
||||
if result.responseBytes > 0 {
|
||||
log.Debug().
|
||||
Str("mode", string(cfg.Mode)).
|
||||
Int("statusCode", resp.StatusCode).
|
||||
Int64("responseBytes", responseBytes).
|
||||
Int64("responseBytes", result.responseBytes).
|
||||
Msg("apprise API response")
|
||||
}
|
||||
|
||||
|
|
@ -3557,12 +3560,12 @@ func templateFuncMap() template.FuncMap {
|
|||
func (n *NotificationManager) generatePayloadFromTemplateWithService(templateStr string, data WebhookPayloadData, service string) ([]byte, error) {
|
||||
tmpl, err := template.New("webhook").Funcs(templateFuncMap()).Parse(templateStr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid template: %w", err)
|
||||
return nil, failNotificationConfiguration("invalid template", err)
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := tmpl.Execute(&buf, data); err != nil {
|
||||
return nil, fmt.Errorf("template execution failed: %w", err)
|
||||
return nil, failNotificationConfiguration("template execution failed", err)
|
||||
}
|
||||
|
||||
// Skip JSON validation for services that use plain text payloads
|
||||
|
|
@ -3575,10 +3578,9 @@ func (n *NotificationManager) generatePayloadFromTemplateWithService(templateStr
|
|||
var jsonCheck interface{}
|
||||
if err := json.Unmarshal(buf.Bytes(), &jsonCheck); err != nil {
|
||||
log.Error().
|
||||
Err(err).
|
||||
Str("payload", buf.String()).
|
||||
Int("payloadBytes", buf.Len()).
|
||||
Msg("generated webhook payload is invalid JSON")
|
||||
return nil, fmt.Errorf("template produced invalid JSON: %w", err)
|
||||
return nil, failNotificationConfiguration("template produced invalid JSON", err)
|
||||
}
|
||||
|
||||
return buf.Bytes(), nil
|
||||
|
|
@ -4243,7 +4245,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
|
|||
case "email":
|
||||
var emailConfig EmailConfig
|
||||
if err = json.Unmarshal(notif.Config, &emailConfig); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal email config: %w", err)
|
||||
return failNotificationConfiguration("failed to unmarshal email config", err)
|
||||
}
|
||||
currentEmailConfig := n.GetEmailConfig()
|
||||
if !n.IsEnabled() || !currentEmailConfig.Enabled {
|
||||
|
|
@ -4266,7 +4268,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
|
|||
case "webhook":
|
||||
var webhookConfig WebhookConfig
|
||||
if err = json.Unmarshal(notif.Config, &webhookConfig); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal webhook config: %w", err)
|
||||
return failNotificationConfiguration("failed to unmarshal webhook config", err)
|
||||
}
|
||||
if !n.IsEnabled() || !n.isQueuedWebhookStillEnabled(webhookConfig) {
|
||||
log.Info().
|
||||
|
|
@ -4289,7 +4291,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
|
|||
case "apprise":
|
||||
var appriseConfig AppriseConfig
|
||||
if err = json.Unmarshal(notif.Config, &appriseConfig); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal apprise config: %w", err)
|
||||
return failNotificationConfiguration("failed to unmarshal apprise config", err)
|
||||
}
|
||||
currentAppriseConfig := n.GetAppriseConfig()
|
||||
if !n.IsEnabled() || !currentAppriseConfig.Enabled {
|
||||
|
|
@ -4310,7 +4312,7 @@ func (n *NotificationManager) ProcessQueuedNotification(notif *QueuedNotificatio
|
|||
err = n.deliverNotificationJob(deliveredJob)
|
||||
|
||||
default:
|
||||
return fmt.Errorf("unknown notification type: %s", baseType)
|
||||
return FailfWithClass(NotificationFailureConfiguration, "unknown notification type")
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
|
|
|
|||
326
internal/notifications/unclassified_failures_test.go
Normal file
326
internal/notifications/unclassified_failures_test.go
Normal file
|
|
@ -0,0 +1,326 @@
|
|||
package notifications
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/textproto"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"text/template"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestUnclassifiedTLSAlertRetainsTLSVerdict(t *testing.T) {
|
||||
listener, err := tls.Listen("tcp4", "127.0.0.1:0", &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
|
||||
return nil, errors.New("synthetic handshake refusal")
|
||||
}})
|
||||
require.NoError(t, err)
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
conn, err := listener.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
_ = conn.SetDeadline(time.Now().Add(3 * time.Second))
|
||||
_ = conn.(*tls.Conn).Handshake()
|
||||
}()
|
||||
t.Cleanup(func() { _ = listener.Close(); <-done })
|
||||
client, err := tls.DialWithDialer(&net.Dialer{Timeout: time.Second}, "tcp4", listener.Addr().String(), &tls.Config{InsecureSkipVerify: true})
|
||||
if client != nil {
|
||||
defer client.Close()
|
||||
}
|
||||
require.Error(t, err)
|
||||
var operation *net.OpError
|
||||
require.ErrorAs(t, err, &operation)
|
||||
require.Equal(t, NotificationFailureTLS, ClassifyNotificationFailureError(err))
|
||||
}
|
||||
|
||||
func TestUnclassifiedWireErrorsUseStructuralEvidence(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
err error
|
||||
want NotificationFailureClass
|
||||
}{
|
||||
{"plain EOF", fmt.Errorf("send failed: %w", io.EOF), NotificationFailureConnectivity},
|
||||
{"authentication stage EOF", fmt.Errorf("SMTP auth failed: %w", io.EOF), NotificationFailureConnectivity},
|
||||
{"TLS stage EOF", fmt.Errorf("TLS dial failed: %w", io.EOF), NotificationFailureConnectivity},
|
||||
{"truncation with misleading context", fmt.Errorf("authentication failed: %w", io.ErrUnexpectedEOF), NotificationFailureConnectivity},
|
||||
{"opaque network operation", &net.OpError{Op: "read", Net: "tcp", Err: errors.New("opaque wire interruption")}, NotificationFailureConnectivity},
|
||||
{"declared rejection wins", FailWithClass(NotificationFailureRejected, io.EOF), NotificationFailureRejected},
|
||||
{"unconfirmed SMTP acceptance remains unknown", &EmailAcceptanceUnconfirmedError{Err: io.EOF}, NotificationFailureUnknown},
|
||||
{"SMTP reply wins", &textproto.Error{Code: 535, Msg: "receiver disconnected"}, NotificationFailureAuthentication},
|
||||
{"certificate wins", &net.OpError{Op: "dial", Net: "tcp", Err: x509.UnknownAuthorityError{}}, NotificationFailureTLS},
|
||||
{"opaque error remains unknown", errors.New("opaque delivery failure"), NotificationFailureUnknown},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
require.Equal(t, tc.want, ClassifyNotificationFailureError(tc.err))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnclassifiedClosedHTTPConnection(t *testing.T) {
|
||||
for _, kind := range []string{"webhook", "apprise"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
conn, _, err := w.(http.Hijacker).Hijack()
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
_ = conn.Close()
|
||||
}))
|
||||
defer server.Close()
|
||||
manager := &NotificationManager{webhookClient: server.Client(), webhookRateLimits: make(map[string]*webhookRateLimit)}
|
||||
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
|
||||
var err error
|
||||
if kind == "webhook" {
|
||||
err = manager.sendWebhookRequest(WebhookConfig{URL: server.URL}, []byte(`{"synthetic":true}`), "test", "synthetic-event")
|
||||
} else {
|
||||
err = manager.SendTestAppriseWithConfig(AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: server.URL})
|
||||
}
|
||||
require.ErrorIs(t, err, io.EOF)
|
||||
require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnclassifiedClosedSMTPConnection(t *testing.T) {
|
||||
for _, stage := range []string{"greeting", "authentication"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
listener, err := net.Listen("tcp4", "127.0.0.1:0")
|
||||
require.NoError(t, err)
|
||||
var sessions sync.WaitGroup
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for {
|
||||
conn, err := listener.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
sessions.Add(1)
|
||||
go func() {
|
||||
defer sessions.Done()
|
||||
defer conn.Close()
|
||||
if stage == "greeting" {
|
||||
return
|
||||
}
|
||||
_ = conn.SetDeadline(time.Now().Add(3 * time.Second))
|
||||
wire := textproto.NewConn(conn)
|
||||
if wire.PrintfLine("220 localhost ESMTP") != nil {
|
||||
return
|
||||
}
|
||||
if _, err := wire.ReadLine(); err != nil {
|
||||
return
|
||||
}
|
||||
if wire.PrintfLine("250-localhost\r\n250 AUTH LOGIN") != nil {
|
||||
return
|
||||
}
|
||||
_, _ = wire.ReadLine() // Close before answering AUTH, with no credential verdict.
|
||||
}()
|
||||
}
|
||||
}()
|
||||
t.Cleanup(func() { _ = listener.Close(); <-done; sessions.Wait() })
|
||||
_, port, err := net.SplitHostPort(listener.Addr().String())
|
||||
require.NoError(t, err)
|
||||
p, err := strconv.Atoi(port)
|
||||
require.NoError(t, err)
|
||||
manager := NewEnhancedEmailManager(EmailProviderConfig{EmailConfig: EmailConfig{
|
||||
SMTPHost: "127.0.0.1", SMTPPort: p, From: "pulse@example.test", To: []string{"ops@example.test"},
|
||||
Username: "synthetic-user", Password: "synthetic-password",
|
||||
}})
|
||||
err = manager.SendEmailWithRetry("synthetic", "<p>synthetic</p>", "synthetic")
|
||||
require.ErrorIs(t, err, io.EOF)
|
||||
require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnclassifiedSMTPAuthDisconnectRecoversWithTransportRetry(t *testing.T) {
|
||||
config, receiver := smtpAuthenticationReceiver(t, "plain", "LOGIN", []string{"User Name", "Password"}, 235)
|
||||
upstreamAddress := net.JoinHostPort(config.SMTPHost, strconv.Itoa(config.SMTPPort))
|
||||
listener, err := net.Listen("tcp4", "127.0.0.1:0")
|
||||
require.NoError(t, err)
|
||||
var connections atomic.Int32
|
||||
var sessions sync.WaitGroup
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for {
|
||||
conn, err := listener.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
attempt := connections.Add(1)
|
||||
sessions.Add(1)
|
||||
go func() {
|
||||
defer sessions.Done()
|
||||
defer conn.Close()
|
||||
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
|
||||
if attempt == 1 {
|
||||
wire := textproto.NewConn(conn)
|
||||
if wire.PrintfLine("220 localhost ESMTP") != nil {
|
||||
return
|
||||
}
|
||||
if _, err := wire.ReadLine(); err != nil {
|
||||
return
|
||||
}
|
||||
if wire.PrintfLine("250-localhost\r\n250 AUTH LOGIN") != nil {
|
||||
return
|
||||
}
|
||||
_, _ = wire.ReadLine() // First AUTH exchange loses its connection.
|
||||
return
|
||||
}
|
||||
// Later connections reach the real credential-checking receiver
|
||||
// behind the same configured endpoint, without replacing SMTP.
|
||||
upstream, err := net.DialTimeout("tcp4", upstreamAddress, time.Second)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
defer upstream.Close()
|
||||
_ = upstream.SetDeadline(time.Now().Add(5 * time.Second))
|
||||
copied := make(chan struct{})
|
||||
go func() { _, _ = io.Copy(upstream, conn); close(copied) }()
|
||||
_, _ = io.Copy(conn, upstream)
|
||||
_ = conn.Close()
|
||||
_ = upstream.Close()
|
||||
<-copied
|
||||
}()
|
||||
}
|
||||
}()
|
||||
t.Cleanup(func() { _ = listener.Close(); <-done; sessions.Wait() })
|
||||
_, port, err := net.SplitHostPort(listener.Addr().String())
|
||||
require.NoError(t, err)
|
||||
config.SMTPPort, err = strconv.Atoi(port)
|
||||
require.NoError(t, err)
|
||||
config.MaxRetries = 1
|
||||
require.NoError(t, NewEnhancedEmailManager(config).SendEmailWithRetry("synthetic", "<p>synthetic</p>", "synthetic"))
|
||||
require.Equal(t, int32(2), connections.Load())
|
||||
require.Equal(t, int32(1), receiver.accepted.Load())
|
||||
require.Equal(t, int32(2), receiver.credentials.Load())
|
||||
}
|
||||
|
||||
func TestUnclassifiedPayloadTemplateFailuresAreConfiguration(t *testing.T) {
|
||||
for _, tc := range []struct{ name, source string }{
|
||||
{"parse", `{"message":"{{synthetic_private_function}}"}`},
|
||||
{"execution", `{"message":"{{.Absent}}"}`},
|
||||
{"JSON", `{"message":"{{.Message}}"}`},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
captured := captureAppriseLogs(t)
|
||||
_, err := (&NotificationManager{}).generatePayloadFromTemplateWithService(tc.source,
|
||||
WebhookPayloadData{Message: "synthetic-private-message\"invalid"}, "generic")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
|
||||
require.NotContains(t, err.Error(), "synthetic_private_function")
|
||||
require.NotContains(t, captured.String(), "synthetic-private-message")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnclassifiedEnhancedNtfyUsesCanonicalTemplateVerdict(t *testing.T) {
|
||||
for _, action := range []string{"send", "test"} {
|
||||
t.Run(action, func(t *testing.T) {
|
||||
manager := &NotificationManager{}
|
||||
config := EnhancedWebhookConfig{WebhookConfig: WebhookConfig{URL: "http://127.0.0.1:1"}, Service: "ntfy", PayloadTemplate: `{{.Absent}}`}
|
||||
var err error
|
||||
if action == "send" {
|
||||
err = manager.SendEnhancedWebhook(config, &alerts.Alert{ID: "synthetic", StartTime: time.Now()})
|
||||
} else {
|
||||
_, _, err = manager.TestEnhancedWebhook(config)
|
||||
}
|
||||
require.Error(t, err)
|
||||
require.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
|
||||
var execution template.ExecError
|
||||
require.ErrorAs(t, err, &execution)
|
||||
require.NotContains(t, err.Error(), "Absent")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnclassifiedQueuedTemplateStopsWithoutSending(t *testing.T) {
|
||||
var requests atomic.Int32
|
||||
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { requests.Add(1); w.WriteHeader(204) }))
|
||||
defer server.Close()
|
||||
dir := t.TempDir()
|
||||
manager := NewNotificationManagerWithDeferredQueue("", dir)
|
||||
t.Cleanup(manager.Stop)
|
||||
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
|
||||
manager.SetGroupingConfig(false, 0, false, false)
|
||||
manager.AddWebhook(WebhookConfig{ID: "synthetic-template", Enabled: true, URL: server.URL, Template: `{"message":"{{.Absent}}"}`})
|
||||
manager.SendAlert(&alerts.Alert{ID: "template-failure", ResourceName: "synthetic-node", Level: alerts.AlertLevelWarning, StartTime: time.Now()})
|
||||
pending, err := manager.queue.GetPending(10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, pending, 1)
|
||||
manager.StartQueueProcessing()
|
||||
waitVerdictAudit(t, manager.queue, pending[0].ID)
|
||||
manager.Stop()
|
||||
queue, err := NewNotificationQueue(dir)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = queue.Stop() })
|
||||
var status, class, message string
|
||||
var attempts int
|
||||
require.NoError(t, queue.db.QueryRow(`SELECT status, attempts, failure_class, error_message FROM notification_audit WHERE notification_id = ?`, pending[0].ID).Scan(&status, &attempts, &class, &message))
|
||||
require.Equal(t, string(QueueStatusDLQ), status)
|
||||
require.Equal(t, 1, attempts)
|
||||
require.Equal(t, string(NotificationFailureConfiguration), class)
|
||||
require.True(t, strings.Contains(message, "template execution failed"))
|
||||
stats, err := queue.GetTelemetryStats(time.Now().Add(-time.Hour))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, stats.Failures)
|
||||
require.Equal(t, 1, stats.FailureClasses.Configuration)
|
||||
require.Zero(t, stats.FailureClasses.Unknown)
|
||||
require.Zero(t, stats.Deliveries)
|
||||
require.Zero(t, requests.Load())
|
||||
}
|
||||
|
||||
func TestUnclassifiedRetainedConfigurationStopsWithoutRetry(t *testing.T) {
|
||||
for _, kind := range []string{"email", "webhook", "apprise", "unsupported"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
manager := NewNotificationManagerWithDeferredQueue("", dir)
|
||||
t.Cleanup(manager.Stop)
|
||||
config := []byte(`{"synthetic-private-field":`)
|
||||
if kind == "unsupported" {
|
||||
config = []byte(`{}`)
|
||||
}
|
||||
notif := &QueuedNotification{ID: "invalid-retained-config", Type: kind, Config: config, MaxAttempts: 3}
|
||||
err := manager.ProcessQueuedNotification(notif)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, NotificationFailureConfiguration, ClassifyNotificationFailureError(err))
|
||||
if kind != "unsupported" {
|
||||
var syntax *json.SyntaxError
|
||||
require.ErrorAs(t, err, &syntax)
|
||||
}
|
||||
require.NoError(t, manager.queue.Enqueue(notif))
|
||||
manager.StartQueueProcessing()
|
||||
waitVerdictAudit(t, manager.queue, notif.ID)
|
||||
manager.Stop()
|
||||
queue, err := NewNotificationQueue(dir)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = queue.Stop() })
|
||||
var status, class string
|
||||
var attempts int
|
||||
require.NoError(t, queue.db.QueryRow(`SELECT status, attempts, failure_class FROM notification_audit WHERE notification_id = ?`, notif.ID).Scan(&status, &attempts, &class))
|
||||
require.Equal(t, string(QueueStatusDLQ), status)
|
||||
require.Equal(t, 1, attempts)
|
||||
require.Equal(t, string(NotificationFailureConfiguration), class)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -1,14 +1,12 @@
|
|||
package notifications
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/template"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
|
|
@ -169,18 +167,6 @@ func (n *NotificationManager) prepareEnhancedWebhookExecution(webhook EnhancedWe
|
|||
}
|
||||
webhook.WebhookConfig = canonical
|
||||
|
||||
if webhook.Service == "ntfy" && strings.TrimSpace(webhook.PayloadTemplate) != "" {
|
||||
tmpl, err := template.New("enhanced-webhook-ntfy").Funcs(templateFuncMap()).Parse(webhook.PayloadTemplate)
|
||||
if err != nil {
|
||||
return webhook, nil, fmt.Errorf("parse ntfy payload template: %w", err)
|
||||
}
|
||||
var rendered bytes.Buffer
|
||||
if err := tmpl.Execute(&rendered, data); err != nil {
|
||||
return webhook, nil, fmt.Errorf("render ntfy payload template: %w", err)
|
||||
}
|
||||
return webhook, rendered.Bytes(), nil
|
||||
}
|
||||
|
||||
payload, err := n.renderWebhookPayloadJSON(canonical, data, webhookRenderModeSingle, func() ([]byte, error) {
|
||||
if strings.TrimSpace(webhook.PayloadTemplate) == "" {
|
||||
return nil, fmt.Errorf("enhanced webhook requires payload template or known service template")
|
||||
|
|
|
|||
|
|
@ -407,7 +407,7 @@ test.describe('Alert operator qualification', () => {
|
|||
).toBeVisible();
|
||||
await expect(
|
||||
page.getByText(
|
||||
'Completed attempts are retained for 7 days. Failures that exhausted retries remain available for 30 days.',
|
||||
'Completed attempts are retained for 7 days. Deliveries Pulse has stopped retrying remain available for 30 days.',
|
||||
{ exact: false },
|
||||
),
|
||||
).toBeVisible();
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue