Skip to content

CIS Notification Framework — Canonical Architecture & Developer Guide

[!IMPORTANT] This document is the authoritative specification and developer reference ("The Bible") for the CIS Notification Framework. It governs all notification channels (EMAIL, SMS, PDF), asynchronous event outbox queues, workflow process notifications, template resolution and caching, document artifact generation, distributed idempotency, audit ledgers, and developer extension rules across cis-api.


1. Scope, Responsibilities and Package Ownership

Package Ownership

service.notification
  NotificationService          public facade
  NotificationFrameworkGateway synchronous application gateway facade
  model/                       shared channel, request and failure-policy types
  context/                     context builder and shared/domain contributors
    cart/, lodgement/,         domain-specific context contributors
    maintenance/, payment/,
    reservation/, workflow/
  render/                      preparation, adapter registry and render results
  template/                    validation, cache invalidation and token checking
  channel/
    pdf/                       PDF payload, rendering, preview and delivery adapter
    email/                     email payload, delivery and attachment naming
    sms/                       SMS payload and delivery
  delivery/                    generic retries, artifact handling and delivery results
  artifact/                    artifact processor contract and workflow document storage
  asset/                       image/barcode resolution, replacement factory and cache
  audit/                       durable delivery observations and JPA sink
  idempotency/                 event identity and renewable Redis ownership
  queue/                       durable queue worker, outbox handler and workflow notification dispatch

Framework tests mirror these responsibility packages. Cross-cutting safety tests remain beside the public facade; template composition tests remain in service.template. Packages do not add new Spring configuration: existing component scanning covers the subpackages, and the original class names, cache names, and persistence contracts are retained.

The framework composes database-managed templates and delivers PDF, EMAIL, and SMS through a shared strategy registry. All 28 active workflow notifications, the generic event listener, public email/SMS REST resources, lodgement/reservation action notifications, search sharing, expiry reminders, and task reminders enter this framework through either the durable queue (GenericEventEmailNotificationEnqueuer) or NotificationFrameworkGateway.

Core Hardening Invariants

  1. Fail-Fast Registries: The registries fail fast if two adapters or delivery strategies claim the same channel, preventing bean ordering from silently altering runtime behavior.
  2. Strict Boundary Validation: Render and delivery boundaries reject null commands, null contexts, empty channel selections, invalid retry counts, and rendered results whose map key does not match their declared channel.
  3. Payload Immutability: Prepared contexts and channel payloads remain strictly immutable once constructed.
  4. Identity Fallbacks: Legacy callers may omit template/workflow IDs when they provide an explicit notification event ID (notification:<id>:<channel>); direct callers without an event ID retain the fallback workflowId:templateId:channel.
  5. Deterministic Channel Sequencing: NotificationService sequences dependent channels; NotificationDeliveryOrchestrator owns channel lookup, ownership, retries, artifact processing, recovery, and audit. PDF delivery runs before email only when PDF is requested because the generated resource is an input to the email attachment. Domain contributors populate template variables, while transport strategies perform physical I/O.

2. Global Architecture & Component Topology

flowchart TD
    subgraph Ingress["1. Ingress Layer (Trigger Sources)"]
        WF["Workflow Engine (ProcessNotificationsVm)"]
        EV["GenericEventListener (Domain Lifecycle Events)"]
        SCH["Schedulers (TaskReminderService, Expiry Reminders)"]
        REST["REST API Endpoints (EmailResource, SmsResource)"]
        PREV["Interactive Letter Preview (Letter Previews)"]
    end

    subgraph EntryPorts["2. Ingress Gateways & Outbox"]
        GW["NotificationFrameworkGateway (Sync Facade)"]
        ENQ["GenericEventEmailNotificationEnqueuer (Outbox Enqueuer)"]
        PREV_SVC["NotificationPdfPreviewService (Preview Engine)"]
        DB_QUEUE[("NOTIFICATION Table (Outbox Queue)")]
        Q_WORKER["NotificationQueueService (Claim & Poller)"]
        GEN_SVC["GenericNotificationService (Dispatcher)"]
        WF_FACTORY["ProcessNotificationFactory (28 Active Mappings)"]
    end

    subgraph PrepLayer["3. Preparation & Context Assembly Pipeline"]
        NPS["NotificationPreparationService (Read-Only Tx)"]
        NCB["NotificationContextBuilder"]
        SHARED_CONTRIB["Shared Contributors (Dates, Metadata, Recipient Fallbacks)"]
        DOMAIN_CONTRIB["Domain Contributors (Cart, Lodgement, Res, Maint, Workflow)"]
        T_VAL["TemplateValidationService (XSS, URL & Token Guards)"]
        T_DEF["TemplateDefinitionService (3-Tier Cache: ID, Name, Type)"]
        RO["NotificationOrchestrator (Channel Adapter Coordinator)"]
    end

    subgraph ChannelAdapters["4. Channel Adapters & Renderers"]
        EMAIL_ADAPT["EmailNotificationChannelAdapter"]
        SMS_ADAPT["SmsNotificationChannelAdapter"]
        PDF_ADAPT["PdfNotificationChannelAdapter"]
        PREPARED["PreparedNotification (Immutable Context + Rendered Payloads)"]
    end

    subgraph DeliveryCore["5. Delivery Coordination & Safety Layer"]
        NS["NotificationService (Public Delivery Facade)"]
        DO["NotificationDeliveryOrchestrator"]
        REDIS_LOCK[("Redis Distributed Renewable Lease")]
        AUDIT_SINK["JpaNotificationAuditSink (REQUIRES_NEW Tx)"]
        AUDIT_OUTBOX[("NOTIFICATION_DELIVERY_OUTBOX (Latest State)")]
        AUDIT_HIST[("NOTIFICATION_DELIVERY_HISTORY (Audit Log)")]
    end

    subgraph ArtifactPipeline["6. Artifact Processing & Storage"]
        PDF_DELIV["PdfNotificationDelivery (FlyingSaucer + Jsoup)"]
        IMG_CACHE["NotificationImageCache & AssetReplacedElementFactory"]
        CHAIN["ArtifactProcessorChain (@Order Directed Pipeline)"]
        WF_DOC_PROC["WorkflowDocumentArtifactProcessor (Atomic Staging & Target Move)"]
        STORAGE[("Shared File Storage (/workflow-docs)")]
        WF_DOC_TABLE[("WORKFLOW_DOCUMENTS Table")]
    end

    subgraph TransportAdapters["7. Physical Transport Adapters"]
        EMAIL_DELIV["EmailNotificationDelivery (JavaMail / SMTP)"]
        SMS_DELIV["SmsNotificationDelivery (HTTP SMS Gateway)"]
    end

    %% Ingress wirings
    EV --> ENQ --> DB_QUEUE
    SCH --> ENQ
    DB_QUEUE --> Q_WORKER --> GEN_SVC
    GEN_SVC --> WF_FACTORY
    GEN_SVC --> NPS
    WF --> WF_FACTORY --> NPS
    REST --> GW --> NPS
    PREV --> PREV_SVC --> NPS

    %% Prep layer wirings
    NPS --> NCB
    NCB --> SHARED_CONTRIB
    NCB --> DOMAIN_CONTRIB
    SHARED_CONTRIB & DOMAIN_CONTRIB --> T_VAL
    T_VAL <--> T_DEF
    NPS --> RO
    RO --> EMAIL_ADAPT & SMS_ADAPT & PDF_ADAPT
    EMAIL_ADAPT & SMS_ADAPT & PDF_ADAPT --> PREPARED

    %% Delivery core wirings
    PREPARED --> NS
    NS --> DO
    DO <--> REDIS_LOCK
    DO --> AUDIT_SINK
    AUDIT_SINK --> AUDIT_OUTBOX & AUDIT_HIST

    %% Artifact & Transport wirings
    DO -->|Channel: PDF| PDF_DELIV
    PDF_DELIV <--> IMG_CACHE
    PDF_DELIV --> CHAIN --> WF_DOC_PROC
    WF_DOC_PROC --> STORAGE & WF_DOC_TABLE
    WF_DOC_PROC -.->|Enrich Email with Reusable PDF Resource| NS

    DO -->|Channel: EMAIL| EMAIL_DELIV
    DO -->|Channel: SMS| SMS_DELIV

3. Entry Points & Invocation Routes

There are four runtime ingress paths into the framework:

  1. GenericEventEmailNotificationEnqueuer: Writes an email event to NOTIFICATION table within the local business transaction. Its overloads provide a convenience API, not separate delivery paths.
  2. NotificationFrameworkGateway: The synchronous application facade. Exposes operations for email, email-with-attachments, SMS, composite PDF+email, and a generic channel-set operation.
  3. GenericNotificationService: The durable queue worker entry. Dispatches generic email events to GenericEventEmailNotificationHandler and workflow events to ProcessNotificationFactory.
  4. ProcessNotification Implementations: The workflow adapter entry point. Handlers prepare context and invoke NotificationService; NotificationScheduler and NotificationQueueService provide the scheduled polling ingress around them.

The preview path (NotificationPdfPreviewService via ReservationAcknowledgementLetterService) reuses context building and PDF composition in a read-only transaction without acquiring leases, writing audit ledgers, or dispatching external transport I/O.

Open/Closed Review

The extension seams are open for new channels, delivery transports, renderers, context contributors, and artifact processors: implement the corresponding interface and register the Spring bean. Existing registries do not require modification.

There is one deliberate exception: NotificationService has PDF-specific sequencing because a generated PDF is an input to the EMAIL attachment payload. Adding an ordinary channel does not require changing it, but adding another artifact-dependent channel would require extending that coordinator. Treat this as a known design boundary, not as a defect.


4. End-to-End Sequence & Execution Flows

4.1 Complete Delivery Sequence

sequenceDiagram
    autonumber
    actor Caller as Ingress (Queue / Gateway / Workflow)
    participant Prep as NotificationPreparationService
    participant Ctx as Context Contributors
    participant Orch as NotificationOrchestrator
    participant NS as NotificationService
    participant DO as NotificationDeliveryOrchestrator
    participant Oracle as Oracle Outbox & History
    participant Redis as Redis Renewable Lease
    participant PDF as PDF Delivery & Artifact Chain
    participant Trans as Transport (SMTP / SMS Gateway)

    Caller->>Prep: templateId, workflowId, attributes, channels
    Note over Prep: Read-Only Transaction Begins
    Prep->>Ctx: Build shared, recipient and domain attributes
    Ctx-->>Prep: Immutable NotificationContext
    Prep->>Orch: Validate and render each requested channel
    Orch-->>Prep: RenderedEmail / RenderedSms / RenderedPdf
    Prep-->>Caller: PreparedNotification
    Note over Prep: Read-Only Transaction Closes

    Caller->>NS: notify(PreparedNotification)
    Note over NS,Trans: External delivery executes outside database transaction

    opt PDF Channel Requested
        NS->>DO: deliver(PDF payload)
        DO->>Oracle: Look up completed event/channel
        alt Stored PDF Exists
            Oracle-->>DO: Usable Resource and Document ID
            DO->>Oracle: Record skip observation and atomic skip increment
        else No Reusable PDF
            DO->>Redis: Acquire token lease (5 min TTL)
            DO->>Oracle: Commit PENDING (REQUIRES_NEW)
            DO->>PDF: Generate and process artifact chain
            PDF-->>DO: Resource and Document ID
            DO->>Oracle: Commit DELIVERED and append history
            DO->>Redis: Atomic compare-and-delete token
        end
        DO-->>NS: PDF Outcome
        NS->>NS: Enrich RenderedEmail attachment list with PDF Resource
    end

    loop For Remaining Channels (EMAIL, SMS)
        NS->>DO: deliver(Channel payload)
        DO->>Oracle: Look up completed event/channel
        alt Already Delivered
            DO->>Oracle: Record skip observation
        else Not Completed
            DO->>Redis: Acquire token lease (5 min TTL)
            DO->>Oracle: Commit PENDING
            DO->>Trans: Send SMTP / HTTP SMS
            Trans-->>DO: Accepted / OK
            DO->>Oracle: Commit DELIVERED and append history
            DO->>Redis: Atomic compare-and-delete token
        end
        DO-->>NS: Channel Outcome
    end

    NS-->>Caller: Aggregate Per-Channel Outcomes

4.2 Durable Outbox Two-Tier Polling Sequence

sequenceDiagram
    autonumber
    participant Sched as NotificationScheduler
    participant Queue as NotificationQueueService
    participant DB as NOTIFICATION Table
    participant GenSvc as GenericNotificationService
    participant NS as NotificationService

    Sched->>Queue: pollPendingNotifications()
    Queue->>DB: SELECT FOR UPDATE SKIP LOCKED (LIMIT 25, STATUS=0, NEXT_ATTEMPT <= NOW)
    DB-->>Queue: Available Rows
    loop For Each Row
        Queue->>DB: UPDATE NOTIFICATION SET CLAIM_TOKEN=:uuid, CLAIM_UNTIL=:now+5m WHERE ID=:id
        Queue->>GenSvc: handleJdbcNotification(notification)
        GenSvc->>NS: Deliver via Framework
        alt Delivery Succeeded
            GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=1, COMPLETION_DATED=:now WHERE CLAIM_TOKEN=:uuid
        else Delivery Failed
            GenSvc->>DB: Calculate Backoff: min(3600, 2^attempts * 15)
            alt Attempts >= 8
                GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=2 (DEAD_LETTER), NEXT_ATTEMPT=null
            else Attempts < 8
                GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=0, NEXT_ATTEMPT=:backoff, DELIVERY_ATTEMPTS=DELIVERY_ATTEMPTS+1
            end
        end
    end

5. Complete Inventory of Migrated Process Notifications

All 28 active workflow process notifications have been migrated to dedicated NotificationContextContributor implementations and NotificationService execution:

Template ID Enum (ProcessTemplateType) Process Notification Class Dedicated Context Contributor Channels Generated Artifacts / Attachments
13 INFORMATION_REQUEST InformationRequestNotification InformationRequestContextContributor EMAIL None
15 INVOICE InvoiceNotification InvoiceContextContributor EMAIL Invoice PDF
16 PAYMENT_CONFIRMATION PaymentVerificationNotification PaymentVerificationContextContributor EMAIL None
17 DISPATCH DispatchNotification DispatchContextContributor EMAIL None
18 INFORMATION_REJECT InformationRequestRejection InformationRequestRejectionContextContributor EMAIL None
19 QUERY_OUTCOME QueryOutcomeNotification QueryOutcomeContextContributor EMAIL None
20 QUERY QueryNotification QueryNotificationContextContributor EMAIL None
107 NOTIFY_MANAGER NotifyManagerLog NotifyManagerContextContributor EMAIL None
170 AUTO_DISPATCH AutoDispatchNotification AutoDispatchContextContributor EMAIL None
247 NOTIFY_MANAGER_OUTCOME NotifyManagerOutcomeNotification NotifyManagerOutcomeContextContributor EMAIL None
268 PAYMENT_UPLOAD PaymentUploadNotification PaymentUploadContextContributor EMAIL None
307 RES_NEW_REQUEST_NOTIFICATION ReservationNewRequestNotification ReservationNewRequestContextContributor EMAIL Reservation Request Letter PDF
308 RES_REQUEST_REJECTION_LETTER ReservationRequestRejectionNotification ReservationRejectionContextContributor EMAIL Reservation Rejection Letter PDF
327 RES_EXPIRY_NOTIFICATION ReservationExpiryEmailNotification Enqueued Outbox Job EMAIL Expiry Warning Notice
328 RES_ACKNOWLEDGEMENT_LETTER ReservationAcknowledgementLetterNotification ReservationAcknowledgementContextContributor PDF, EMAIL Reservation Acknowledgement Letter PDF
348 RES_NEW_REQUEST_RESUBMISSION ReservationRequestResubmission ReservationResubmissionContextContributor EMAIL Reservation Resubmission Letter PDF
367 LODGEMENT_NEW_REQUEST_NOTIFICATION LodgementNewRequestNotification LodgementNewRequestContextContributor EMAIL Lodgement Request Notice PDF
368 LODGEMENT_LETTER LodgementAcknowledgementNotification LodgementAcknowledgementContextContributor EMAIL Lodgement Acknowledgement Letter PDF
369 LODGEMENT_REQ_REJECTION_LETTER LodgementRequestRejectNotification LodgementRejectionContextContributor EMAIL Lodgement Rejection Letter PDF
370 LODGEMENT_REQ_UPDATE_AMEND LodgementRequestAmendNotification LodgementAmendContextContributor EMAIL Lodgement Amendment Letter PDF
388 RES_TRANSFER_APPROVAL_OUTCOME ReservationTransferApprovalOutcome ReservationTransferApprovalContextContributor EMAIL Reservation Transfer Outcome Letter PDF
487 RELODGEMENT_NEW_REQUEST_NOTIFICATION RelodgementNewRequestNotification RelodgementNewRequestContextContributor EMAIL Relodgement Request Notice PDF
488 RELODGE_LODGEMENT_LETTER RelodgementAcknowledgementNotification RelodgementAcknowledgementContextContributor EMAIL Relodgement Letter PDF
489 RELODGE_REQUEST_AMMEND RelodgeRequestAmmendNotification RelodgeAmendContextContributor EMAIL Relodgement Amendment Letter PDF
490 REQUEST_REJECTION_LETTER RelodgeRequestRejectNotification RelodgeRejectionContextContributor EMAIL Relodgement Rejection Letter PDF
687 MAINTENANCE_NEW_REQUEST_NOTIFICATION MaintenanceNewRequestNotification MaintenanceNewRequestContextContributor EMAIL Maintenance Request Notice
688 MAINTENANCE_OUTCOME_LETTER MaintenanceOutcomeLetterNotification MaintenanceOutcomeContextContributor EMAIL Maintenance Excision Outcome Letter PDF
707 PUBLIC_PLACE_CLOSURE_LETTER MaintenancePPCOutcomeLetterNotification MaintenancePPCOutcomeContextContributor EMAIL Public Place Closure Letter PDF
727 STREET_RENAME_LETTER MaintenanceStreetRenameOutcomeLetterNotification MaintenanceStreetRenameOutcomeContextContributor EMAIL Street Renaming Outcome Letter PDF
748 MAINTENANCE_GENERIC_COMPLETION_LETTER MaintenanceGenericCompletionLetterNotification MaintenanceGenericCompletionContextContributor EMAIL Generic Completion Letter PDF

6. Identity, Recipients and Context Assembly

The scheduler's NOTIFICATION.ID is mapped to the internal, JSON-ignored ProcessNotificationsVm.notificationId. It enters the context and produces channel keys: notification:<id>:<channel>

Two events on the same workflow/template no longer suppress one another. Direct callers without an event ID retain the legacy workflowId:templateId:channel fallback and must supply an event ID for distinct deliveries. A retry uses the same event ID; an intentional resend creates a new notification event.

The context makes a defensive, unmodifiable map copy that permits null optional values: * CommonPdfContextContributor supplies formatted dates and timestamps. * TemplateMetadataContextContributor supplies template names and identifiers. * ArtifactMetadataContextContributor supplies artifactWorkflowId consumed by the workflow artifact processor. * CommonRecipientContextContributor supplies recipient aliases (to, cc, bcc) with delivery fallbacks. * Domain-specific contributors add applicant details, request steps, reference numbers, and custom metadata.

Recipient selection remains the applicant's primary email/mobile. It does not switch automatically to NOTIFICATION.USER_ID or a draft alternate email.


7. Template Engine, Fragment Composition and Caching

NotificationPreparationService encloses context loading and HTML rendering in one read transaction. PdfNotificationChannelAdapter composes the complete HTML there; delivery receives a RenderedPdf marked as composed, avoiding later lazy entity traversal or a second Thymeleaf pass. NotificationService enriches the already rendered email with attachments without re-rendering it. Shared email formatting uses cis-emailer/email.html.

7.1 Dynamic Fragment Composition

PDF composition replaces common header/footer markers with database fragments: * Missing optional fragments remain empty; processing errors propagate. * Styles are collected into the document <head>, then Jsoup performs one XHTML conversion. * Named media use a shared Redis byte cache; only cold/missing names enter a batched repository lookup and disk read. * Image payloads are Base64 encoded for the Redis JSON serializer and decoded into fresh arrays for callers. * LogoService add/update/delete rotates a shared generation after transaction completion, preventing in-flight old reads from repopulating the current namespace. Cache defaults retain the 30-minute TTL. * NotificationAssetReplacedElementFactory consumes the image map and delegates barcode generation to the framework resolver.

flowchart LR
    B[Resolved body snapshot] --> C[Compose only referenced cached fragments]
    C --> T[Thymeleaf processing]
    T --> J[Single Jsoup normalization: styles to head and XHTML output]
    J --> F[Flying Saucer parses XHTML]
    F --> A[Discover assets from the same W3C document]
    A --> P[PDF layout and output]

HtmlTextGenerationService performs no Jsoup parse. TemplateUtils.convertToXhtml owns the sole Jsoup normalization/serialization pass; the replacement factory receives ITextRenderer.getDocument() and performs no HTML string parsing.

PDF file output uses a 64 KiB BufferedOutputStream to coalesce small renderer writes. Parent directories are created before opening the output; the stream is closed before file-size validation, so delayed flush/close failures follow the incomplete-file cleanup path.

7.2 Template Invalidation

TemplateService.saveTemplate persists and invokes TemplateInvalidationService. Invalidation defers until after commit when called inside a surrounding transaction. Direct operator SQL changes require explicit cache eviction or cache expiry.

Name-only legacy PDF fragment lookups use TemplateDefinitionService, through the template-definitions-by-name cache. All three lookup indexes (by-id, by-name, by-item-id-module) are invalidated together. The PDF adapter supplies its already-resolved body to the composer, which fetches only the common fragments whose markers actually occur. A portrait notification never fetches the landscape footer or reloads its body by name.


8. Ownership, Audit, and Recovery

stateDiagram-v2
    [*] --> Lookup
    Lookup --> Recovered: Durable completion and usable PDF if required
    Recovered --> [*]: Preserve result and append skip observation
    Lookup --> Claim: No completed outcome
    Claim --> Deferred: Another worker owns lease
    Deferred --> [*]: Queue retries later
    Claim --> PENDING: Acquire token and recheck completion
    PENDING --> DELIVERED: Transport and processors succeed
    PENDING --> FAILED: Attempts exhausted
    DELIVERED --> Released: Commit ledger and history
    FAILED --> Released: Commit ledger and history
    Released --> [*]: Compare-and-delete token
    Lookup --> BLOCKED: Required PDF unavailable and email not already delivered
    BLOCKED --> Released: Persist blocked outcome under ownership

Redis is a 5-minute renewable worker lease, not the durable success store. UUID tokens are checked during delivery, renewed every minute, and released using atomic Lua compare-and-delete. An expired worker cannot delete a newer worker's token. Lost ownership produces an uncertain failure and prevents that worker from finalizing the channel ledger. Completed outcomes are checked again after ownership acquisition.

JpaNotificationAuditSink commits independently using REQUIRES_NEW: PENDING before I/O, final state after transport and artifact processing. Final audit precedes lease release. The latest-state ledger (NOTIFICATION_DELIVERY_OUTBOX) uses optimistic versioning; skip counts use atomic increments. The same transaction appends a row to NOTIFICATION_DELIVERY_HISTORY, preserving retry and duplicate observations. Blocked dependency handling first recovers an already-delivered email instead of overwriting it. PDF recovery returns its stored path and document ID; a missing file permits regeneration.

SMTP is synchronous for framework notifications and propagates attachment/transport errors. Blank SMS templates are omitted. Nonblank SMS requires SMS_SUCCESS_RESPONSE_PATTERN, a full-match regex from the provider's accepted-response contract. Gateway credentials and message text are not explicitly logged, and RestTemplate debug URL logging is disabled. Gateway error URLs are sanitized and SMS query values are encoded. SMTP connect/read/write timeouts are exposed in environment files. External transport remains at-least-once: a crash after SMTP/provider acceptance and before Oracle outcome commit may produce a duplicate retry.


9. Crash Recovery & Staging Verification

Crash recovery has two independent ownership layers. The durable queue owns the NOTIFICATION row with CLAIM_TOKEN and CLAIM_UNTIL; channel delivery owns a Redis lease keyed by the event and channel. A worker must hold both ownerships while processing. If the JVM dies, the database claim and Redis lease expire independently and the next worker can retry the event. A completed Oracle channel outcome always wins over an expired lease and is reused instead of sending again.

sequenceDiagram
    participant Q as Queue worker
    participant N as NOTIFICATION row
    participant X as Redis lease
    participant O as Oracle delivery ledger
    participant T as External transport
    Q->>N: Claim row with token and expiry
    Q->>X: Acquire event/channel lease
    Q->>O: Record PENDING
    Q->>T: Generate PDF or send email/SMS
    alt Worker completes normally
        T-->>Q: Success
        Q->>O: Record DELIVERED and history
        Q->>X: Compare-and-delete owned lease
        Q->>N: Mark queue row complete
    else Worker crashes before finalization
        Note over N,X: Claim and lease expire independently
        Q-->>N: No completion update
        Q-->>X: No release - TTL expires
        Q->>N: Next poll reclaims row
        Q->>O: Recheck completed outcome
        alt Completed outcome exists
            O-->>Q: Reuse stored result/artifact
        else No completed outcome
            Q->>X: Acquire a new lease
            Q->>O: Retry delivery
        end
    end

PDF-Specific Crash Handling

WorkflowDocumentArtifactProcessor makes PDF persistence retry-safe: 1. Derives a stable NOTIFICATION_KEY for the PDF channel. 2. Reuses the existing WORKFLOW_DOCUMENTS row when the event is retried. 3. Writes a .target checkpoint before replacing the target file. 4. Copies to a .part staging file and atomically moves it into place. 5. After the document transaction commits, the checkpoint is removed. 6. On a retry, an orphan checkpoint is reconciled: a PDF whose document row never committed is removed, while unrelated files are untouched.

Staging Crash Test Matrix

Scenario Inject Failure Expected Recovery
Before PDF delivery Kill worker after queue claim Claim expires; same event is retried and completes
During PDF generation Kill process before final audit Staging/checkpoint files are reconciled; no duplicate document created
After workflow-document commit Kill before channel audit Existing document is reused by notification key
After SMTP acceptance Kill before final audit Retry may send a duplicate; history records both observations
Redis lease loss Stop/block Redis during delivery Worker cannot finalize; next worker retries after lease expiry
JVM restart during batch Stop application while several rows run Claimed rows become eligible again; no permanent stuck claim

Staging Verification Queries

SELECT ID, STATUS, DELIVERY_ATTEMPTS, CLAIM_TOKEN, CLAIM_UNTIL, NEXT_ATTEMPT
FROM NOTIFICATION
WHERE ID = :notification_id;
SELECT IDENTITY_KEY, STATUS, ATTEMPTS, ERROR_MESSAGE, RECORDED_AT
FROM NOTIFICATION_DELIVERY_HISTORY
WHERE IDENTITY_KEY LIKE '%' || :notification_id || '%'
ORDER BY ID;
SELECT IDEMPOTENCY_KEY, STATUS, ATTEMPTS, ARTIFACT_PATH, DOCUMENT_ID
FROM NOTIFICATION_DELIVERY_OUTBOX
WHERE IDEMPOTENCY_KEY LIKE '%' || :notification_id || '%';

Also inspect the workflow storage directory for leftover .part files, .target checkpoint files, and duplicate {documentId}.pdf files after each injected failure.

Synchronous Gateway Boundary

Queue-backed workflow and generic-event notifications recover automatically. A direct NotificationFrameworkGateway call has no queue row to resume if the JVM dies during delivery; the caller must retry it, or the operation must first enqueue a durable notification event.


10. Queue Retries and Operational Recovery

Email PDF attachment names use the template's configured file prefix (or template name), followed by the reference number: Reservation-acknowledgement-RES0023KZN.pdf. Missing references fall back to the workflow/event/template identifier. Names are sanitized and bounded. A resource decorator changes only the MIME filename; workflow storage remains {documentId}.pdf, so document links and recovered artifacts retain their existing paths.

NotificationQueueService selects at most 25 eligible IDs, atomically claims each row with a UUID and expiry, renews claims, and updates outcomes only when its token still matches. Polls use fixed delay; expired scheduler locks do not bypass row ownership. Failed rows are delayed by:

\[\text{delaySeconds} = \min(3600, 2^{\text{attempts}} \times 15)\]

The eighth failed attempt moves the row to STATUS = 2 (Dead Letter Queue). Status 0 is pending; status 1 is complete; status 2 is dead letter.

Operational Dead-Letter Runbook

After correcting the root failure, an operator can reset a dead-letter row's STATUS to 0, DELIVERY_ATTEMPTS to 0, and NEXT_ATTEMPT to null, only when it has no active claim:

UPDATE NOTIFICATION
SET STATUS = 0,
    DELIVERY_ATTEMPTS = 0,
    NEXT_ATTEMPT = SYSTIMESTAMP,
    CLAIM_TOKEN = NULL,
    CLAIM_UNTIL = NULL
WHERE ID = :notification_id
  AND STATUS = 2
  AND (CLAIM_UNTIL IS NULL OR CLAIM_UNTIL < SYSTIMESTAMP);
COMMIT;

Keep its event ID to resume unfinished channels. Do not delete confirmed delivery rows or flush Redis to request a new message; create a new notification event for an intentional resend.

Database Concurrency Safeguards

Virtual threads are enabled for the application, but database-bound event handling is bounded. AsynchronousSpringEventsConfig uses a dedicated bounded ThreadPoolTaskExecutor with configurable core/max/queue values and CallerRunsPolicy; it does not use an unbounded SimpleAsyncTaskExecutor. The default event limits are 1 core thread, 3 maximum threads, and a queue of 100. These limits must remain below the combined Oracle connection capacity reserved for the service. Hikari connection wait timeouts are environment-configurable; the development profile uses the same 30-second default as UAT and production rather than an excessive override.


11. Consolidated Schema and Environment Rollout

All notification DDL is consolidated in V3__notification_delivery_outbox.sql at the user's explicit request to reset earlier interim Flyway versions. V4/V5 source files were removed. The script creates/extends the ledger, history, workflow document identity, and queue ownership/retry fields. Existing rows and sequence values are preserved on reapplication.

This changes the checksum of previously applied V3. Stop notification workers and complete the explicitly agreed notification-history reset before normal Flyway startup; if V4/V5 were applied from an interim checkout, their history must be reconciled as well. Removing history is an operator action, not performed by this implementation. Do not change unrelated migration history or drop notification data merely to reapply the consolidated script.

Existing legacy workflow/template ledger keys cannot be mapped reliably to event IDs. Before requeuing pre-upgrade notifications, reconcile their already-sent outcomes to avoid resending messages after the event-key transition. Configure the provider acknowledgement regex before enabling framework SMS and ensure all nodes share workflow storage.


12. Verification & Post-Implementation Review

The regression suite covers recovered PDFs, failed artifact processing, unresolved ownership, distinct event identity, immutable optional null context, preserving delivered email on a blocked dependency, final audit before release, provider acknowledgement rejection, audit skip preservation, missing artifact recovery, channel selection, and template lookup.

The combined notification/event regression run passed 125 focused tests with zero failures and zero skips. Tests cover: * Queue success/backoff/contention. * Prepared attachment enrichment without re-rendering. * Event-document reuse and crash-file reconciliation without removing unrelated files. * Generic event listener transactions and in-app side effects. * Resource routing and search-share guards. * Lodgement and reservation action routing. * Expiry reminders and task reminders.

Live isolated checks also verified Redis stale-token rejection/renewal/release and Oracle consolidated V3 reapplication with delivery-row preservation. The disposable Oracle schema was removed. SGUSER schema/history was not changed, and no external email or SMS was dispatched during tests.


13. Known Bypasses and Stale Boundaries

The legacy EmailServiceHandler and SmsServiceHandler call sites in cis-api have been removed from the migrated notification paths. One workflow-module hook remains outside this module's framework boundary: WorkflowProcessingService calls emailHandlerService.triggerTaskEmail(...). It is intentionally not counted as a CIS API framework entry point and requires a cis-workflow change to migrate.

PDF generation is not globally unified yet. Non-notification reports and previews still call PdfGeneratorService directly (such as examination dockets and administrative reports), while notification PDF delivery uses the channel adapter and generic delivery orchestrator. The reservation acknowledgement preview now uses NotificationPdfPreviewService, which reuses framework context/template composition without creating a durable delivery. The remaining report and examination paths are valid document-generation paths today, but they do not receive queue, idempotency, durable delivery audit, or workflow-artifact semantics. Do not describe the framework as owning every PDF in the application until those contracts are deliberately migrated.

Within the framework itself, PdfNotificationDelivery remains the concrete PdfDocumentDelivery port implementation and PdfNotificationDeliveryAdapter is its generic-delivery bridge; neither is dead code.


14. Extending the Framework

Follow these six canonical rules when extending the notification system:

  1. Add an ordered domain context contributor and populate the live template variables.
  2. Carry the persisted notification ID into NotificationPreparationService.prepare.
  3. Pass the prepared payload to NotificationService; select the supported channels.
  4. Propagate failed results so the durable queue can back off or dead-letter the event.
  5. Add ordered artifact processors only for extra artifact responsibilities.
  6. Test event reuse, distinct events, missing optional data, transport failure, and artifact recovery.

[!CAUTION] Anti-Pattern Warning: Do NOT duplicate PDF generation, email wrapping, attachment sequencing, leases, or audit logic inside domain notification classes. The obsolete NotificationChannelRenderer and logging-only sink were removed; the adapter registry and persistent sink are the sole maintained paths.

Developer Recipes & Code Examples

Recipe A: Implementing a Domain Context Contributor

package com.dw.ngms.cis.service.notification.context.custom;

import com.dw.ngms.cis.service.notification.context.NotificationContextContributor;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import java.util.Map;

@Component
@Order(500)
public class CustomProcessContextContributor implements NotificationContextContributor {

    @Override
    public boolean supports(Long templateId, Long workflowId, Map<String, ?> attributes) {
        return Long.valueOf(999L).equals(templateId);
    }

    @Override
    public void contribute(Long templateId, Long workflowId, Map<String, ?> attributes, Map<String, Object> target) {
        target.put("referenceNumber", attributes.get("referenceNo"));
        target.put("statusDescription", "Approved");
        // Add domain-specific template variables here
    }
}

Recipe B: Triggering via Synchronous Gateway

@Service
@AllArgsConstructor
public class CustomAlertService {

    private final NotificationFrameworkGateway notificationGateway;

    public void dispatchAlert(Long workflowId, String userEmail, String referenceNumber) {
        Map<String, Object> attributes = Map.of(
            "referenceNo", referenceNumber,
            "notificationId", UUID.randomUUID().toString()
        );
        // Automatically validates, prepares, acquires Redis lease, audits, and delivers
        notificationGateway.sendEmail(999L, workflowId, userEmail, null, null, attributes);
    }
}

Recipe C: Triggering via Asynchronous Durable Outbox

@Component
@AllArgsConstructor
public class CustomEventListener {

    private final GenericEventEmailNotificationEnqueuer genericEmailEnqueuer;

    @EventListener(condition = "#event.eventType.name() == 'CUSTOM_EVENT'")
    public void onCustomEvent(GenericNotificationEvent<User> event) {
        User user = event.getContent();
        // Enqueues into NOTIFICATION table within the current database transaction
        genericEmailEnqueuer.enqueue(
            event.getEventType(),
            999L,
            user.getUserId(),
            user.getEmail(),
            null,
            Map.of("firstname", user.getFirstName(), "surname", user.getSurname())
        );
    }
}

Recipe D: Adding a Custom Artifact Processor

@Component
@Order(200) // Runs after standard workflow document processor
public class CustomArchivalArtifactProcessor implements ArtifactProcessor {

    @Override
    public boolean supports(ArtifactContext context) {
        return context.getChannel() == NotificationChannel.PDF;
    }

    @Override
    public Resource process(Resource inputArtifact, ArtifactContext context) throws Exception {
        // Perform additional operations (e.g. upload to external secure storage or add cryptographic watermark)
        return inputArtifact;
    }
}