CIS Notification Framework — Canonical Architecture & Developer Guide¶
[!IMPORTANT] This document is the authoritative specification and developer reference ("The Bible") for the CIS Notification Framework. It governs all notification channels (EMAIL, SMS, PDF), asynchronous event outbox queues, workflow process notifications, template resolution and caching, document artifact generation, distributed idempotency, audit ledgers, and developer extension rules across
cis-api.
1. Scope, Responsibilities and Package Ownership¶
Package Ownership¶
service.notification
NotificationService public facade
NotificationFrameworkGateway synchronous application gateway facade
model/ shared channel, request and failure-policy types
context/ context builder and shared/domain contributors
cart/, lodgement/, domain-specific context contributors
maintenance/, payment/,
reservation/, workflow/
render/ preparation, adapter registry and render results
template/ validation, cache invalidation and token checking
channel/
pdf/ PDF payload, rendering, preview and delivery adapter
email/ email payload, delivery and attachment naming
sms/ SMS payload and delivery
delivery/ generic retries, artifact handling and delivery results
artifact/ artifact processor contract and workflow document storage
asset/ image/barcode resolution, replacement factory and cache
audit/ durable delivery observations and JPA sink
idempotency/ event identity and renewable Redis ownership
queue/ durable queue worker, outbox handler and workflow notification dispatch
Framework tests mirror these responsibility packages. Cross-cutting safety tests remain beside the public facade; template composition tests remain in service.template. Packages do not add new Spring configuration: existing component scanning covers the subpackages, and the original class names, cache names, and persistence contracts are retained.
The framework composes database-managed templates and delivers PDF, EMAIL, and SMS through a shared strategy registry. All 28 active workflow notifications, the generic event listener, public email/SMS REST resources, lodgement/reservation action notifications, search sharing, expiry reminders, and task reminders enter this framework through either the durable queue (GenericEventEmailNotificationEnqueuer) or NotificationFrameworkGateway.
Core Hardening Invariants¶
- Fail-Fast Registries: The registries fail fast if two adapters or delivery strategies claim the same channel, preventing bean ordering from silently altering runtime behavior.
- Strict Boundary Validation: Render and delivery boundaries reject null commands, null contexts, empty channel selections, invalid retry counts, and rendered results whose map key does not match their declared channel.
- Payload Immutability: Prepared contexts and channel payloads remain strictly immutable once constructed.
- Identity Fallbacks: Legacy callers may omit template/workflow IDs when they provide an explicit notification event ID (
notification:<id>:<channel>); direct callers without an event ID retain the fallbackworkflowId:templateId:channel. - Deterministic Channel Sequencing:
NotificationServicesequences dependent channels;NotificationDeliveryOrchestratorowns channel lookup, ownership, retries, artifact processing, recovery, and audit. PDF delivery runs before email only when PDF is requested because the generated resource is an input to the email attachment. Domain contributors populate template variables, while transport strategies perform physical I/O.
2. Global Architecture & Component Topology¶
flowchart TD
subgraph Ingress["1. Ingress Layer (Trigger Sources)"]
WF["Workflow Engine (ProcessNotificationsVm)"]
EV["GenericEventListener (Domain Lifecycle Events)"]
SCH["Schedulers (TaskReminderService, Expiry Reminders)"]
REST["REST API Endpoints (EmailResource, SmsResource)"]
PREV["Interactive Letter Preview (Letter Previews)"]
end
subgraph EntryPorts["2. Ingress Gateways & Outbox"]
GW["NotificationFrameworkGateway (Sync Facade)"]
ENQ["GenericEventEmailNotificationEnqueuer (Outbox Enqueuer)"]
PREV_SVC["NotificationPdfPreviewService (Preview Engine)"]
DB_QUEUE[("NOTIFICATION Table (Outbox Queue)")]
Q_WORKER["NotificationQueueService (Claim & Poller)"]
GEN_SVC["GenericNotificationService (Dispatcher)"]
WF_FACTORY["ProcessNotificationFactory (28 Active Mappings)"]
end
subgraph PrepLayer["3. Preparation & Context Assembly Pipeline"]
NPS["NotificationPreparationService (Read-Only Tx)"]
NCB["NotificationContextBuilder"]
SHARED_CONTRIB["Shared Contributors (Dates, Metadata, Recipient Fallbacks)"]
DOMAIN_CONTRIB["Domain Contributors (Cart, Lodgement, Res, Maint, Workflow)"]
T_VAL["TemplateValidationService (XSS, URL & Token Guards)"]
T_DEF["TemplateDefinitionService (3-Tier Cache: ID, Name, Type)"]
RO["NotificationOrchestrator (Channel Adapter Coordinator)"]
end
subgraph ChannelAdapters["4. Channel Adapters & Renderers"]
EMAIL_ADAPT["EmailNotificationChannelAdapter"]
SMS_ADAPT["SmsNotificationChannelAdapter"]
PDF_ADAPT["PdfNotificationChannelAdapter"]
PREPARED["PreparedNotification (Immutable Context + Rendered Payloads)"]
end
subgraph DeliveryCore["5. Delivery Coordination & Safety Layer"]
NS["NotificationService (Public Delivery Facade)"]
DO["NotificationDeliveryOrchestrator"]
REDIS_LOCK[("Redis Distributed Renewable Lease")]
AUDIT_SINK["JpaNotificationAuditSink (REQUIRES_NEW Tx)"]
AUDIT_OUTBOX[("NOTIFICATION_DELIVERY_OUTBOX (Latest State)")]
AUDIT_HIST[("NOTIFICATION_DELIVERY_HISTORY (Audit Log)")]
end
subgraph ArtifactPipeline["6. Artifact Processing & Storage"]
PDF_DELIV["PdfNotificationDelivery (FlyingSaucer + Jsoup)"]
IMG_CACHE["NotificationImageCache & AssetReplacedElementFactory"]
CHAIN["ArtifactProcessorChain (@Order Directed Pipeline)"]
WF_DOC_PROC["WorkflowDocumentArtifactProcessor (Atomic Staging & Target Move)"]
STORAGE[("Shared File Storage (/workflow-docs)")]
WF_DOC_TABLE[("WORKFLOW_DOCUMENTS Table")]
end
subgraph TransportAdapters["7. Physical Transport Adapters"]
EMAIL_DELIV["EmailNotificationDelivery (JavaMail / SMTP)"]
SMS_DELIV["SmsNotificationDelivery (HTTP SMS Gateway)"]
end
%% Ingress wirings
EV --> ENQ --> DB_QUEUE
SCH --> ENQ
DB_QUEUE --> Q_WORKER --> GEN_SVC
GEN_SVC --> WF_FACTORY
GEN_SVC --> NPS
WF --> WF_FACTORY --> NPS
REST --> GW --> NPS
PREV --> PREV_SVC --> NPS
%% Prep layer wirings
NPS --> NCB
NCB --> SHARED_CONTRIB
NCB --> DOMAIN_CONTRIB
SHARED_CONTRIB & DOMAIN_CONTRIB --> T_VAL
T_VAL <--> T_DEF
NPS --> RO
RO --> EMAIL_ADAPT & SMS_ADAPT & PDF_ADAPT
EMAIL_ADAPT & SMS_ADAPT & PDF_ADAPT --> PREPARED
%% Delivery core wirings
PREPARED --> NS
NS --> DO
DO <--> REDIS_LOCK
DO --> AUDIT_SINK
AUDIT_SINK --> AUDIT_OUTBOX & AUDIT_HIST
%% Artifact & Transport wirings
DO -->|Channel: PDF| PDF_DELIV
PDF_DELIV <--> IMG_CACHE
PDF_DELIV --> CHAIN --> WF_DOC_PROC
WF_DOC_PROC --> STORAGE & WF_DOC_TABLE
WF_DOC_PROC -.->|Enrich Email with Reusable PDF Resource| NS
DO -->|Channel: EMAIL| EMAIL_DELIV
DO -->|Channel: SMS| SMS_DELIV
3. Entry Points & Invocation Routes¶
There are four runtime ingress paths into the framework:
GenericEventEmailNotificationEnqueuer: Writes an email event toNOTIFICATIONtable within the local business transaction. Its overloads provide a convenience API, not separate delivery paths.NotificationFrameworkGateway: The synchronous application facade. Exposes operations for email, email-with-attachments, SMS, composite PDF+email, and a generic channel-set operation.GenericNotificationService: The durable queue worker entry. Dispatches generic email events toGenericEventEmailNotificationHandlerand workflow events toProcessNotificationFactory.ProcessNotificationImplementations: The workflow adapter entry point. Handlers prepare context and invokeNotificationService;NotificationSchedulerandNotificationQueueServiceprovide the scheduled polling ingress around them.
The preview path (NotificationPdfPreviewService via ReservationAcknowledgementLetterService) reuses context building and PDF composition in a read-only transaction without acquiring leases, writing audit ledgers, or dispatching external transport I/O.
Open/Closed Review¶
The extension seams are open for new channels, delivery transports, renderers, context contributors, and artifact processors: implement the corresponding interface and register the Spring bean. Existing registries do not require modification.
There is one deliberate exception: NotificationService has PDF-specific sequencing because a generated PDF is an input to the EMAIL attachment payload. Adding an ordinary channel does not require changing it, but adding another artifact-dependent channel would require extending that coordinator. Treat this as a known design boundary, not as a defect.
4. End-to-End Sequence & Execution Flows¶
4.1 Complete Delivery Sequence¶
sequenceDiagram
autonumber
actor Caller as Ingress (Queue / Gateway / Workflow)
participant Prep as NotificationPreparationService
participant Ctx as Context Contributors
participant Orch as NotificationOrchestrator
participant NS as NotificationService
participant DO as NotificationDeliveryOrchestrator
participant Oracle as Oracle Outbox & History
participant Redis as Redis Renewable Lease
participant PDF as PDF Delivery & Artifact Chain
participant Trans as Transport (SMTP / SMS Gateway)
Caller->>Prep: templateId, workflowId, attributes, channels
Note over Prep: Read-Only Transaction Begins
Prep->>Ctx: Build shared, recipient and domain attributes
Ctx-->>Prep: Immutable NotificationContext
Prep->>Orch: Validate and render each requested channel
Orch-->>Prep: RenderedEmail / RenderedSms / RenderedPdf
Prep-->>Caller: PreparedNotification
Note over Prep: Read-Only Transaction Closes
Caller->>NS: notify(PreparedNotification)
Note over NS,Trans: External delivery executes outside database transaction
opt PDF Channel Requested
NS->>DO: deliver(PDF payload)
DO->>Oracle: Look up completed event/channel
alt Stored PDF Exists
Oracle-->>DO: Usable Resource and Document ID
DO->>Oracle: Record skip observation and atomic skip increment
else No Reusable PDF
DO->>Redis: Acquire token lease (5 min TTL)
DO->>Oracle: Commit PENDING (REQUIRES_NEW)
DO->>PDF: Generate and process artifact chain
PDF-->>DO: Resource and Document ID
DO->>Oracle: Commit DELIVERED and append history
DO->>Redis: Atomic compare-and-delete token
end
DO-->>NS: PDF Outcome
NS->>NS: Enrich RenderedEmail attachment list with PDF Resource
end
loop For Remaining Channels (EMAIL, SMS)
NS->>DO: deliver(Channel payload)
DO->>Oracle: Look up completed event/channel
alt Already Delivered
DO->>Oracle: Record skip observation
else Not Completed
DO->>Redis: Acquire token lease (5 min TTL)
DO->>Oracle: Commit PENDING
DO->>Trans: Send SMTP / HTTP SMS
Trans-->>DO: Accepted / OK
DO->>Oracle: Commit DELIVERED and append history
DO->>Redis: Atomic compare-and-delete token
end
DO-->>NS: Channel Outcome
end
NS-->>Caller: Aggregate Per-Channel Outcomes
4.2 Durable Outbox Two-Tier Polling Sequence¶
sequenceDiagram
autonumber
participant Sched as NotificationScheduler
participant Queue as NotificationQueueService
participant DB as NOTIFICATION Table
participant GenSvc as GenericNotificationService
participant NS as NotificationService
Sched->>Queue: pollPendingNotifications()
Queue->>DB: SELECT FOR UPDATE SKIP LOCKED (LIMIT 25, STATUS=0, NEXT_ATTEMPT <= NOW)
DB-->>Queue: Available Rows
loop For Each Row
Queue->>DB: UPDATE NOTIFICATION SET CLAIM_TOKEN=:uuid, CLAIM_UNTIL=:now+5m WHERE ID=:id
Queue->>GenSvc: handleJdbcNotification(notification)
GenSvc->>NS: Deliver via Framework
alt Delivery Succeeded
GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=1, COMPLETION_DATED=:now WHERE CLAIM_TOKEN=:uuid
else Delivery Failed
GenSvc->>DB: Calculate Backoff: min(3600, 2^attempts * 15)
alt Attempts >= 8
GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=2 (DEAD_LETTER), NEXT_ATTEMPT=null
else Attempts < 8
GenSvc->>DB: UPDATE NOTIFICATION SET STATUS=0, NEXT_ATTEMPT=:backoff, DELIVERY_ATTEMPTS=DELIVERY_ATTEMPTS+1
end
end
end
5. Complete Inventory of Migrated Process Notifications¶
All 28 active workflow process notifications have been migrated to dedicated NotificationContextContributor implementations and NotificationService execution:
| Template ID | Enum (ProcessTemplateType) |
Process Notification Class | Dedicated Context Contributor | Channels | Generated Artifacts / Attachments |
|---|---|---|---|---|---|
| 13 | INFORMATION_REQUEST |
InformationRequestNotification |
InformationRequestContextContributor |
None | |
| 15 | INVOICE |
InvoiceNotification |
InvoiceContextContributor |
Invoice PDF | |
| 16 | PAYMENT_CONFIRMATION |
PaymentVerificationNotification |
PaymentVerificationContextContributor |
None | |
| 17 | DISPATCH |
DispatchNotification |
DispatchContextContributor |
None | |
| 18 | INFORMATION_REJECT |
InformationRequestRejection |
InformationRequestRejectionContextContributor |
None | |
| 19 | QUERY_OUTCOME |
QueryOutcomeNotification |
QueryOutcomeContextContributor |
None | |
| 20 | QUERY |
QueryNotification |
QueryNotificationContextContributor |
None | |
| 107 | NOTIFY_MANAGER |
NotifyManagerLog |
NotifyManagerContextContributor |
None | |
| 170 | AUTO_DISPATCH |
AutoDispatchNotification |
AutoDispatchContextContributor |
None | |
| 247 | NOTIFY_MANAGER_OUTCOME |
NotifyManagerOutcomeNotification |
NotifyManagerOutcomeContextContributor |
None | |
| 268 | PAYMENT_UPLOAD |
PaymentUploadNotification |
PaymentUploadContextContributor |
None | |
| 307 | RES_NEW_REQUEST_NOTIFICATION |
ReservationNewRequestNotification |
ReservationNewRequestContextContributor |
Reservation Request Letter PDF | |
| 308 | RES_REQUEST_REJECTION_LETTER |
ReservationRequestRejectionNotification |
ReservationRejectionContextContributor |
Reservation Rejection Letter PDF | |
| 327 | RES_EXPIRY_NOTIFICATION |
ReservationExpiryEmailNotification |
Enqueued Outbox Job | Expiry Warning Notice | |
| 328 | RES_ACKNOWLEDGEMENT_LETTER |
ReservationAcknowledgementLetterNotification |
ReservationAcknowledgementContextContributor |
PDF, EMAIL | Reservation Acknowledgement Letter PDF |
| 348 | RES_NEW_REQUEST_RESUBMISSION |
ReservationRequestResubmission |
ReservationResubmissionContextContributor |
Reservation Resubmission Letter PDF | |
| 367 | LODGEMENT_NEW_REQUEST_NOTIFICATION |
LodgementNewRequestNotification |
LodgementNewRequestContextContributor |
Lodgement Request Notice PDF | |
| 368 | LODGEMENT_LETTER |
LodgementAcknowledgementNotification |
LodgementAcknowledgementContextContributor |
Lodgement Acknowledgement Letter PDF | |
| 369 | LODGEMENT_REQ_REJECTION_LETTER |
LodgementRequestRejectNotification |
LodgementRejectionContextContributor |
Lodgement Rejection Letter PDF | |
| 370 | LODGEMENT_REQ_UPDATE_AMEND |
LodgementRequestAmendNotification |
LodgementAmendContextContributor |
Lodgement Amendment Letter PDF | |
| 388 | RES_TRANSFER_APPROVAL_OUTCOME |
ReservationTransferApprovalOutcome |
ReservationTransferApprovalContextContributor |
Reservation Transfer Outcome Letter PDF | |
| 487 | RELODGEMENT_NEW_REQUEST_NOTIFICATION |
RelodgementNewRequestNotification |
RelodgementNewRequestContextContributor |
Relodgement Request Notice PDF | |
| 488 | RELODGE_LODGEMENT_LETTER |
RelodgementAcknowledgementNotification |
RelodgementAcknowledgementContextContributor |
Relodgement Letter PDF | |
| 489 | RELODGE_REQUEST_AMMEND |
RelodgeRequestAmmendNotification |
RelodgeAmendContextContributor |
Relodgement Amendment Letter PDF | |
| 490 | REQUEST_REJECTION_LETTER |
RelodgeRequestRejectNotification |
RelodgeRejectionContextContributor |
Relodgement Rejection Letter PDF | |
| 687 | MAINTENANCE_NEW_REQUEST_NOTIFICATION |
MaintenanceNewRequestNotification |
MaintenanceNewRequestContextContributor |
Maintenance Request Notice | |
| 688 | MAINTENANCE_OUTCOME_LETTER |
MaintenanceOutcomeLetterNotification |
MaintenanceOutcomeContextContributor |
Maintenance Excision Outcome Letter PDF | |
| 707 | PUBLIC_PLACE_CLOSURE_LETTER |
MaintenancePPCOutcomeLetterNotification |
MaintenancePPCOutcomeContextContributor |
Public Place Closure Letter PDF | |
| 727 | STREET_RENAME_LETTER |
MaintenanceStreetRenameOutcomeLetterNotification |
MaintenanceStreetRenameOutcomeContextContributor |
Street Renaming Outcome Letter PDF | |
| 748 | MAINTENANCE_GENERIC_COMPLETION_LETTER |
MaintenanceGenericCompletionLetterNotification |
MaintenanceGenericCompletionContextContributor |
Generic Completion Letter PDF |
6. Identity, Recipients and Context Assembly¶
The scheduler's NOTIFICATION.ID is mapped to the internal, JSON-ignored ProcessNotificationsVm.notificationId. It enters the context and produces channel keys:
notification:<id>:<channel>
Two events on the same workflow/template no longer suppress one another. Direct callers without an event ID retain the legacy workflowId:templateId:channel fallback and must supply an event ID for distinct deliveries. A retry uses the same event ID; an intentional resend creates a new notification event.
The context makes a defensive, unmodifiable map copy that permits null optional values:
* CommonPdfContextContributor supplies formatted dates and timestamps.
* TemplateMetadataContextContributor supplies template names and identifiers.
* ArtifactMetadataContextContributor supplies artifactWorkflowId consumed by the workflow artifact processor.
* CommonRecipientContextContributor supplies recipient aliases (to, cc, bcc) with delivery fallbacks.
* Domain-specific contributors add applicant details, request steps, reference numbers, and custom metadata.
Recipient selection remains the applicant's primary email/mobile. It does not switch automatically to NOTIFICATION.USER_ID or a draft alternate email.
7. Template Engine, Fragment Composition and Caching¶
NotificationPreparationService encloses context loading and HTML rendering in one read transaction. PdfNotificationChannelAdapter composes the complete HTML there; delivery receives a RenderedPdf marked as composed, avoiding later lazy entity traversal or a second Thymeleaf pass. NotificationService enriches the already rendered email with attachments without re-rendering it. Shared email formatting uses cis-emailer/email.html.
7.1 Dynamic Fragment Composition¶
PDF composition replaces common header/footer markers with database fragments:
* Missing optional fragments remain empty; processing errors propagate.
* Styles are collected into the document <head>, then Jsoup performs one XHTML conversion.
* Named media use a shared Redis byte cache; only cold/missing names enter a batched repository lookup and disk read.
* Image payloads are Base64 encoded for the Redis JSON serializer and decoded into fresh arrays for callers.
* LogoService add/update/delete rotates a shared generation after transaction completion, preventing in-flight old reads from repopulating the current namespace. Cache defaults retain the 30-minute TTL.
* NotificationAssetReplacedElementFactory consumes the image map and delegates barcode generation to the framework resolver.
flowchart LR
B[Resolved body snapshot] --> C[Compose only referenced cached fragments]
C --> T[Thymeleaf processing]
T --> J[Single Jsoup normalization: styles to head and XHTML output]
J --> F[Flying Saucer parses XHTML]
F --> A[Discover assets from the same W3C document]
A --> P[PDF layout and output]
HtmlTextGenerationService performs no Jsoup parse. TemplateUtils.convertToXhtml owns the sole Jsoup normalization/serialization pass; the replacement factory receives ITextRenderer.getDocument() and performs no HTML string parsing.
PDF file output uses a 64 KiB BufferedOutputStream to coalesce small renderer writes. Parent directories are created before opening the output; the stream is closed before file-size validation, so delayed flush/close failures follow the incomplete-file cleanup path.
7.2 Template Invalidation¶
TemplateService.saveTemplate persists and invokes TemplateInvalidationService. Invalidation defers until after commit when called inside a surrounding transaction. Direct operator SQL changes require explicit cache eviction or cache expiry.
Name-only legacy PDF fragment lookups use TemplateDefinitionService, through the template-definitions-by-name cache. All three lookup indexes (by-id, by-name, by-item-id-module) are invalidated together. The PDF adapter supplies its already-resolved body to the composer, which fetches only the common fragments whose markers actually occur. A portrait notification never fetches the landscape footer or reloads its body by name.
8. Ownership, Audit, and Recovery¶
stateDiagram-v2
[*] --> Lookup
Lookup --> Recovered: Durable completion and usable PDF if required
Recovered --> [*]: Preserve result and append skip observation
Lookup --> Claim: No completed outcome
Claim --> Deferred: Another worker owns lease
Deferred --> [*]: Queue retries later
Claim --> PENDING: Acquire token and recheck completion
PENDING --> DELIVERED: Transport and processors succeed
PENDING --> FAILED: Attempts exhausted
DELIVERED --> Released: Commit ledger and history
FAILED --> Released: Commit ledger and history
Released --> [*]: Compare-and-delete token
Lookup --> BLOCKED: Required PDF unavailable and email not already delivered
BLOCKED --> Released: Persist blocked outcome under ownership
Redis is a 5-minute renewable worker lease, not the durable success store. UUID tokens are checked during delivery, renewed every minute, and released using atomic Lua compare-and-delete. An expired worker cannot delete a newer worker's token. Lost ownership produces an uncertain failure and prevents that worker from finalizing the channel ledger. Completed outcomes are checked again after ownership acquisition.
JpaNotificationAuditSink commits independently using REQUIRES_NEW: PENDING before I/O, final state after transport and artifact processing. Final audit precedes lease release. The latest-state ledger (NOTIFICATION_DELIVERY_OUTBOX) uses optimistic versioning; skip counts use atomic increments. The same transaction appends a row to NOTIFICATION_DELIVERY_HISTORY, preserving retry and duplicate observations. Blocked dependency handling first recovers an already-delivered email instead of overwriting it. PDF recovery returns its stored path and document ID; a missing file permits regeneration.
SMTP is synchronous for framework notifications and propagates attachment/transport errors. Blank SMS templates are omitted. Nonblank SMS requires SMS_SUCCESS_RESPONSE_PATTERN, a full-match regex from the provider's accepted-response contract. Gateway credentials and message text are not explicitly logged, and RestTemplate debug URL logging is disabled. Gateway error URLs are sanitized and SMS query values are encoded. SMTP connect/read/write timeouts are exposed in environment files. External transport remains at-least-once: a crash after SMTP/provider acceptance and before Oracle outcome commit may produce a duplicate retry.
9. Crash Recovery & Staging Verification¶
Crash recovery has two independent ownership layers. The durable queue owns the NOTIFICATION row with CLAIM_TOKEN and CLAIM_UNTIL; channel delivery owns a Redis lease keyed by the event and channel. A worker must hold both ownerships while processing. If the JVM dies, the database claim and Redis lease expire independently and the next worker can retry the event. A completed Oracle channel outcome always wins over an expired lease and is reused instead of sending again.
sequenceDiagram
participant Q as Queue worker
participant N as NOTIFICATION row
participant X as Redis lease
participant O as Oracle delivery ledger
participant T as External transport
Q->>N: Claim row with token and expiry
Q->>X: Acquire event/channel lease
Q->>O: Record PENDING
Q->>T: Generate PDF or send email/SMS
alt Worker completes normally
T-->>Q: Success
Q->>O: Record DELIVERED and history
Q->>X: Compare-and-delete owned lease
Q->>N: Mark queue row complete
else Worker crashes before finalization
Note over N,X: Claim and lease expire independently
Q-->>N: No completion update
Q-->>X: No release - TTL expires
Q->>N: Next poll reclaims row
Q->>O: Recheck completed outcome
alt Completed outcome exists
O-->>Q: Reuse stored result/artifact
else No completed outcome
Q->>X: Acquire a new lease
Q->>O: Retry delivery
end
end
PDF-Specific Crash Handling¶
WorkflowDocumentArtifactProcessor makes PDF persistence retry-safe:
1. Derives a stable NOTIFICATION_KEY for the PDF channel.
2. Reuses the existing WORKFLOW_DOCUMENTS row when the event is retried.
3. Writes a .target checkpoint before replacing the target file.
4. Copies to a .part staging file and atomically moves it into place.
5. After the document transaction commits, the checkpoint is removed.
6. On a retry, an orphan checkpoint is reconciled: a PDF whose document row never committed is removed, while unrelated files are untouched.
Staging Crash Test Matrix¶
| Scenario | Inject Failure | Expected Recovery |
|---|---|---|
| Before PDF delivery | Kill worker after queue claim | Claim expires; same event is retried and completes |
| During PDF generation | Kill process before final audit | Staging/checkpoint files are reconciled; no duplicate document created |
| After workflow-document commit | Kill before channel audit | Existing document is reused by notification key |
| After SMTP acceptance | Kill before final audit | Retry may send a duplicate; history records both observations |
| Redis lease loss | Stop/block Redis during delivery | Worker cannot finalize; next worker retries after lease expiry |
| JVM restart during batch | Stop application while several rows run | Claimed rows become eligible again; no permanent stuck claim |
Staging Verification Queries¶
SELECT ID, STATUS, DELIVERY_ATTEMPTS, CLAIM_TOKEN, CLAIM_UNTIL, NEXT_ATTEMPT
FROM NOTIFICATION
WHERE ID = :notification_id;
SELECT IDENTITY_KEY, STATUS, ATTEMPTS, ERROR_MESSAGE, RECORDED_AT
FROM NOTIFICATION_DELIVERY_HISTORY
WHERE IDENTITY_KEY LIKE '%' || :notification_id || '%'
ORDER BY ID;
SELECT IDEMPOTENCY_KEY, STATUS, ATTEMPTS, ARTIFACT_PATH, DOCUMENT_ID
FROM NOTIFICATION_DELIVERY_OUTBOX
WHERE IDEMPOTENCY_KEY LIKE '%' || :notification_id || '%';
Also inspect the workflow storage directory for leftover .part files, .target checkpoint files, and duplicate {documentId}.pdf files after each injected failure.
Synchronous Gateway Boundary¶
Queue-backed workflow and generic-event notifications recover automatically. A direct NotificationFrameworkGateway call has no queue row to resume if the JVM dies during delivery; the caller must retry it, or the operation must first enqueue a durable notification event.
10. Queue Retries and Operational Recovery¶
Email PDF attachment names use the template's configured file prefix (or template name), followed by the reference number: Reservation-acknowledgement-RES0023KZN.pdf. Missing references fall back to the workflow/event/template identifier. Names are sanitized and bounded. A resource decorator changes only the MIME filename; workflow storage remains {documentId}.pdf, so document links and recovered artifacts retain their existing paths.
NotificationQueueService selects at most 25 eligible IDs, atomically claims each row with a UUID and expiry, renews claims, and updates outcomes only when its token still matches. Polls use fixed delay; expired scheduler locks do not bypass row ownership. Failed rows are delayed by:
The eighth failed attempt moves the row to STATUS = 2 (Dead Letter Queue). Status 0 is pending; status 1 is complete; status 2 is dead letter.
Operational Dead-Letter Runbook¶
After correcting the root failure, an operator can reset a dead-letter row's STATUS to 0, DELIVERY_ATTEMPTS to 0, and NEXT_ATTEMPT to null, only when it has no active claim:
UPDATE NOTIFICATION
SET STATUS = 0,
DELIVERY_ATTEMPTS = 0,
NEXT_ATTEMPT = SYSTIMESTAMP,
CLAIM_TOKEN = NULL,
CLAIM_UNTIL = NULL
WHERE ID = :notification_id
AND STATUS = 2
AND (CLAIM_UNTIL IS NULL OR CLAIM_UNTIL < SYSTIMESTAMP);
COMMIT;
Keep its event ID to resume unfinished channels. Do not delete confirmed delivery rows or flush Redis to request a new message; create a new notification event for an intentional resend.
Database Concurrency Safeguards¶
Virtual threads are enabled for the application, but database-bound event handling is bounded. AsynchronousSpringEventsConfig uses a dedicated bounded ThreadPoolTaskExecutor with configurable core/max/queue values and CallerRunsPolicy; it does not use an unbounded SimpleAsyncTaskExecutor. The default event limits are 1 core thread, 3 maximum threads, and a queue of 100. These limits must remain below the combined Oracle connection capacity reserved for the service. Hikari connection wait timeouts are environment-configurable; the development profile uses the same 30-second default as UAT and production rather than an excessive override.
11. Consolidated Schema and Environment Rollout¶
All notification DDL is consolidated in V3__notification_delivery_outbox.sql at the user's explicit request to reset earlier interim Flyway versions. V4/V5 source files were removed. The script creates/extends the ledger, history, workflow document identity, and queue ownership/retry fields. Existing rows and sequence values are preserved on reapplication.
This changes the checksum of previously applied V3. Stop notification workers and complete the explicitly agreed notification-history reset before normal Flyway startup; if V4/V5 were applied from an interim checkout, their history must be reconciled as well. Removing history is an operator action, not performed by this implementation. Do not change unrelated migration history or drop notification data merely to reapply the consolidated script.
Existing legacy workflow/template ledger keys cannot be mapped reliably to event IDs. Before requeuing pre-upgrade notifications, reconcile their already-sent outcomes to avoid resending messages after the event-key transition. Configure the provider acknowledgement regex before enabling framework SMS and ensure all nodes share workflow storage.
12. Verification & Post-Implementation Review¶
The regression suite covers recovered PDFs, failed artifact processing, unresolved ownership, distinct event identity, immutable optional null context, preserving delivered email on a blocked dependency, final audit before release, provider acknowledgement rejection, audit skip preservation, missing artifact recovery, channel selection, and template lookup.
The combined notification/event regression run passed 125 focused tests with zero failures and zero skips. Tests cover: * Queue success/backoff/contention. * Prepared attachment enrichment without re-rendering. * Event-document reuse and crash-file reconciliation without removing unrelated files. * Generic event listener transactions and in-app side effects. * Resource routing and search-share guards. * Lodgement and reservation action routing. * Expiry reminders and task reminders.
Live isolated checks also verified Redis stale-token rejection/renewal/release and Oracle consolidated V3 reapplication with delivery-row preservation. The disposable Oracle schema was removed. SGUSER schema/history was not changed, and no external email or SMS was dispatched during tests.
13. Known Bypasses and Stale Boundaries¶
The legacy EmailServiceHandler and SmsServiceHandler call sites in cis-api have been removed from the migrated notification paths. One workflow-module hook remains outside this module's framework boundary: WorkflowProcessingService calls emailHandlerService.triggerTaskEmail(...). It is intentionally not counted as a CIS API framework entry point and requires a cis-workflow change to migrate.
PDF generation is not globally unified yet. Non-notification reports and previews still call PdfGeneratorService directly (such as examination dockets and administrative reports), while notification PDF delivery uses the channel adapter and generic delivery orchestrator. The reservation acknowledgement preview now uses NotificationPdfPreviewService, which reuses framework context/template composition without creating a durable delivery. The remaining report and examination paths are valid document-generation paths today, but they do not receive queue, idempotency, durable delivery audit, or workflow-artifact semantics. Do not describe the framework as owning every PDF in the application until those contracts are deliberately migrated.
Within the framework itself, PdfNotificationDelivery remains the concrete PdfDocumentDelivery port implementation and PdfNotificationDeliveryAdapter is its generic-delivery bridge; neither is dead code.
14. Extending the Framework¶
Follow these six canonical rules when extending the notification system:
- Add an ordered domain context contributor and populate the live template variables.
- Carry the persisted notification ID into
NotificationPreparationService.prepare. - Pass the prepared payload to
NotificationService; select the supported channels. - Propagate failed results so the durable queue can back off or dead-letter the event.
- Add ordered artifact processors only for extra artifact responsibilities.
- Test event reuse, distinct events, missing optional data, transport failure, and artifact recovery.
[!CAUTION] Anti-Pattern Warning: Do NOT duplicate PDF generation, email wrapping, attachment sequencing, leases, or audit logic inside domain notification classes. The obsolete
NotificationChannelRendererand logging-only sink were removed; the adapter registry and persistent sink are the sole maintained paths.
Developer Recipes & Code Examples¶
Recipe A: Implementing a Domain Context Contributor¶
package com.dw.ngms.cis.service.notification.context.custom;
import com.dw.ngms.cis.service.notification.context.NotificationContextContributor;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import java.util.Map;
@Component
@Order(500)
public class CustomProcessContextContributor implements NotificationContextContributor {
@Override
public boolean supports(Long templateId, Long workflowId, Map<String, ?> attributes) {
return Long.valueOf(999L).equals(templateId);
}
@Override
public void contribute(Long templateId, Long workflowId, Map<String, ?> attributes, Map<String, Object> target) {
target.put("referenceNumber", attributes.get("referenceNo"));
target.put("statusDescription", "Approved");
// Add domain-specific template variables here
}
}
Recipe B: Triggering via Synchronous Gateway¶
@Service
@AllArgsConstructor
public class CustomAlertService {
private final NotificationFrameworkGateway notificationGateway;
public void dispatchAlert(Long workflowId, String userEmail, String referenceNumber) {
Map<String, Object> attributes = Map.of(
"referenceNo", referenceNumber,
"notificationId", UUID.randomUUID().toString()
);
// Automatically validates, prepares, acquires Redis lease, audits, and delivers
notificationGateway.sendEmail(999L, workflowId, userEmail, null, null, attributes);
}
}
Recipe C: Triggering via Asynchronous Durable Outbox¶
@Component
@AllArgsConstructor
public class CustomEventListener {
private final GenericEventEmailNotificationEnqueuer genericEmailEnqueuer;
@EventListener(condition = "#event.eventType.name() == 'CUSTOM_EVENT'")
public void onCustomEvent(GenericNotificationEvent<User> event) {
User user = event.getContent();
// Enqueues into NOTIFICATION table within the current database transaction
genericEmailEnqueuer.enqueue(
event.getEventType(),
999L,
user.getUserId(),
user.getEmail(),
null,
Map.of("firstname", user.getFirstName(), "surname", user.getSurname())
);
}
}
Recipe D: Adding a Custom Artifact Processor¶
@Component
@Order(200) // Runs after standard workflow document processor
public class CustomArchivalArtifactProcessor implements ArtifactProcessor {
@Override
public boolean supports(ArtifactContext context) {
return context.getChannel() == NotificationChannel.PDF;
}
@Override
public Resource process(Resource inputArtifact, ArtifactContext context) throws Exception {
// Perform additional operations (e.g. upload to external secure storage or add cryptographic watermark)
return inputArtifact;
}
}