Pipeline deployment audit and RustFS rollout¶
Scope and verified findings¶
The current checkout deploys cis-api and cis-workflow. The default registry is now registry.digitalocean.com/cadastral in publishing scripts, Compose defaults and shipped UAT/production templates. Existing Bitbucket CONTAINER_REGISTRY overrides must also be changed in Bitbucket settings; repository code cannot overwrite those remote settings.
The old pipeline copied files, then attempted execution through a separate ssh-run command containing nested shell quotes and interpolated secret JSON. Copy success did not establish script execution or application health. Without the failed run's SSH logs, the exact observed server failure cannot be proven. The repository audit established these gaps:
| Finding | Change |
|---|---|
| Publish step had no deployment context, so deployment-scoped registry secrets could be absent | Publishing and remote deployment now share the same Bitbucket deployment step |
| Registry authentication warned and continued when credentials were absent | Local preflight fails before publishing/copying |
| Generic user/host fallbacks could resolve unintended destinations; remote directory silently defaulted | Explicit settings or named UAT/PROD aliases required; APP_DIR must already exist and be writable |
| Secrets were embedded in a nested YAML/SSH command string | Bash-serialized exports travel over SSH stdin; tracing disabled; credentials stored in temporary Docker config directories |
| Several duplicated UAT/prod/custom flows could drift | YAML anchors call shared publish/remote/server scripts |
| Deployment ignored the build's commit tag and used server env image settings | Server exports both exact commit-tagged images before Compose interpolation |
No CIS_ENV was explicitly selected remotely |
Server exports the validated target environment and exact application env-file paths |
Success meant up -d returned, not that applications were ready |
Server waits for running/healthy application containers and propagates failure |
| Shell files could arrive with CRLF | LF Git attributes and normalized copied script bundle |
| Host-path preparation did not use Compose's actual mount resolution | Removed guessed directory creation; provision bind mounts/network explicitly and let Compose use the selected env files |
Atlassian documents deployment variable scoping and pipeline SSH identity handling. The pipeline retains existing branch behavior: UAT from release/uat or uat; production from main with a manual publish/deploy gate; custom UAT/production flows reuse the same steps. This change does not restrict backend deployments to the separate frontend repository's branch policy.
Execution contract¶
- CI runs deployment contract tests with mocked tools, then Gradle test-class compilation.
- A deployment step resolves the environment and runs
scripts/deploy.sh. scripts/deploy.shverifies local preflight without contacting the server, logs into the registry, builds both multi-stage Dockerfiles and pushes<environment>-<first-seven-commit-characters>plus the environment alias.- It checks SSH access and the existing writable APP_DIR before copying. It copies only deployment Compose and LDAP/Oracle supporting assets. No
--deleteis used; server environment/data files are preserved. - The script prints
3/3 Executing deployment script on serverand sends its orchestration directly over SSH stdin. - The server prints
[server] Started deployment, verifies Docker access, Compose and all three env files, validates the rendered Compose config, logs into the registry, pulls the exact application images, starts the stack and waits for application readiness. - SSH returns the actual server exit status.
[deploy] Remote script completed successfullyis printed only after success. Errors identify the failing phase without printing secrets.
Required Bitbucket settings¶
Configure these as repository variables or, for environment-specific secrets/hosts, variables under the matching uat / production deployment environment:
| Setting | Meaning |
|---|---|
SSH_HOST, SSH_USER, APP_DIR |
Explicit target host/user and existing absolute directory; spaces and shell metacharacters in APP_DIR are rejected |
SSH_PORT |
Optional; defaults to 22 |
SSH_KEY |
Secured raw or base64-encoded SSH private key; alternatively use the configured Bitbucket key through BITBUCKET_SSH_KEY_FILE/default SSH identity |
SSH_KNOWN_HOSTS |
Verified known-hosts lines; alternatively configure Bitbucket's known-hosts file. Nondefault SSH ports require the corresponding [host]:port entry |
CONTAINER_REGISTRY |
Optional; defaults to registry.digitalocean.com/cadastral; update any old override |
DOCKER_CONFIG_JSON / DOCKER_CONFIG_B64 |
Secured Docker auth JSON or base64 JSON |
DIGITALOCEAN_ACCESS_TOKEN |
Alternative registry login token; optional DOCKER_REGISTRY_USER override |
DEPLOY_HEALTH_TIMEOUT |
Optional application readiness deadline in seconds; default 240 |
DEPLOY_SERVICES |
Optional cis-api, cis-workflow, or both; default updates the full profile-selected stack |
Existing UAT_SSH_USER/UAT_SERVER_HOST/UAT_APP_DIR/UAT_SSH_PORT/UAT_SSH_KEY and PROD equivalents remain supported. Generic settings take precedence. The script never falls back to the pipeline container's USER/HOST. DEPLOY_TARGET_NAME is selected by the YAML step; IMAGE_TAG is derived from BITBUCKET_COMMIT unless explicitly provided for a controlled manual deployment.
The SSH user needs Bash, Docker CLI/Compose v2 and Docker daemon access. Provision the external Traefik network and required bind-mount directories/permissions. In APP_DIR, supply .env.uat or .env.prod, plus matching files under cis-api/ and cis-workflow/. DEPLOY_ENV_FILE, DEPLOY_API_ENV_FILE and DEPLOY_WORKFLOW_ENV_FILE can select alternative paths. Deployment Compose remains the fixed docker-compose-deploy.yml entrypoint.
Upload storage path¶
Bitbucket's APP_DIR is forwarded over SSH and is authoritative during deployment. The server script exports HOST_UPLOADS_DIR=${APP_DIR}/data/upload, overriding stale upload paths in server environment files. Compose mounts that host directory at /data/uploads and explicitly sets the API's APP_STORAGE_LOCATION=/data/uploads inside the container. For APP_DIR=/opt/cis/backend, uploads are stored on the server at /opt/cis/backend/data/upload.
Do not set APP_STORAGE_LOCATION to the host path inside the API environment file: the process accesses the container mount path. Provision the host directory with write permissions for the API container user before deployment. Existing files under an older uploads directory are not moved automatically; copy/restore them to the new directory before switching the mount.
RustFS instead of MinIO¶
Local and deployed companion stacks now use rustfs/rustfs and the RustFS rc client (rustfs/rc) for bucket creation and the existing public bucket policy. No MinIO image, mc command or legacy MINIO variable is needed. The existing S3-compatible application property names DO_SPACE_* remain because they describe the application's S3 client rather than a particular server implementation.
- S3 endpoint:
http://rustfs:9000inside the Compose network. - Console: port 9001; server-published ports remain loopback-bound by default.
- Credentials:
RUSTFS_ACCESS_KEYandRUSTFS_SECRET_KEY, matching the API's DO_SPACE_KEY/DO_SPACE_SECRET. - Images: RUSTFS_IMAGE, RUSTFS_INIT_IMAGE and RUSTFS_PERMISSIONS_IMAGE; pin deployment-approved tags/digests through env files.
- Storage: HOST_RUSTFS_DATA_DIR; a permissions init job changes the mount root owner to RustFS uid/gid 10001. It does not recursively change existing object ownership.
- Readiness:
/health/ready; the bucket init job waits for the server and fails on command errors instead of unconditionally exiting zero. The API waits for successful bucket initialization when that companion service is enabled. - Routing/resource variables use RUSTFS names, including CIS_RUSTFS_HOST and TRAEFIK_RUSTFS_ENABLE.
The implementation follows the official RustFS Docker instructions and RustFS CLI bucket commands. Local/sample credentials must be replaced for deployment.
Existing object data is not migrated by renaming a service. Use a new RustFS data directory; do not point it at MinIO's internal data directory. Back up and copy objects using S3-compatible tooling, validate counts/content, then switch the application endpoint. Old storage is not deleted by these changes. Server files are not overwritten by the pipeline bundle: apply the new root RUSTFS variables and matching application DO_SPACE credentials/endpoint on the server before rollout.
Validation and limits¶
Restricted Bitbucket Buildx failure¶
An observed pipeline run failed before reading the Dockerfile with authorization denied by plugin pipelines: --privileged=true is not allowed, while starting moby/buildkit through the docker-container driver. publish.sh now explicitly sets DOCKER_BUILDKIT=0 so Docker 27 uses its daemon-native classic builder. Both application Dockerfiles therefore use ordinary RUN instructions rather than BuildKit-only RUN --mount cache syntax. Existing Docker layer caching remains; BuildKit cache mounts are no longer used. The regression test starts publishing with BuildKit enabled in its parent environment and verifies that both builds override it.
This follows Atlassian's documented BuildKit restrictions and disable switch. It does not enable privileged containers or change server-side deployment. A real Bitbucket rerun is still required to verify publishing on that runner.
After this fix, all 18 deployment contract checks passed, including forced classic-builder selection when the parent environment enables BuildKit. Java production/test compilation also passed; a live Docker image build remains unverified locally.
Run bash scripts/deploy/test-deployment.sh. These contract tests execute the real orchestration/server scripts against fake SSH, rsync and Docker, covering successful copy-to-execution, exact image/environment selection, secret quoting, preserved env files, and failures before/after copying. CI runs the same checks before Java compilation. Also validate each shell file with bash -n and render local/UAT/production Compose with docker compose ... config --quiet.
No real registry publish or server deployment is performed by these checks. Production acceptance still needs the configured SSH host/key, remote env/mount permissions, registry access and real application dependencies. Docker Desktop's Linux daemon was unavailable during this pass, so RustFS image startup/bucket initialization could not be exercised live; Compose rendering and the mocked deployment contracts are distinct from live integration.
Recorded validation: all 17 deployment contract checks passed, all deployment/entrypoint Bash files passed syntax checks, pipeline/Compose YAML parsed, and local/UAT/production Compose configuration checks passed. Environment scans found no legacy MinIO settings in actual or example root/API environment files. No remote deployment was initiated.
UAT readiness follow-up — 2026-10-16¶
Status: not yet ready for an unconditional UAT release. Current production and test sources compile successfully with gradle testClasses; this does not run application tests. Compose rendering confirms the aligned API/workflow Oracle service configuration, LDAP/Redis/RustFS credential consistency and upload mount. The observed restricted-runner Buildx failure has a repository fix above, pending a real Bitbucket rerun.
Outstanding code findings:
| Priority | Finding | Required action |
|---|---|---|
| Resolved in code; live Oracle validation pending | compose/oracle/init/01_import_dump.sh previously marked failed imports completed. |
SQL/OS errors now fail; only Data Pump exit 0 creates the marker. Populated schemas and concurrent/stale locks block imports; REPLACE removed. CI regression tests cover failures, empty-schema retry, partial-import refusal and success/skip. Verify legacy markers and the actual UAT import state before rollout; see compose/oracle/init/README.md. |
| High | PersistenceConfig.flywayMigrationStrategy() runs flyway.repair() automatically before migrate. With the branch's consolidated/deleted migration files, this can accept changed history instead of stopping for review. |
Remove unconditional startup repair and reconcile actual UAT history explicitly before applying the consolidated migration set. |
| High | FtpService.open() installs PrintCommandListener(PrintWriter) without login suppression, then sends the real FTP credentials. |
Remove protocol console logging or enable credential suppression, and verify secrets never reach application logs. Apache's listener implementation defaults suppression to false. |
Remaining rollout/verification conditions:
- Runtime application settings still come from server env files. Updated ignored local
.env.uatfiles are not transferred by remote.sh; update the server files through the approved configuration path, or implement secure delivery from Bitbucket settings before relying on those local corrections. - Provision writable upload/log bind mounts for the non-root application containers and migrate old uploads/object data before switching directories/services.
- Verify existing UAT Flyway history, Oracle dump contents/import result, LDAP login, mail, external FTP/SMS connectivity and RustFS bucket operations against the actual environment.
- Docker Desktop's daemon is unavailable locally, so no fresh image build, container startup or live server smoke test was completed. Existing JARs on disk were not treated as proof of a current image build.
- CI currently runs deployment contract tests and Java test-class compilation, not the full Java application test suite.
The Oracle import finding has since been addressed with guarded initialization and regression tests. Oracle migration behavior and FTP logging remain unchanged; real Oracle import validation is still pending.