Bulk PDF Upload v2 — Design¶

Companion implementation plan: 2026-08-11-bulk-pdf-upload-v2-plan.md. Decision record: ADR-012.

1. Context¶

Project admins cannot upload full-text PDFs themselves; they email them to the SyRF team, who copy files manually onto the Edinburgh web server (https://ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of existing PDFs). Epic #2223 (open since 2020) asks for self-service bulk upload with zero broken links, validation reporting, and safe archive handling. Spec/plan/backlog: discussions #2093 / #2094 / #2095.

Two prior implementation attempts did not land:

#2298 (Jan 2026) — closed unmerged.
#2373 (Mar–May 2026) — a functionally complete pipeline, but with no solution entries, no CI build, no chart, no feature flag, no production deploy path for its two new services, an ADR contradicting the implemented architecture, 36 unresolved review threads, and failing quality gates. It is ~4 months behind main.

v2 is a fresh implementation. #2373 is used as reference material only — its domain-model patterns, guard logic, and test cases inform this design; its code is not merged. #2373 will be closed as superseded when this plan merges.

2. Architecture overview¶

flowchart LR
    subgraph Browser
        A[Folder picker /<br/>drag-and-drop] --> B[Client validation<br/>PDFs only]
        B --> C[Pre-upload match preview<br/>vs search PdfRelativePaths]
        C --> D[Client-side ZIP<br/>web worker, STORE]
    end
    D -- presigned PUT --> S3[(S3 syrfapp-uploads*<br/>Projects/_bulk-staging/)]
    S3 -- ObjectCreated --> L[s3-notifier Lambda<br/>uploadkind=BulkPdfUpload]
    L -- IProcessBulkPdfUploadCommand<br/>+ presigned GET --> MQ[(RabbitMQ)]
    MQ --> AG[PDF agent<br/>.NET worker]
    AG -- INSTREAM scan --> CL[clamd sidecar]
    AG -- extract + copy --> ST[(Final storage<br/>prod: ecrf1 CIFS<br/>staging/preview: PVC)]
    AG -- IFinalizeBulkPdfUploadCommand --> MQ2[(RabbitMQ)]
    MQ2 --> PM[PM single-writer consumer<br/>match, mark studies,<br/>counts, CSV report]
    PM -- SignalR via Project stream --> Browser

Key properties:

The API never touches the bytes. Browser PUTs directly to S3 with a SigV4 signature from the existing FileUploadSignature pattern (SearchController.cs:95-162).
One processing locus. All scanning, validation, extraction, and copying happen in the agent. The Lambda is a thin notifier (extension of the existing s3-notifier).
Domain authority stays in PM. The agent is a "dumb pipe": it never matches files to studies and holds no cloud credentials. Matching, study marking, counts, and the CSV report are computed by the PM service from the agent's reported per-file outcomes.
Outbound-only from the university network. The agent pulls from S3 via a presigned GET and talks AMQP outbound. Nothing connects inbound to UoE hosts.

3. Resolved decisions¶

The full decision register (batch-grill, 2026-08-11). IDs are referenced throughout.

ID	Decision
A1	Per-search upload from the Systematic Searches page; effective payload cap 1 GB
A2	Matching = normalized relative path vs `Study.PdfRelativePath`; normalizer test vectors ported from #2373's `PdfPathNormalizerTests`
A3	All three conflict policies: `skip-existing` (default), `replace-always`, `replace-if-differs` (SHA-256)
A4	Folder-based upload: user selects/drops a folder of PDFs; client validates, previews matches, and builds the ZIP itself (web worker, STORE). ZIP is a transport detail
A5	Feature flag `bulkPdfUpload` (env-mapping generator); off in previews/production until rollout
A6	Per-file outcome taxonomy: Copied / Skipped / Replaced / Unmatched / Missing / Invalid / Infected. UI summary + downloadable CSV report
A7	Dedicated `BulkPdfUpload` authorization policy, initially granted to project-design roles
B1	`BulkPdfUploadJob` embedded in the Project aggregate (SearchImportJob precedent); absolute-snapshot counts; single terminal-writer consumer; `BsonClassMap` registered
B2	Signature endpoint mirrors `getSignature`; ZIP key under `Projects/_bulk-staging/` (satisfies both notification models with zero infra change)
B3	Status via the existing Project SignalR stream; no polling
B4	Files land at `projects/{projectId}/searches/{searchId}/pdfs/{entry-path}`; `PdfRelativePath` is never rewritten — link generation composes the prefix for bulk-delivered PDFs
B5	`PdfBaseUrl` env-supplied config (no hardcoded Edinburgh URL)
C1	Notifier dispatch rewritten to `Enum.TryParse` + `switch`, unknown kinds = logged no-op (also fixes live single-study-PDF Lambda errors)
C2	Point-to-point `Send` of `IProcessBulkPdfUploadCommand` (one owner), not publish
C3	Agent reads S3 via 12 h presigned GET generated by the Lambda; no AWS credentials on the agent
C4	Staging-ZIP cleanup via S3 lifecycle rule on `Projects/_bulk-staging/` (7 days), declared in the ACK Bucket CR
C5	Notifier ships via the standard zip-version promotion
D1	New .NET worker `src/services/pdf-agent/` with slnf, GitVersion, Docker image, Helm chart
D2	Extract with guards, then per-file clamd INSTREAM scan + `%PDF` magic check; any infection ⇒ whole upload `Infected`, nothing copied
D3	Batched absolute-snapshot progress + exactly one terminal finalize (always sent); idempotent under redelivery
D4	Same agent image in every environment: in-cluster (chart) for staging + previews writing a PVC; docker app on arrnc-api for production writing CIFS
D5	RabbitMQ over the existing `rabbitmq.camarades.net:5672`; AMQPS is a separate, non-gating hardening PR
E1	Production agent is gatekeeper-deployed on arrnc-api (mission-control-migration-worker precedent)
E2	Single bind volume `/srv/data/syrf-pdf/production` containing `scratch/` (ext4) and `output/` (nested CIFS mountpoint); sentinel-file guard before writes
E3	clamd as an Ansible-managed companion container (mailpit precedent) on a new non-internal `syrf-pdf-net` bridge; reached as `clamd:3310`; freshclam keeps signatures current
E4	CIFS mount via the existing `cifs_mounts` role + a new `cifs/<name>` age secret scope; `mount_user` uid = agent `container_uid` from day one
E5	New `github_runner` registration for `camaradesuk/syrf` on arrnc-api; GHCR pulls via ephemeral `GITHUB_TOKEN`; container secrets via workflow `-e` from GitHub environment secrets
F1	Delivery as waves of small PRs (see plan doc); ADR-010 Phase-5 unblock is a bundled prerequisite wave
F2	#2373 closed as superseded at plan merge; branch kept for reference
F3	ADR-012 records the architecture; stale docs updated in the PRs that touch them

4. Component design¶

4.1 Domain (Project Management core)¶

New embedded entity on the Project aggregate (pattern: SearchImportJob):

BulkPdfUploadJob
  Id                 Guid (uploadId; client-visible)
  SystematicSearchId Guid
  Status             Queued | Uploaded | Scanning | Copying | Finalizing
                     | Complete | Failed | Infected | Expired
  ConflictPolicy     SkipExisting | ReplaceAlways | ReplaceIfDiffers
  Counts             { Total, Copied, Skipped, Replaced, Unmatched, Missing, Invalid }
                     (Missing is null until finalize — it is only computable once the whole
                      ZIP has been processed and compared against the search's study paths;
                      progress snapshots never set it and the UI shows "—" until completion)
  InfectedFiles      string[]   (bounded: first 100)
  FailureReason      string?
  ReportFileKey      string?    (S3 key of the CSV report)
  CreatedAt / CompletedAt / CreatedBy

Rules:

Status transitions are idempotent guards: any command arriving for a terminal job is a logged no-op. Progress snapshots are absolute and monotonic (a stale snapshot never regresses counts).
Per-file outcomes are not stored in Mongo (unbounded); they live in the CSV report on S3. Mongo stores counts + bounded infected list.
Abandoned uploads expire: a job still Queued 24 h after creation (signature issued but the browser never completed the PUT — closed tab, lost connectivity) is surfaced as Expired and treated as terminal; the transition is applied lazily (computed on read, persisted on the next aggregate save) so no scheduler is needed.
History is bounded: on new-job creation, terminal jobs beyond the most recent 20 per search are pruned from the aggregate in the same save (their CSV reports persist in S3 independently). Prevents unbounded Project-document growth toward the 16 MB BSON limit.
SignalR delivery is explicit wiring, not free: ProjectDetailsDto enumerates job collections explicitly (SearchImportJobs, BulkStudyUpdateJobs, RiskOfBiasJobs) and AutoMapper drops unmapped properties — PR-1 must add BulkPdfUploadJobDto, the BulkPdfUploadJobs property, and the mapping, or the progress UI stays dark.
BsonClassMap.RegisterClassMap entries for BulkPdfUploadJob and its value objects with UnmapProperty(x => x.Project), matching every other embedded entity in ProjectRepository.cs (#2373 omitted this — a runtime serialization crash risk).

Study side:

Study.BulkPdfDeliveredAt : DateTime? and Study.BulkPdfDeliveredPath : string? — set together when a bulk upload actually delivers this study's PDF. The delivered path is the normalized path the agent really wrote, stored immutably: a later ordinary study update that rewrites PdfRelativePath (e.g. StudyReferenceFileParser, bulk study update) can then never break the link — the URL keeps serving the file that exists.
Link generation: when delivered, the PDF URL is {PdfBaseUrl}projects/{projectId}/searches/{searchId}/pdfs/{BulkPdfDeliveredPath} with each path segment percent-encoded (preserving /) so filenames containing #, ?, or spaces produce working links; otherwise the legacy {PdfBaseUrl}{PdfRelativePath}. Legacy studies are untouched (D-B4), including the existing absolute-URL special case: today's Study.GetLinkToPdf returns http-prefixed PdfRelativePath values unchanged, and the centralized builder must preserve exactly that behaviour or working legacy links corrupt.
One URL builder, all call sites: the repository currently has five independent PDF-URL constructions that bypass Study.GetLinkToPdf — StudyROBDto, StudyListItemDto, StatsWithIncompleteDto/StudyBaseDto, StudyDto, and PdfConverterService (used for risk-of-bias conversion). PR-1 centralizes URL construction in one bulk-aware helper and routes every one of those call sites through it; leaving any of them out ships broken links or failed ROB conversion for bulk-delivered studies.

4.2 Path normalization contract (correctness backbone)¶

One rule set, three consumers: client pre-upload matching (TS), link generation (C# and TS), agent/PM matching (C#).

Rules: Unicode NFC → invariant casefold → \ → / → collapse repeated / → trim → strip leading ./ and /. Two companion contracts ride with normalization:

Portability contract (the final storage is a Windows-backed IIS/CIFS share): entries containing Windows-reserved characters (< > : " | ? *, control chars), reserved device names (CON, PRN, AUX, NUL, COM1-9, LPT1-9), trailing dots/spaces on any component, or components longer than 240 UTF-8 bytes are rejected as Invalid — enforced in the client (friendly pre-upload error) and re-enforced in the agent (authoritative).
Collision contract: the normalized-key map is built up front; if two entries collapse to the same canonical key (A.pdf vs a.pdf from a case-sensitive filesystem), the upload is rejected before any write — otherwise ZIP order would silently decide which file wins. Client checks first; agent re-checks authoritatively.
Root-relative contract: the browser strips the selected folder's own name (folder-picker webkitRelativePath includes it), so ZIP entry paths are relative to the inside of the chosen folder for picker, drag-and-drop, preview, and matching alike.

The canonical test-vector file src/libs/kernel/SyRF.SharedKernel/PdfPathNormalization/normalization-vectors.json is loaded by both the C# and TS test suites; a vector added on one side fails the other side's build if unimplemented. Vectors seed from #2373's PdfPathNormalizerTests (252 lines) plus PDFs (2).csv real-world names.

4.3 API endpoints (flag-gated, `BulkPdfUpload` policy)¶

Endpoint	Purpose
`POST api/projects/{p}/searches/{s}/bulkPdfUpload/signature`	Create job (Queued) + return `FileUploadSignature` for `{prefix}Projects/_bulk-staging/{uploadId}.zip` with metadata `projectid, searchid, uploadid, conflictpolicy, virtualhost, uploadkind=BulkPdfUpload`
`GET api/projects/{p}/searches/{s}/bulkPdfUpload/history`	Upload history for the search
`GET api/projects/{p}/searches/{s}/bulkPdfUpload/{uploadId}/report`	Streams the CSV report from S3
`GET api/projects/{p}/searches/{s}/pdfPaths`	Study `PdfRelativePath` list for the client-side pre-upload match preview. Returns only studies with a non-null `PdfRelativePath` (all the preview needs), keeping the response lean; expected worst case a few MB for very large searches — no pagination

Metadata stays far below the 2 KB S3 cap: no manifest travels in metadata (the job doc and the report carry state).

4.4 Notifier (s3-notifier extension)¶

Dispatch: Enum.TryParse<UploadKind> + switch; default: logs and returns (no throw, no retry). This also stops the current production behaviour where objects without uploadkind/virtualhost metadata (today's single-study PDF uploads) make the Lambda throw (S3FileReceivedFunction.cs:107-142).
case BulkPdfUpload: validate metadata; generate 12 h presigned GET for the object. The Lambda role already holds s3:GetObject in every environment — via the ACK chart for staging/previews (iam-role.yaml:88-89) and via Terraform for production until ADR-010 Phase 6 (camarades-infrastructure/terraform/lambda/main.tf); Send IProcessBulkPdfUploadCommand { ProjectId, SearchId, UploadId, ConflictPolicy, ZipDownloadUrl, DateTimeEventOccurred } to the agent queue on the vhost from metadata (preview routing works unchanged).
Contract lives in SyRF.ProjectManagement.Messages (the notifier already depends on it).

4.5 Agent (`src/services/pdf-agent/`, new)¶

.NET worker, MassTransit consumer, concurrency 1. Per job:

Sentinel check: output root must contain the marker file .syrf-storage-ok. Missing marker ⇒ fail closed with StorageUnavailable (protects against writing "under" an absent CIFS mount — see §6.3). Provisioning: server-config creates it on the production share (mount-guarded task); the agent chart creates it on staging/preview PVCs via an init container — a fresh PVC must not fail-closed forever.
Download the ZIP via the presigned URL, streaming to scratch/{uploadId}/upload.zip with a byte counter that aborts past the 1 GB cap (client enforcement is advisory; a modified client could sign any size) — over-cap ⇒ terminal Failed. An expired URL (agent down long enough that a redelivered message outlives the 12 h presign) returns 403 — this maps to a terminal Failed status with a clear reason, not a retry loop; the user re-uploads.
Extract entry-by-entry with guards: Zip-Slip/path-traversal rejection, __MACOSX/ and ._*/.DS_Store skip (defence in depth — our client builds the ZIP), zip-bomb caps (≤ 10,000 entries, ≤ 4 GB total uncompressed, ≤ 500 MB per entry), the portability and collision contracts from §4.2, entry paths capped at 512 UTF-8 bytes (longer ⇒ Invalid), and only .pdf extensions (case-insensitive) — anything else is Invalid regardless of content, so a crafted payload.html starting with %PDF can never land under the IIS document root with an actively-served extension.
Scan each extracted file via clamd INSTREAM (clamd:3310). clamd is deployed with StreamMaxLength/MaxScanSize/MaxFileSize raised to cover the 500 MB entry cap (the image defaults are far lower and would silently truncate or reject) — a scan error or limit hit is a terminal Failed, never an unscanned pass. Any hit ⇒ record infected names, delete scratch, finalize as Infected — nothing is copied (all-or-nothing, D-D2).
Validate %PDF magic bytes per file (non-PDF ⇒ per-file Invalid outcome).
Copy to output/projects/{p}/searches/{s}/pdfs/{normalized-entry-path} via temp-file + atomic rename, honouring the conflict policy (ReplaceIfDiffers = SHA-256 compare against the existing file).
Report: IReportBulkPdfUploadProgressCommand (absolute snapshot) every 50 files; exactly one IFinalizeBulkPdfUploadCommand carrying status + per-file outcome entries, sent in a finally so a crash still terminates the job as Failed. Size budget: entries are { path ≤ 512 B, outcome, detail ≤ 256 B } (study identity is attached later by PM, which owns matching) — the caps bound the message at ≈ 8 MB absolute worst case, ~2 MB typical; well inside broker limits, and the caps exist precisely to keep that bound.
Outcome journal (crash-safe idempotency), two-phase: before each entry's atomic rename, the agent appends a pending record ({path, plannedOutcome}) to scratch/{uploadId}/journal.jsonl; immediately after the rename returns, it appends the matching committed record. A single-phase journal (write only after, or only before, the rename) leaves a crash window in one direction or the other; two phases plus a reconciliation rule close both. Redelivery replays the journal: a path with a committed record reuses that recorded outcome outright (no recompute — a file copied pre-crash stays Copied, never degrades to Skipped under skip-existing); a path with only a pending record is reconciled against the filesystem — if the destination file's hash matches the still-available scratch copy, the rename evidently completed (upgrade to committed and reuse the outcome); otherwise the rename never completed and the copy is redone. Counts/CSV stay truthful either way.
Cleanup scratch. The startup orphan sweep only removes upload directories with no active-lock file and an mtime older than 24 h — never a directory another live instance may own (in-cluster rolling updates can briefly run two pods against one PVC; the chart additionally uses the Recreate deployment strategy to avoid overlap by construction).

The agent holds: RabbitMQ credentials (workflow-injected secret) and a filesystem path. It has no AWS keys, no Mongo access, no HTTP surface, no inbound ports.

4.6 PM consumers (single writer)¶

One receive endpoint, concurrency 1, hosting both consumers — progress and finalize are serialized, eliminating aggregate write races (the lesson #2373 learned late):

Progress: monotonic snapshot update of Counts on the job.
Finalize: match reported file paths against IStudyRepository.GetPdfPathsBySearchId (normalized both sides); set BulkPdfDeliveredAt/BulkPdfDeliveredPath only for studies whose file outcome actually delivered bytes — Copied, Replaced, or verified-existing Skipped; Invalid and Failed outcomes leave the study untouched, and an Infected finalization marks nothing (all-or-nothing) — the zero-broken-links invariant lives or dies on this qualification. Compute Unmatched (file, no study) and Missing (study, no file); generate the CSV report and store it via IFileService to {prefix}Projects/{p}/Bulk PDF Uploads/reports/{uploadId}.csv; write terminal state + counts + report key in one aggregate save. Idempotent via the job-status guard.
Cross-replica safety: endpoint concurrency 1 is per process, and a rolling PM deploy briefly runs two pods against the shared queue. The endpoint therefore carries a retry policy for Mongo optimistic-concurrency conflicts (the idempotent guards + monotonic merge make retries safe), so a losing concurrent write re-runs instead of dead-lettering a finalize and stranding the job nonterminal.
Deletion guard: a systematic search (or its project) with a non-terminal bulk upload job cannot be deleted — the domain rejects it. Otherwise the agent would keep writing to a deterministic path whose aggregate no longer exists, stranding files and an unprocessable finalize.

Once PR-1's BulkPdfUploadJobDto + ProjectDetailsDto.BulkPdfUploadJobs + AutoMapper mapping are in place (§4.1), subsequent Project saves stream to subscribed clients automatically — no additional plumbing per consumer needed.

4.7 Frontend¶

Angular standalone dialog + @ngrx/signals store on the Systematic Searches page (flag-gated menu action):

Folder selection: drag-and-drop (webkitGetAsEntry traversal) and folder picker (webkitdirectory).
Validation: only directories and PDFs (extension + first-bytes %PDF sniff); offenders listed and upload blocked. OS junk (.DS_Store, __MACOSX/, ._*, Thumbs.db) is silently skipped, never an offender — a valid folder picked on macOS must not be rejected for artifacts the agent's own skip list would ignore anyway.
Pre-upload preview: matched / unmatched / missing table computed client-side against GET .../pdfPaths using the shared normalization rules; user confirms with eyes open.
ZIP built in a web worker with a streaming zip library, STORE mode (PDFs are already compressed; avoids CPU and memory blowup at 1 GB), written to a Blob (browsers spill large Blobs to disk). Memory contract for the SigV4 payload hash: the signature needs the payload SHA-256, but the ZIP must never be materialised via arrayBuffer() — the hash is computed incrementally by chunked reads of blob.stream(), then the Blob is PUT to S3 (streamed from disk) with progress. Peak memory stays at chunk size, not payload size.
Live status via the existing Project SignalR stream (selectSignal per repo modernisation rules); result summary panel (headline counts, infected/unmatched highlights); history table; CSV report download.

4.8 CSV report columns¶

file_path, normalized_path, outcome, study_id, study_title, detail — one row per ZIP entry, plus one row per Missing study (outcome missing). Header row + UTF-8 BOM for Excel friendliness. RFC 4180 quoting throughout, and formula neutralization: any cell beginning with =, +, -, or @ is prefixed with ' — ZIP paths and study titles are user-controlled and the report is opened in Excel by other admins. Covered by CSV tests.

5. Environments¶

	Production	Staging	Preview (per PR)
Bucket	`syrfapp-uploads`	`syrfapp-uploads-staging`	`syrfapp-uploads-pr-{n}`
Notifier	shared Lambda (per ADR-010 state)	staging Lambda (ACK)	per-PR Lambda (ACK)
RabbitMQ vhost	production	staging	per-PR
Agent hosting	docker app on arrnc-api (gatekeeper)	in-cluster (chart)	in-cluster (chart)
Final storage	ecrf1 CIFS mount	PVC	PVC
PDF serving	existing IIS site	static server in agent chart	static server in agent chart
Flag	off until rollout complete	on for rehearsal	per-PR `#preview-config`

Why PVC rather than ecrf1 subfolders for staging/previews: (1) in-cluster agents cannot reach the share — the UoE firewall allows no inbound SMB from the internet, which is the very reason the production agent lives on arrnc-api; (2) per-PR agents cannot live on arrnc-api — its container slots are statically declared in server-config with manual production applies, while previews are created/destroyed automatically per PR; (3) lifecycle isolation — preview PDFs should die with the preview namespace (PVC deletion is automatic; ecrf1 subfolders would orphan test files on the backed-up production estate); (4) security — preview/staging environments must not hold production file-server credentials.

Why not host the staging/preview agents on arrnc-api as gatekeeper docker apps: previews are structurally impossible there — the gatekeeper deploys only pre-declared, manually-applied whitelist slots (its security model), and a preview must run the PR's own agent image to E2E-test agent changes, which a static shared container never could. Staging is feasible but a bad trade: it could not rehearse the real CIFS write anyway (staging data must not touch the production share, so it would write local disk — no more production-like than a PVC), while it would add a second bespoke deploy lane outside the standard cluster-gitops promotion, new serving infrastructure on the pet host for staging's PdfBaseUrl, and a staging environment that no longer validates the same chart previews use. The rule: everything stays in the declarative GitOps world unless network reality forces it out — only the production writer is forced out, because only a UoE host can reach the SMB share. The production-hosting specifics are rehearsed by the Wave-3 smoke test on the real slot instead. The agent image and code path are identical everywhere; only the mount and PdfBaseUrl differ, and the production-only pieces (CIFS mount, gatekeeper deploy) are covered by the Wave-3 smoke test.

Verified connectivity from arrnc-api (2026-08-11): SMB 445 to ecrf1.clinicaltrials.ed.ac.uk (= igmm-app2.igmm.ed.ac.uk) OK; S3 443 OK; RabbitMQ rabbitmq.camarades.net:5672 OK (5671 closed); cifs-utils installed.

6. Hosting on arrnc-api (production)¶

Managed via camaradesuk/server-config (Ansible) + camaradesuk/arrnc-api-deploy (gatekeeper). All patterns below have direct precedent in those repos.

6.1 Agent container¶

Gatekeeper-deployed daemon (precedent: mission-control-migration-worker-production, vars/projects.yml:748-765): nominal host_port: 8089 (nothing listens; 8070–8088 taken), no domains:, default service lifecycle, --restart unless-stopped, volume_path: /srv/data/syrf-pdf/production → /app/data, skip_web_acl: true, container_uid = the Dockerfile's dedicated non-root uid. Deploys run sudo /usr/local/bin/container-web-deploy syrf-pdf-agent-production ghcr.io/camaradesuk/syrf-pdf-agent:<tag> -e ... from a syrf deploy workflow.

Runner-boundary gate. CLAUDE.md currently rules that deploy/secret-bearing syrf jobs stay GitHub-hosted and prohibits SyRF-specific runner labels — written for the shared juniper CI pool. Hosting a deploy-only runner on arrnc-api (the established pattern every other tenant of that host uses: futurems-ecrf, edd-intake, deployment-portal) is a deliberate exception that must be reviewed and recorded before Wave 3: PR-3 updates the CLAUDE.md runner-boundary section to define the deploy-runner category (server-config-managed host, gatekeeper-restricted sudo, no build/test workloads, repo-scoped runner) as part of that review. Fallback if the review rejects it: a GitHub-hosted job invoking container-web-deploy on arrnc-api over SSH with a dedicated restricted key. See the Wave-3 prerequisite in the plan doc.

6.2 clamd companion¶

Ansible-managed container (mailpit precedent, roles/mailpit_staging/): new roles/clamav running clamav/clamav, joined to new bridge syrf-pdf-net (declared in docker_networks, non-internal so freshclam keeps outbound 443), no published ports — the agent reaches it as clamd:3310 via docker DNS. Signature DB on a host volume so restarts don't re-download. freshclam (bundled in the image) refreshes signatures on a timer and hot-reloads clamd. The role mounts a clamd.conf drop-in raising StreamMaxLength/MaxScanSize/MaxFileSize to cover the 500 MB entry cap (image defaults are far lower); the dev docker-compose clamd and the chart's clamd sidecar carry the same settings so limits behave identically everywhere.

6.3 CIFS mount¶

One new item in the existing cifs_mounts list (vars/projects.yml:972-994 precedent): //igmm-app2.igmm.ed.ac.uk/w3dev/csena/Camarades (the share root behind https://ecrf1.clinicaltrials.ed.ac.uk/camarades/; ecrf1 is a DNS alias of igmm-app2) → /srv/data/syrf-pdf/production/output, mount_user = agent uid, credentials from a new cifs/syrf-pdfs age scope (secrets_age_to_cifs bridge — zero new machinery, same pattern as the NeuroCARE share mounts). Credential handling: values stored in LastPass, then age-encrypted into vars/secrets.age.yaml in server-config — all configuration lives in server-config. The mountpoint nests inside the single gatekeeper bind volume; roles/docker_volumes ACLs are non-recursive (tasks/main.yml:22-80) so they don't touch the CIFS submount. Because a bind mount snapshots submounts at container start, a CIFS remount mid-flight is invisible to the running container — hence the sentinel file (§4.5) and a runbook note: restart the agent container after any manual remount.

6.4 Secrets and env¶

Non-secret env via vars/portal-env-overrides.yaml; the RabbitMQ password via the deploy workflow's -e args from GitHub environment secrets (the tenant-owned channel per arrnc-api-deploy README). CIFS credentials are host-level age secrets, never in the container env.

7. ADR-010 interaction (prerequisite)¶

Audited 2026-08-11: the production s3-notifier ACK cutover (cluster-gitops#344) merged but stalled — the ArgoCD app has been OutOfSync/Degraded since 2026-05-12 with the Function CR in ACK.Terminal "Resource already exists"; the production Lambda is still effectively Terraform's. Two primed hazards directly affect this feature:

The chart's Bucket manifest (never yet applied to production) declares a bucket-wide, unfiltered notification and no lifecycle rules. Its first successful production sync would drop Terraform's Projects/ prefix filter and the live Incomplete Multipart Cleanup lifecycle rule (ACK treats spec as whole desired state).
The production IAM role lacks the ignore_changes handover block staging has — a routine terraform apply and ACK fight over a live role.

This rollout therefore bundles a minimal unblock as Wave 0 (see plan doc): template notification filters + lifecycle rules in the chart (carrying the existing multipart rule), add the missing ignore_changes block, resolve the Function CR adoption conflict, and verify a no-op production sync. Our C4 lifecycle rule and the notifier code change then ride the normal promotion path. Full audit is recorded in the session memory and summarised in the plan doc's Wave 0.

8. Security model¶

No cloud credentials on UoE hosts (C3): presigned GET only, 12 h expiry.
No inbound connections to UoE: agent pulls S3 and AMQP outbound only.
All-or-nothing infection policy: an infected archive delivers nothing.
Server-side validation is authoritative: client checks are UX; the agent re-validates structure, scans every file, and checks magic bytes regardless of client behaviour.
Extraction hardening: Zip-Slip guards, entry/size caps, atomic writes.
AuthZ: dedicated BulkPdfUpload policy (A7); internal reporting flows via RabbitMQ only — no internal HTTP endpoints, no API keys (unlike #2373).
Known accepted risk (existing posture): AMQP to rabbitmq.camarades.net:5672 is plaintext over the public internet — same as the production Lambda today. A separate hardening PR adds AMQPS 5671; the agent flips its URI when it lands (D5).

9. Out of scope (MVP)¶

Retry of individual failed files, cancel mid-run, strict mode, orphan-upload sweeper beyond the S3 lifecycle rule, migration of the existing 700 GB into the new layout, AMQPS enablement (separate PR), ADR-010 Phases 3-cleanup/6 beyond the minimal Wave-0 unblock.

10. Open inputs¶

~~ecrf1 SMB share name/path~~ Resolved 2026-08-11: share root is \\igmm-app2.igmm.ed.ac.uk\w3dev\csena\Camarades. Credentials to be added to LastPass and age-encrypted into server-config (cifs/syrf-pdfs scope) before the Wave-3 production apply — NeuroCARE CIFS precedent.
Confirmed 2026-08-11: the IIS site serves new subdirectories without extra configuration.

11. Reference index¶

Discussions #2093/#2094/#2095/#2241/#1930 · epic #2223 · issues #2313, #2406/#2407, #2516–#2546,

2588/#2589 · prior PRs #2298 (closed), #2373 (to be superseded) · ADR-010 and its audit ·¶

docs/architecture/systematic-search-upload-flow.md (notifier reference; stale sections updated in Wave 2).