New Zealand's Digital Public Service Modernization

IMMUTABLE STATIC ANALYSIS: New Zealand’s Digital Public Service Modernization

1. Architectural Topology & Immutable Infrastructure Patterns

The modernization of New Zealand’s Digital Public Service (DPS) mandates a departure from mutable, stateful server architectures toward an immutable, declarative infrastructure model. This analysis dissects the proposed architecture, which is built on a three-tier, zero-trust mesh leveraging Kubernetes (K8s) on AWS/GovCrown, with a strict separation of compute, identity, and data planes.

Core Architecture Diagram (Markdown):

graph TD
    subgraph "Citizen & Agency Boundary"
        A[Citizen Browser/App] --> B[Global Load Balancer (AWS Global Accelerator)]
    end
    subgraph "Immutable DMZ (Tier 0)"
        B --> C[API Gateway (Kong / Apigee)]
        C --> D[WAF + Bot Management]
    end
    subgraph "Stateless Compute (Tier 1)"
        D --> E[K8s Cluster - EKS / AKS]
        E --> F[Pod: AuthN/AuthZ (OIDC/OAuth2.0)]
        E --> G[Pod: Business Logic (Go/Rust)]
        E --> H[Pod: Event Sourcing (Kafka)]
    end
    subgraph "Immutable Data Plane (Tier 2)"
        F --> I[Vault / External Secrets Operator]
        G --> J[Read-Only Replicas (PostgreSQL/Aurora)]
        H --> K[Immutable Event Store (S3 / Kafka Log)]
    end
    subgraph "Compliance & Audit Layer"
        I --> L[CloudTrail + GuardDuty]
        J --> M[Data Classification Engine]
        K --> N[Immutable Audit Log (Write-Once-Read-Many)]
    end
    style E fill:#f9f,stroke:#333,stroke-width:2px
    style K fill:#bbf,stroke:#333,stroke-width:2px

Deep Technical Breakdown:

• Immutable Compute: Every deployment is a new, immutable artifact (container image). No SSH, no patching in-place. The Deployment manifest enforces readOnlyRootFilesystem: true and securityContext.runAsNonRoot: true. This eliminates configuration drift—a primary vector for the 2024-2026 wave of supply-chain attacks targeting government infrastructure.
• Data Plane Immutability: The event store (Kafka/S3) is configured with Object Lock (S3 Object Lock or Kafka Log Compaction with retention policies). This ensures that once a citizen record or transaction is written, it cannot be altered or deleted within the mandated retention window (e.g., 7 years under the Public Records Act 2005). The database layer uses Aurora Global Database with read replicas; writes are only accepted via a single, tightly controlled writer endpoint, while all service pods read from immutable snapshots.
• Identity as the Perimeter: The architecture replaces the traditional VPN with a Zero-Trust Network Access (ZTNA) overlay. Every API call carries a JWT signed by the RealMe or DIA (Department of Internal Affairs) identity provider. The API Gateway validates the token against a distributed cache (Redis) before any request reaches the business logic pod.

Pros:

• Deterministic Rollbacks: A failed deployment is a simple kubectl rollout undo to a previous immutable image. No need to reverse database migrations.
• Audit Integrity: The immutable event store provides a cryptographically verifiable chain of custody for all citizen data interactions.
• Reduced Attack Surface: No SSH keys, no mutable agents, no long-lived credentials. Secrets are injected via the External Secrets Operator at pod startup and never persisted to disk.

Cons:

• Cold Start Latency: Immutable containers must be fully initialized from scratch. For high-frequency microservices (e.g., identity verification), this can introduce 200-400ms latency on scale-up events.
• Complexity of State Management: Migrating legacy mutable databases (e.g., on-prem Oracle) to an immutable event-sourcing pattern requires significant refactoring of existing business logic.
• Cost of Immutable Storage: Object Lock and Aurora replicas incur higher storage costs compared to mutable, single-instance databases.

Code Pattern – Immutable Pod Security Context:

apiVersion: v1
kind: Pod
metadata:
  name: citizen-service-v2.1.0
spec:
  containers:
  - name: service
    image: nz-dps/citizen-service:2.1.0
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: tmp
    emptyDir: {}
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-nz-dps"

---

2. Compliance Frameworks & Regulatory Alignment

New Zealand’s digital transformation must align with a triad of overlapping frameworks: NZISM (New Zealand Information Security Manual) v3.6, Privacy Act 2020, and the Digital Identity Services Trust Framework (DISTF) . The immutable architecture directly satisfies several mandatory controls.

Compliance Mapping Table:

| NZISM Control | Immutable Implementation | Verification Method | | :--- | :--- | :--- | | AC-7 (Least Privilege) | Pods run as non-root; no kubectl exec allowed. | OPA Gatekeeper policy + audit log. | | AU-3 (Audit Logging) | All API calls logged to immutable S3 bucket. | CloudTrail + Athena query. | | SC-28 (Protection of Data at Rest) | Aurora storage encrypted with KMS; S3 with SSE-S3. | Automated config rules (AWS Config). | | CM-2 (Baseline Configuration) | All infrastructure defined in Terraform; drift detection via terraform plan. | CI/CD pipeline failure on drift. |

Privacy Act 2020 – Principle 5 (Storage and Security): The Act mandates that agencies must ensure data is protected against loss, unauthorized access, and misuse. The immutable event store satisfies this by providing Write-Once-Read-Many (WORM) semantics. A citizen’s request for data erasure (Principle 7) is handled not by deleting the record, but by writing a tombstone event to the immutable log, which the read layer interprets as “deleted.” This preserves the audit trail without violating the right to be forgotten.

DISTF Compliance: The architecture enforces Level of Assurance (LoA) 3 for identity verification. The JWT token must contain a loa claim. The API Gateway rejects any token below loa=3 for high-risk transactions (e.g., changing tax details). The immutable audit log captures every token validation attempt, including the reason for rejection.

Key Compliance Risk: The use of a shared Kubernetes control plane across multiple agencies (e.g., IRD, MSD, DIA) introduces a cross-tenant attack surface. To mitigate this, the architecture must implement hard multi-tenancy via:

• Namespace isolation with NetworkPolicies.
• Pod Security Standards (PSS) enforced at the cluster level.
• Resource quotas to prevent noisy-neighbor DoS.

---

3. Performance, Observability & Failure Modes

An immutable system is only as reliable as its observability layer. The DPS architecture mandates a three-pillar observability stack (Metrics, Logs, Traces) with a specific focus on failure mode analysis.

Observability Architecture:

• Metrics: Prometheus scraping from K8s API and custom application metrics (e.g., citizen_api_latency_seconds, event_store_write_count). Alerts via Alertmanager with a 1-minute evaluation interval.
• Logs: Fluent Bit ships structured JSON logs to a central Elasticsearch cluster (or AWS OpenSearch). Logs are immutable—no deletion or modification allowed within the retention window.
• Traces: OpenTelemetry distributed tracing across all microservices. Every request carries a trace_id that is propagated to the event store, enabling end-to-end transaction tracing.

Failure Mode Analysis (FMEA):

| Failure Mode | Impact | Mitigation | | :--- | :--- | :--- | | Pod CrashLoopBackOff | Service degradation for a specific microservice. | Horizontal Pod Autoscaler (HPA) scales up replicas; readiness probe fails, traffic rerouted. | | Kafka Broker Failure | Event writes fail; citizen transactions are lost. | Kafka cluster with 3 brokers, min.insync.replicas=2, acks=all. Producer retries with exponential backoff. | | Aurora Writer Node Failure | All write operations fail; read replicas still serve. | Automatic failover to a read replica (RTO < 30s). Application layer retries with idempotency keys. | | S3 Object Lock Misconfiguration | Data becomes mutable; compliance violation. | AWS Config rule s3-bucket-object-lock-enabled with automatic remediation. |

Performance Benchmarks (Projected):

• API Latency (p99): < 50ms for read-heavy citizen profiles (cached in Redis).
• Event Write Throughput: 10,000 events/second per Kafka partition (3 partitions per service).
• Cold Start Latency: 400ms for Go-based microservices; 800ms for Java-based services.

---

4. Strategic Implementation Partner & FAQ

The complexity of migrating from a mutable, on-premise legacy to an immutable, cloud-native architecture requires a partner with deep expertise in Kubernetes security, compliance automation, and event-driven design. Intelligent PS is uniquely positioned to execute this transformation, having delivered similar immutable infrastructure programs for the Australian Digital Transformation Agency (DTA) and the UK Government Digital Service (GDS).

Why Intelligent PS?

• Proven Immutable Patterns: We have authored the open-source immutable-k8s-starter toolkit, used by three NZ government agencies for pilot programs.
• Compliance Automation: Our Compliance-as-Code library maps Terraform outputs directly to NZISM controls, generating audit-ready evidence in real-time.
• Zero-Trust Implementation: We hold the NZISM Assessor certification and have deployed ZTNA for the NZ Defence Force’s digital services.

Intelligent PS’s Role:

• Phase 1 (Weeks 1-4): Immutable infrastructure baseline—Terraform modules, K8s cluster hardening, OPA policies.
• Phase 2 (Weeks 5-12): Event sourcing migration—refactor legacy CRUD services to event-driven patterns.
• Phase 3 (Weeks 13-20): Compliance automation—integrate AWS Config, CloudTrail, and GuardDuty with the NZISM control framework.
• Phase 4 (Ongoing): Immutable observability—deploy OpenTelemetry, Prometheus, and immutable logging.

---

FAQ: High-Value Questions for Engineering Teams

Q1: How do we handle database schema migrations in an immutable architecture? A: Schema changes are treated as events, not mutations. You deploy a new version of the service that writes to a new table (e.g., citizen_v2). The old service continues reading from citizen_v1. A background migration worker reads events from the immutable log and replays them into the new schema. This is known as the Expand-Migrate-Contract pattern. No in-place ALTER TABLE is ever executed.

Q2: Can we use a single K8s cluster for multiple agencies with different compliance requirements? A: Yes, but only with hard multi-tenancy. Each agency gets a dedicated namespace with its own NetworkPolicy, ResourceQuota, and Pod Security Standard. The cluster must run a policy engine (OPA Gatekeeper) that rejects any pod that violates the agency’s specific NZISM control profile. Cross-namespace traffic is blocked by default.

Q3: What happens if the immutable event store (Kafka/S3) becomes unavailable? A: The system enters a graceful degradation mode. Read operations continue from the Aurora read replicas. Write operations are queued in a local, ephemeral buffer (Redis list) with a TTL of 5 minutes. If the event store does not recover within that window, the write is rejected with a 503 Service Unavailable and the citizen is asked to retry. This prevents data loss while maintaining availability.

Q4: How do we ensure the immutable audit log is tamper-proof? A: Each