Skip to content

GitOps Architecture

Design for the BRAC POC's GitOps control plane — how the openshift-platform-gitops repo, OpenShift GitOps (ArgoCD), and RHACM cooperate to drive all four OpenShift clusters.

Grounded in: Red Hat Advanced Cluster Management GitOps docs (RHACM 2.13) and Red Hat OpenShift GitOps 1.18 documentation (verified 2026-04-24).

The decisions that shape this

Per DECISION #025 — GitOps-only operational model:

  • All cluster configuration + workload deployment flows through Git
  • oc apply / kubectl apply are reserved for bootstrap-only cases (initial GitOps operator install, break-glass situations)
  • Everything else: commit → push → ArgoCD reconciles → cluster updated

Per DECISION #024 — Red Hat operators only: OpenShift GitOps is the Red Hat build of ArgoCD (not community Argo CD).

Pattern: App-of-Apps + ApplicationSets + Pull Mode

Three Red Hat-recommended patterns combined for the right trade-offs:

1. App-of-Apps (governance)

A single root Application on hub-dc points at the bootstrap/ folder in the Git repo. That folder contains child Applications/ApplicationSets — which in turn manage everything else. One bootstrap command, one Git repo, everything else self-manages.

2. ApplicationSet (multi-cluster scaling)

ApplicationSet dynamically generates individual Application resources from a generator (cluster generator, list, git, matrix). One YAML definition → N target clusters. This is how we deploy the same workload to both spokes.

3. Pull mode via RHACM (scalability + isolation)

Instead of hub-dc's ArgoCD directly pushing manifests to spoke-dc, the hub propagates a ManifestWork that tells the spoke's own ArgoCD to pull from Git. This means:

  • Hub controller load stays low (no N-way sync fan-out)
  • Spoke is the source of truth for its own state (fits the DR model)
  • Spoke continues reconciling even if hub is briefly unreachable
  • Status aggregates back via MulticlusterApplicationSetReport

Status (as of 2026-04-24): pull model via ApplicationSet + ACM annotations is GA in RHACM 2.13. The GitOps add-on (auto-installing the GitOps operator on spokes) is Technology Preview — we'll install manually via Subscription CRs in the bootstrap repo to avoid TP risk.

Hub-spoke wiring (RHACM integration)

Three CRs connect ArgoCD on hub-dc to the managed clusters:

flowchart LR
    CS["ManagedClusterSet<br/>'brac-poc-clusters'"] -->|binds to namespace| CSB["ManagedClusterSetBinding<br/>in openshift-gitops"]
    CSB --> PL["Placement<br/>selects managed clusters"]
    PL --> GOC["GitOpsCluster<br/>registers selected clusters<br/>to ArgoCD's cluster list"]
    GOC --> AS["ApplicationSet<br/>uses clusterDecisionResource<br/>generator → fans out"]
    AS -->|ManifestWork<br/>pull mode| spoke1["spoke-dc ArgoCD"]
    AS -->|ManifestWork<br/>pull mode| spoke2["spoke-dr ArgoCD"]
    AS -->|direct sync| hubDR["hub-dr ArgoCD<br/>(for hub mgmt workloads)"]

Example GitOpsCluster (lives in bootstrap/acm-integration/):

```yaml apiVersion: cluster.open-cluster-management.io/v1beta2 kind: ManagedClusterSet metadata: name: brac-poc-clusters spec: clusterSelector: selectorType: ExclusiveClusterSetLabel

apiVersion: cluster.open-cluster-management.io/v1beta2 kind: ManagedClusterSetBinding metadata: name: brac-poc-clusters namespace: openshift-gitops spec: clusterSet: brac-poc-clusters

apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: brac-poc-placement namespace: openshift-gitops spec: clusterSets: - brac-poc-clusters tolerations: - key: cluster.open-cluster-management.io/unreachable operator: Exists

apiVersion: apps.open-cluster-management.io/v1beta1 kind: GitOpsCluster metadata: name: brac-poc-gitops namespace: openshift-gitops spec: argoServer: cluster: local-cluster argoNamespace: openshift-gitops placementRef: kind: Placement apiVersion: cluster.open-cluster-management.io/v1beta1 name: brac-poc-placement ```

Then each spoke is labeled with its set membership + its role:

hub-dr: cluster.open-cluster-management.io/clusterSet=brac-poc-clusters brac.poc/role=hub brac.poc/site=dr spoke-dc: cluster.open-cluster-management.io/clusterSet=brac-poc-clusters brac.poc/role=spoke brac.poc/site=dc spoke-dr: cluster.open-cluster-management.io/clusterSet=brac-poc-clusters brac.poc/role=spoke brac.poc/site=dr

These labels drive which ApplicationSet targets which cluster.

Repo structure — openshift-platform-gitops

Host: GitLab HA deployed on spoke-dc (POC Issue #4). Repo visibility: internal to the BRAC team; mirrored to comptech-lab/openshift-platform-gitops on GitHub for review.

openshift-platform-gitops/ ├── README.md ├── bootstrap/ │ ├── 00-namespace-rbac.yaml # openshift-gitops ns + cluster-admin binding │ ├── 01-argocd-instance.yaml # ArgoCD CR (Red Hat GitOps operator CRD) │ ├── 02-root-application.yaml # Points at bootstrap/ itself (self-managing) │ └── acm-integration/ │ ├── managed-cluster-set.yaml │ ├── managed-cluster-set-binding.yaml │ ├── placement-all.yaml │ └── gitops-cluster.yaml │ ├── applicationsets/ │ ├── hub-platform.yaml # ACM, ACS, GitOps, Compliance, COO → hubs │ ├── hub-keycloak.yaml # Keycloak + CNPG Postgres → hubs │ ├── spoke-platform.yaml # Compliance Op, PSA, defaults → spokes │ ├── spoke-workloads.yaml # 9 POC components → spokes │ └── all-clusters-baseline.yaml # sysctl, audit rules, allowed-registries │ ├── components/ # Kustomize bases (reused across ApplicationSets) │ ├── _base/ │ │ ├── acm/ │ │ ├── acs-central/ │ │ ├── openshift-gitops/ │ │ ├── compliance-operator/ │ │ ├── cluster-observability-operator/ │ │ ├── keycloak-rhbk/ │ │ └── keycloak-realm-brac-poc/ │ │ │ └── workloads/ │ ├── kafka/ │ ├── redis-sentinel/ │ ├── otel-signoz/ │ ├── wso2-apim/ │ ├── wso2-is/ │ ├── nginx-middleware/ │ ├── open-liberty-sample/ │ ├── trivy/ │ ├── jboss-domain/ │ └── sample-app-otel/ │ ├── overlays/ # Per-cluster customizations │ ├── hub-dc/ │ ├── hub-dr/ │ ├── spoke-dc/ │ └── spoke-dr/ │ ├── policies/ # ACM Policy CRs (compliance baseline) │ ├── etcd-encryption.yaml │ ├── audit-profile.yaml │ ├── allowed-registries.yaml │ ├── default-deny-network-policy.yaml │ ├── pod-security-restricted.yaml │ └── placement-bindings.yaml │ └── projects/ # ArgoCD AppProjects for RBAC ├── hub-project.yaml ├── spoke-project.yaml └── platform-baseline.yaml

Kustomize layering

flowchart LR
    Base["components/_base/acm"] --> OV["overlays/hub-dc"]
    Base --> OV2["overlays/hub-dr"]
    OV --> App["ApplicationSet (hub-platform.yaml)<br/>targets clusters labeled role=hub"]
    OV2 --> App
    App --> ArgoCD["hub-dc ArgoCD"]
    ArgoCD -->|ManifestWork| HubDC[hub-dc cluster]
    ArgoCD -->|ManifestWork| HubDR[hub-dr cluster]

Each overlay is a small kustomization.yaml that patches the base with cluster-specific values (hostnames, replica counts, resource sizing).

Example: deploying RHACM via GitOps

File: applicationsets/hub-platform.yaml

yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: hub-platform namespace: openshift-gitops spec: generators: - clusterDecisionResource: configMapRef: ocm-placement-generator labelSelector: matchLabels: cluster.open-cluster-management.io/placement: brac-poc-placement matchExpressions: - key: brac.poc/role operator: In values: [hub] template: metadata: name: '{{name}}-hub-platform' annotations: apps.open-cluster-management.io/ocm-managed-cluster: '{{name}}' apps.open-cluster-management.io/ocm-managed-cluster-app-namespace: openshift-gitops argocd.argoproj.io/skip-reconcile: "true" labels: apps.open-cluster-management.io/pull-to-ocm-managed-cluster: "true" spec: project: platform-baseline source: repoURL: https://gitlab.apps.brac-poc.comptech-lab.com/brac-poc/openshift-platform-gitops.git targetRevision: main path: overlays/{{name}}/hub-platform destination: server: https://kubernetes.default.svc namespace: openshift-gitops syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true

Critical annotations for pull-mode:

Annotation Purpose
apps.open-cluster-management.io/ocm-managed-cluster: '{{name}}' Identifies which cluster gets this app
argocd.argoproj.io/skip-reconcile: "true" Hub ArgoCD does NOT sync this Application itself
apps.open-cluster-management.io/pull-to-ocm-managed-cluster: "true" Label → ACM propagates via ManifestWork

Status comes back via the auto-generated MulticlusterApplicationSetReport in the same namespace.

Compliance enforcement: ACM Policy vs ArgoCD

Both tools + different jobs. Per Red Hat guidance:

Tool Job Example
ArgoCD ApplicationSet Continuous delivery — "install this Deployment on those clusters" RHACM operator, Kafka StatefulSet, sample apps
ACM Policy Governance — "these clusters MUST have this config; auto-remediate drift" restricted PSA on every namespace · audit profile = WriteRequestBodies · allowed registries list · default-deny NetworkPolicy · FIPS mode verification

Why both? Policy enforces state; ApplicationSet deploys apps. A cluster can be policy-compliant (baseline) without having workloads, or have workloads without being compliant. We want both.

Example policy (policies/audit-profile.yaml):

```yaml apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: require-audit-writerequestbodies namespace: openshift-gitops annotations: policy.open-cluster-management.io/standards: PCI-DSS policy.open-cluster-management.io/categories: "AC Access Control, AU Audit Accountability" policy.open-cluster-management.io/controls: "PCI-DSS 10.2, 10.3" spec: remediationAction: enforce policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: audit-writerequestbodies spec: object-templates: - complianceType: musthave objectDefinition: apiVersion: config.openshift.io/v1 kind: APIServer metadata: name: cluster spec: audit: profile: WriteRequestBodies

apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: require-audit-writerequestbodies namespace: openshift-gitops placementRef: apiGroup: cluster.open-cluster-management.io kind: Placement name: brac-poc-placement subjects: - apiGroup: policy.open-cluster-management.io kind: Policy name: require-audit-writerequestbodies ```

All 4 clusters get the WriteRequestBodies audit profile automatically; any drift is auto-remediated.

Bootstrap sequence — the only oc commands we run

Step 1 (manual, once per hub): install the OpenShift GitOps operator on hub-dc and hub-dr via oc apply on these three files:

```yaml

00-namespace.yaml

apiVersion: v1 kind: Namespace metadata: name: openshift-gitops-operator

01-operatorgroup.yaml

apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-gitops-operator namespace: openshift-gitops-operator

02-subscription.yaml

apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-gitops-operator namespace: openshift-gitops-operator spec: channel: gitops-1.18 # ← verify in Red Hat catalog before applying installPlanApproval: Automatic name: openshift-gitops-operator source: redhat-operators sourceNamespace: openshift-marketplace ```

Wait for the openshift-gitops namespace to appear + the ArgoCD Route to be ready (~2 min).

Step 2 (manual, once): apply the root Application that points at the repo:

```yaml

02-root-application.yaml

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: bootstrap-root namespace: openshift-gitops finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://gitlab.apps.brac-poc.comptech-lab.com/brac-poc/openshift-platform-gitops.git path: bootstrap/ targetRevision: main destination: server: https://kubernetes.default.svc namespace: openshift-gitops syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true ```

From this point on, nothing else uses oc. The root Application syncs bootstrap/ which contains the ACM integration CRs + ApplicationSets, which in turn deploy every operator, every workload, and every policy.

Operations — how to change anything

Want to… Do
Deploy a new workload Add components/workloads/<name>/ + reference in an ApplicationSet; commit + push
Change Kafka replicas on spoke-dc only Edit overlays/spoke-dc/kafka/kustomization.yaml; commit + push
Enforce a new compliance control cluster-wide Add a Policy in policies/; commit + push
Roll out a change to all spokes Edit the base in components/_base/…; commit + push
Urgently break-glass a cluster oc is acceptable — then reflect the change back in Git within 24h
Drift an ArgoCD-managed resource manually Don't — ArgoCD selfHeal: true will revert it

No one has write access to clusters without going through Git + PR review. Audit trail = git log.

PR workflow

flowchart LR
    Dev["Engineer"] -->|branch: feat/xyz| Change["Edit YAML<br/>in openshift-platform-gitops"]
    Change -->|git push| PR["GitLab MR"]
    PR -->|CI: kubeconform +<br/>policy lint +<br/>yamllint| Review["Human review"]
    Review -->|approve + merge| Main["main branch"]
    Main -->|webhook / poll| ArgoCD["hub-dc ArgoCD"]
    ArgoCD -->|"&nbsp;ManifestWork<br/>(pull mode)"| Spoke["target cluster(s)"]
    Spoke -->|status| Report["MulticlusterAppSetReport"]
    Report --> ACM["RHACM UI on hub-dc"]

The GitLab CI validates every merge request before it can be merged:

Check Tool
YAML syntax yamllint
Kubernetes schema validity kubeconform
Kustomize builds cleanly kustomize build
ACM Policy structure policy-generator-plugin --lint
ArgoCD AppProject ownership custom check — App destinations must match the AppProject's allowlist
Secret leak scan gitleaks

What we lose vs full cluster-scoped freedom

To set expectations honestly:

  • Urgent hotfixes still require a git commit. Mitigation: MR auto-merge for hotfix/* branches after CI passes. Still auditable.
  • Experimentation requires branching, not live-edit on cluster. Mitigation: dev namespace on spoke-dc with relaxed policy; experiments there don't need PRs.
  • Initial debugging curve — team members who are used to oc apply need a week to adjust. Mitigation: paired sessions during Phase 1.

What we gain: every cluster change is logged in git log with who, when, why. Full audit trail. Reproducibility. DR is "deploy the same Git state to new clusters". This is exactly what BRAC's compliance audit would want to see.

References

Created: 2026-04-24 · Owner: DevOps + Platform Leads · Status: Ready for execution