GitOps: Git as the Source of Truth for Deployment
Declarative desired state in Git plus a reconciler that continuously converges the running system toward it. Covers pull versus push delivery, repository topology, drift, promotion, secrets, and where the model stops working.
GitOps is an operating model in which the desired state of a system is declared in Git, and a controller running next to that system continuously compares reality against the declaration and corrects the difference. The deployment action is not "run a script against the cluster"; it is "merge a commit", after which an autonomous reconciler does the work. Git stops being a place where deployment code lives and becomes the actual control input.
What GitOps Actually Is
The idea only works on systems that can be described declaratively and reconciled repeatedly. That is why it emerged around Kubernetes, whose entire API is built on controllers that observe desired state, diff it against observed state, and act — see Kubernetes for that control-loop model. GitOps extends the same loop outward so that the desired state lives in version control instead of in whatever a human last applied.
The Four Properties
1. Declarative
The system is described by what it should be, not by the steps to get there.
2. Versioned and immutable
The description lives in Git: every change is a commit, signed and reviewable,
and every prior state is recoverable by checking out an earlier commit.
3. Pulled automatically
An agent inside the environment fetches the desired state. Nobody pushes
into the environment from outside with standing credentials.
4. Continuously reconciled
The agent does not run once at deploy time. It runs forever, detecting and
correcting drift whether the drift came from a human, a script, or a bug.Property four is the one that distinguishes GitOps from "we keep our manifests in Git". Storing YAML in a repository and applying it from a CI job gives you versioning and review, but the moment someone edits the live system by hand, the repository becomes a historical document rather than a description of what is running. Continuous reconciliation is what keeps the claim true.
Pull-Based vs Push-Based Delivery
In the push model, a CI job holds credentials to the target environment and applies changes to it. In the pull model, a controller inside the environment reads the repository and applies changes to itself. Argo CD and Flux are the two widely used pull-mode reconcilers for Kubernetes.
| Concern | Push (CI applies) | Pull (in-cluster agent) |
|---|---|---|
| Credentials | CI holds standing write access to production | Cluster holds read access to the repo; no inbound path |
| Network | CI must reach the cluster API | Cluster reaches out; no exposed control plane needed |
| Drift | Detected only on the next pipeline run, if at all | Detected continuously by the reconciler |
| Blast radius of a CI compromise | Every environment CI can reach | The repository, which is reviewable and auditable |
| Feedback to the developer | Immediate in the pipeline | Indirect; needs health status surfaced back |
The last row is the real cost of pull mode and the one teams underestimate. A merged commit no longer means "deployed"; it means "requested". You need the reconciler's sync status and health assessment surfaced somewhere a developer will look, or people will assume the merge finished the job and discover otherwise during an incident.
Repository Topology
Almost every GitOps setup separates application source from deployment configuration. The application repository builds and publishes an artifact; a configuration repository declares which artifact version runs where. Keeping them separate avoids a loop in which a deployment commit retriggers the build that produced it, and it lets deployment state change without touching application history.
Modelling Environments
Directory-per-environment (usually preferred)
clusters/
staging/ kustomization + patches for staging
production/ kustomization + patches for production
base/ the shared definition
→ Differences between environments are visible as a diff between directories.
→ Promotion is an explicit edit, not a merge of accumulated history.
Branch-per-environment (usually regretted)
main ──▶ staging ──▶ production (cherry-picks and merges between branches)
→ Environments drift apart through merge order, not through intent.
→ "What is different about production?" becomes hard to answer.
→ Hotfixes applied to one branch go missing from another.The directory model has a related advantage: the rendered-manifest pattern, where a pipeline renders templates into plain, fully-resolved manifests and commits those. What the reconciler applies is then exactly what a reviewer read, with no templating engine standing between the diff and the effect.
Promotion Between Environments
Promotion in GitOps means changing a version reference in the configuration for the next environment. Three common mechanisms, in increasing order of control: an automated image updater that watches a registry and commits new tags; a pipeline that opens a pull request against the target environment's directory after upstream checks pass; or a human edit. The middle option is usually the right default, because it makes promotion reviewable and auditable while keeping it a single click.
Whichever mechanism you pick, pin artifacts by immutable digest rather than by a mutable tag. A floating tag breaks the central promise of the model: that the repository describes what is running. Two clusters reconciling the same floating tag at different times are running different code while claiming to be identical.
Secrets in a Public-by-Default Model
The one thing that cannot simply be committed is a secret. Three families of solution are in common use, and they differ in where the trust sits:
- Encrypted in the repository. Values are encrypted before commit and decrypted by a controller holding the key. The ciphertext is versioned with everything else; losing the decryption key means losing the ability to restore the environment from Git.
- External secret references. The repository holds only a pointer to a secret in a dedicated secret manager, and an operator resolves it at apply time. Git stays free of ciphertext, at the cost of a runtime dependency on the secret store.
- Injected at runtime. The workload fetches its own credentials using a workload identity. Nothing secret passes through the delivery path at all, which is the strongest option and the most invasive to adopt.
Whichever you choose, treat the decryption key or the secret-store trust relationship as the crown jewel of the whole system, and make sure its recovery procedure is written down somewhere that is not itself encrypted with it. See secrets management for the broader pattern.
Failure Modes
| Failure mode | Mechanism | Correction |
|---|---|---|
| Controller fights a controller | An autoscaler or mutating webhook owns a field the repo also declares | Exclude that field from reconciliation or stop declaring it |
| Silent non-reconciliation | The agent is unhealthy, rate-limited, or stuck; nothing alerts | Alert on sync age and on the agent itself, not just on workloads |
| Merged but not applied | Developers read merge as deployment | Surface sync and health status in the same place as the pull request |
| Emergency hand-edit | An operator patches the live system during an incident; the reconciler reverts it | Have an explicit break-glass mode, then reconcile the fix back into Git |
| Destructive pruning | Pruning removes resources the repo no longer mentions, including ones nobody meant to remove | Scope pruning narrowly; never let one reconciler own resources it did not create |
| Bootstrap circularity | The reconciler's own definition lives in the repo it reconciles | Keep a minimal, documented bootstrap path outside the loop |
| Repository as a queue | Automated commits at high frequency; history becomes unreadable | Batch automated updates; keep human intent separable from machine noise |
When GitOps Does Not Fit
GitOps requires a target that can be declared and converged. Work that is inherently imperative and one-shot — a data backfill, a schema migration with an ordering constraint, a manual failover, a cache rebuild — does not become declarative because you put it in a repository. Those belong in a job runner with its own audit trail, triggered deliberately.
It also adds real operating cost: another controller to run and upgrade, a repository structure to maintain, and an indirection between merging and shipping that a small team on a single environment may simply not need. If the environment is one application on one platform with a working CI/CD pipeline, the honest answer is often that push-mode delivery is enough, and that infrastructure as code with reviewed plans already gives you the versioning and audit benefits you were after.