How to design and build an internal developer platform. Covers golden paths, self-service infrastructure, developer experience metrics, and platform team operating models.
Internal Developer Platform Design Patterns
This page collects the recurring structural patterns that appear when engineering teams build or evolve an Internal Developer Platform (IDP). It matters when you find yourself repeating the same scaffolding steps, when onboarding latency creeps up, or when different teams independently converge on incompatible abstractions. The patterns below are not prescriptions; they are documented solutions to problems you will actually hit, written so you can map them to whatever runtime or workflow your platform uses.
What the reader is trying to do Create a minimal, consistent starting point for a new service so that individual teams don’t have to assemble base images, directory structures, and CI configs from scratch.
Concrete pattern A common approach is a version‑controlled “service scaffold” checked into the platform repo. Teams invoke it with a single command:
git clone https://platform.example.com/servicetemplates/go-service.git my-new-service
cd my-new-service
helm dependency build
helm install . --name my-new-service -n dev
The scaffold includes a Chart.yaml with appVersion set to the platform’s runtime baseline, a values.yaml that references platform‑managed secrets via {{ .Values.secrets.{{ .Release.Name }}.apiKey }}, and a Dockerfile that builds against the organization’s base image tag stored in a constants.yaml file consumed by the CI pipeline.
Sharp edges
constants.yaml is updated but the scaffold isn’t re‑cloned, new deployments may pull an image that doesn’t match the platform’s security baseline. The silent failure is a runtime error that only appears in production logs after a deployment has started. helm install . --name my-new-service will fail if a release with that exact name already exists in the namespace. Teams often forget to append a unique suffix, and the error message from Helm (“release “my-new-service” already exists”) is easy to misread as a network issue. values.yaml placeholder {{ .Values.secrets.{{ .Release.Name }}.apiKey }} only resolves if the platform’s secret store (e.g., a Kubernetes Secret populated by external‑secrets or a Vault Agent) has already created a secret matching that exact key path. If the secret is missing, the pod starts but the application logs “missing required env var API_KEY” — a failure that is invisible until the first request hits the endpoint.What the reader is trying to do Promote a single Helm chart or Docker image through dev, staging, and prod without copy‑pasting configuration or maintaining forked charts.
Concrete pattern Use kustomize patches layered over a base Helm chart. The base chart lives in platform/helm/charts/my-service/, and each environment has a kustomization.yaml:
resources:
- ../helm/charts/my-service
patchesStrategicMerge:
- values-prod.yaml
# environments/dev/values-prod.yaml
replicaCount: 1
image:
tag: dev-1234
# environments/prod/values-prod.yaml
replicaCount: 3
image:
tag: registry.example.com/my-service:1.4.0
Promotion is then a matter of applying the correct kustomize overlay:
cd environments/prod
kustomize edit set image my-service=registry.example.com/my-service:1.5.0
kubectl apply -k .
Sharp edges
values-staging.yaml and values-prod.yaml to the same overlay directory, the strategic merge patch may silently overwrite dev‑specific values with prod defaults, especially when both files define the same top‑level key like replicaCount. The result is a “works in dev, breaks in prod” scenario that requires kustomize build . --verbose to diagnose. image.tag is left unspecified in the base chart, kustomize may resolve to latest if the Docker registry’s tagging policy permits it, causing unpredictable rollouts. The sharp edge is a silent pull of an untagged or stale image, visible only as a mismatched container image ID in kubectl get pods. {{ .Values.secrets.prod.dbPassword }} but the dev overlay only provides {{ .Values.secrets.dev.dbPassword }}, the pod will start with an empty or default password. The failure mode is a connection refused or authentication error that surfaces during the first integration test, not during helm lint.What the reader is trying to do Ensure that any new service registers itself with the platform’s DNS and service‑mesh routing without manual edits to ingress or config maps.
Concrete pattern Leverage a service‑mesh control plane (e.g., Istio, Linkerd) combined with automatic sidecar injection and a wildcard ingress. The platform sets a default ClusterIssuer for TLS and a Gateway resource that matches *.platform.example.com. A service’s Service definition uses metadata.labels that the mesh watches:
apiVersion: v1
kind: Service
metadata:
name: my-service
labels:
app: my-service
platform.example.com/ingress-host: my-service.platform.example.com
spec:
selector:
app: my-service
ports:
- port: 80
targetPort: 8080
The ingress controller is configured with:
# Example using NGINX Ingress Controller with a platform‑managed values file
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"/=/
--set controller.extraArgs=--default-backend-service=default/nginx-backend
Sharp edges
platform.example.com/ingress-host label, the ingress controller may route traffic to the first matching service it encounters, depending on the controller’s sorting algorithm. The result is intermittent 5xx errors for one of the services, and the root cause is not obvious from the ingress resource alone. ClusterIssuer may use an ACME provider with a short‑lived certificate. If the sidecar that handles TLS challenges fails to renew (e.g., due to rate‑limiting from the DNS provider), the ingress returns 502 Bad Gateway until the certificate is manually refreshed. The failure window can span hours, and the error log only shows “tls: failed to renew certificate”. Service is created with the correct label, the mesh may take up to 60 seconds to propagate the routing rule. Deployments that immediately query the service DNS name may hit a “connection reset” error. The pattern to mitigate this is to add a kubectl wait --for=condition=Ready pod/<pod> --timeout=120s step in CI before integration tests run.What the reader is trying to do Add structured logging, metrics, and tracing to every service with a single platform‑level configuration, avoiding per‑team SDK integration bugs.
Concrete pattern Deploy a cluster‑wide OpenTelemetry Collector as a DaemonSet, and configure the Kubernetes kube‑state‑metrics and prometheus‑adapter to scrape standard metrics. Services opt‑in by setting environment variables that the sidecar injector reads:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-service
spec:
template:
spec:
containers:
- name: my-service
env:
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.name=my-service,deployment.environment=prod
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector:4317
image: o11y/base-image:latest
The collector itself is configured via a ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
data:
config.yaml:
receivers:
otlp:
protocols:
grpc:
processors:
batch:
exporters:
logging:
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging]
Sharp edges
OTEL_RESOURCE_ATTRIBUTES includes keys that the backend (e.g., Grafana Cloud or Jaeger) does not recognize, the collector may drop the entire span or log a warning that is easy to miss in noisy pod logs. The sharp edge is a “silent drop” where traces appear in the UI but without the expected service name, leading to mis‑attributed latency. resource detection API, the exporter may fail to serialize certain attributes, resulting in incomplete trace payloads. The failure surface is a “traces missing from dashboard” symptom that only appears after a platform upgrade. cluster-admin‑equivalent permissions to mount the collector’s serving certificates, the OTLP handshake fails with rpc error: code = Unauthenticated. The error surface is a sudden “no traces received” alert, and the diagnostic path requires checking both the service account token and the collector’s TLS configuration.What the reader is trying to do Ensure every configuration change passes through a review pipeline, is tracked in Git, and is applied automatically, reducing “shadow” changes made directly to clusters.
Concrete pattern Use Argo CD or Flux as the canonical source‑of‑truth engine. The platform repo contains applications.yaml objects that declaratively describe the desired state:
# platform/gitops/applications/my-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-service
namespace: argocd
spec:
project: default
source:
repoURL: https://platform.example.com/platform.git
targetRevision: HEAD
path: environments/prod
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
A developer updates the Helm chart version in a feature branch, opens a PR, and Argo CD automatically syncs the application once the CI pipeline (e.g., gitlab-runner or github-actions) passes. The selfHeal flag ensures that if someone manually patches a Deployment outside Git, Argo CD reverts it on the next sync loop.
Sharp edges
kubectl set image to unblock a critical bug, the next Argo CD sync (typically every 3 minutes) will revert the image tag to the Git‑pinned version. The silent failure is an unexpected rollback that can re‑introduce the bug if the Git change isn’t coordinated. The pattern to mitigate this is to require a “hot‑fix branch” that goes through the same PR workflow, even under pressure. prune: true is set and a Helm chart is deleted from the repo, Argo CD will automatically prune the associated Kubernetes resources. Teams sometimes discover that a deleted chart took down a critical sidecar or ingress resource, because the PR diff only shows “- chart/removed” without highlighting the downstream kubectl delete that will occur. The sharp edge is a production outage triggered by a routine repo cleanup. selfHeal: true monitors live cluster state and applies the manifest from Git. If two teams have overlapping resource definitions (e.g., both define a ConfigMap named shared-settings in different namespaces), the heal loop may oscillate, repeatedly overwriting one team’s changes with another’s. The symptom is a ConfigMap that alternates values every few minutes, and the diagnosis requires inspecting argocd application health --refresh-history to see the sync back‑and‑forth.app.kubernetes.io/name) are often overloaded. When two teams define the same label with different values in the same namespace, the Kubernetes API stores only one value (the last one written), and the platform’s own controllers may read the “wrong” value, causing routing or security policy mis‑application. The failure is subtle: kubectl describe pod shows the expected label, but the platform’s admission controller sees the other. values.yaml → values file → --set flag → env var) is documented but frequently misunderstood. A team may pass --set env=prod via CI, env var ENV=staging in the pod spec, and the default values.yaml env: dev. The runtime resolves to dev because env vars in the pod spec take precedence over --set flags in the Helm command, but the team assumes the flag wins. The result is a service that reads the wrong configuration at startup, and the only clue is a mismatch between the intended and actual config dump. metadata.annotations.ingress\.kubernetes\.io/ingress.class: internal, the NGINX controller may ignore it, and the service becomes unreachable from internal pods. The failure mode is a “service exists but cannot be resolved” error that only appears when a developer tries to curl the internal cluster IP from a pod in a different namespace. 401 Unauthorized from the secret store, and the application logs “failed to fetch secret: context deadline exceeded”. The outage is hard to reproduce in staging because token TTLs are usually longer, and the first production manifestation is a cascade of auth failures across micro‑services.These patterns map to the structural decisions any IDP inevitably reaches. They are written generically—using standard CLI tools (kubectl, helm, git, Argo CD, Flux, OpenTelemetry Collector)—so you can translate each example to the runtime, secret‑store, and Git‑ops system your organization actually uses. The “sharp edges” sections capture the failure modes that surface only after a platform has been running at scale, because those are the details that separate a design that looks clean in documentation from one that survives production traffic.
This page was rewritten on 26 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at hello@thegarnetwiki.com and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy