Riddam Jain — Engineering

Self-Service With Guardrails: Repositories and Pipelines as Data

Tue, 04 Aug 2026 00:00:00 GMT

There's a stage every platform team hits. You've built something good, teams want to use it, and now you spend your week creating repositories, adding people to groups, wiring webhooks, and setting up pipelines. You've become a ticket queue with extra steps.

The obvious fix is to give teams access and let them do it themselves. That works right up until you have a thousand-odd repositories, no two configured the same way, and no way to answer "which of these have branch protection?" without opening a thousand browser tabs.

Self-service and consistency look like opposites. They aren't — but getting both means the thing teams edit has to be data, not a UI.

This is the layer above the infrastructure. Rebuilding the platform itself — agents, scaling, upgrades — is a separate story, and worth doing first: self-service on top of a platform you're afraid to touch just distributes the fear.

Two registries, one source of truth

Everything starts from declarations in version control rather than state in a web interface:

A pipeline registry — one entry per repository that has a pipeline, naming the owning teams, the hosting platform it deploys to, and the grouping it belongs to.
A repository registry — the repositories themselves, their teams, their webhooks, their branch protection.

A team wanting a pipeline opens a pull request adding a few lines. That's the whole interface. The same reasoning I've written about for configuration as data applies to organizational structure: the moment your source of truth is a UI, you have no history, no review, and no way to ask a question across the whole estate.

What matters is that the registry is the source of truth and not a description of one. Which means something has to continuously make reality match it — and something has to refuse entries that would break it.

The scale this operates at is roughly 1,400 projects and around 2,400 build configurations. At that size, nobody is going to notice a hand-made exception, which is the real argument for the registry: not elegance, but the fact that no human review process survives four figures.

Validate before you touch anything

The validation step runs first, and it exits non-zero on the first problem. Nothing reaches the CI server until every check passes.

Some checks are the obvious schema ones — every entry validates against a JSON Schema, and duplicate entries are rejected. Two others are worth calling out, because they're the ones I'd port to any similar system.

Entries must be in alphabetical order. This sounds like fussiness and isn't. An unordered append-only list means every team adding a pipeline in the same week touches the same last line, and they all conflict. Enforced ordering distributes the edits across the file, so a class of merge conflict simply stops occurring. It also makes review trivial — a diff is one line in a predictable place rather than a reordering nobody can read.

The owning team must already exist. A pipeline entry names the teams that own it, and validation checks those teams exist in the CI server before accepting the entry. Without this, the ownership field decays into free text — someone types a team name slightly wrong, nothing complains, and eighteen months later nobody can work out who owns a failing build. The check also tells the author exactly how to fix it and lists valid values, because a guardrail that blocks you without telling you what to do next is just an obstacle.

The rest are conditional-shape rules: a mobile entry must use the mobile team field and not the general grouping field; a package entry must declare a package type. These are the rules that would otherwise live in a wiki page nobody reads.

Validation is where a convention becomes a guarantee. Anything you merely document, you don't have.

Reconcile, don't create

The naive version of this tooling creates things. The useful version makes reality match a declaration, which is a different job.

For pipelines, the reconciliation is a set difference. Derive a deterministic identifier for each declared entry, list what actually exists on the server, and subtract: what's declared but missing gets created, and what exists in both gets synchronized. That's it — and because the identifier is derived from the declaration rather than stored, there's no mapping table to drift out of sync.

For repositories it's more involved, because there are several independent concerns and each one needs its own answer to "has this drifted?" — team membership, webhooks, branch protection. Each concern gets a drift check before its sync, so the tooling can report what changed rather than blindly reapplying everything.

That distinction matters more than it sounds. Blind reapplication is safe only if every write is genuinely idempotent, and it makes your logs useless — every run reports doing everything. Checking first means an ordinary run is quiet and an interesting run is loud.

A drift check is the difference between infrastructure as code and infrastructure as a strongly worded suggestion.

Dry run is a feature, not a flag

Every operation supports a dry run, and it's the default rather than an afterthought.

This is easy to dismiss as a nicety. It isn't — it's the reason anyone lets you run the thing at all. Tooling that reconciles an entire organization's repositories has a genuinely frightening blast radius: it can close pull requests, remove people from teams, and rewrite branch protection. Nobody should approve that on the promise that the logic is correct.

With a dry run, the change becomes reviewable before it's real. You get the same output you'd get from an apply, with every mutation described and none performed. It also makes the tooling testable in production, which is where the surprising inputs live.

The property worth designing for is that dry run isn't a separate code path. If it is, it will diverge from the real one and lie to you. It should be the same path with the writes suppressed at the boundary.

Compliance as a report before it's a gate

There's a strong temptation, once you can check things automatically, to start blocking on them. That's usually the wrong first move.

The compliance checks run across every declared repository and answer a fixed set of questions: does it have a license file, does the default branch have protection, does it require at least one reviewer, has its team configuration drifted from the declaration, have its webhooks drifted, does it have unresolved critical dependency alerts, and has anyone committed to it in the last year. The output is a table for humans and JSON for anything else.

Reporting first does two things. It tells you how bad the situation actually is, which is almost never what you assumed. And it gives teams a chance to fix things before non-compliance starts failing their builds — which is the difference between a platform team that raised the bar and a platform team that broke everyone's Tuesday.

Once the report is mostly green, turning individual checks into gates is uncontroversial. Turning them into gates first is how platform tooling gets a reputation.

The enforcement did arrive, and where it landed is the part I'd repeat: inside the pipelines teams already own, rather than in a central gatekeeper. Policy-as-code for the rules that have to hold organization-wide, template linting for infrastructure definitions, language-specific linters and code-quality checks, and test pipelines that actually gate a merge.

The same check feels like part of your build when it runs in your pipeline, and like an audit when it runs in someone else's cron job. Identical logic, opposite reception.

Retirement is part of the lifecycle

Almost all platform tooling creates things. Very little of it retires them, which is why every organization accumulates repositories nobody has touched in three years and nobody is willing to delete.

So the same tooling detects staleness — no commits for a configurable window, excluding already-archived repositories — and sorts by how long it's been, with a recommendation attached based on whether anything is still open against it. A repository with no commits in two years and no open work is a different case from one with no commits and a stack of open pull requests nobody closed.

And when something is retired, there's an actual sequence rather than a single archive call: close the open issues, close the open pull requests, remove branch protection, and leave a note in the README explaining what happened and where the work went. Archiving without that leaves a repository that looks abandoned rather than deliberately closed — and the person who finds it in two years can't tell the difference.

Every tool that creates resources should know how to retire them. Otherwise you've automated accumulation.

The unglamorous parts that make it usable

Three things that no design document ever includes and every real system needs.

An audit log. Every action records who ran it, what it targeted, what changed, whether it was a dry run, and whether it succeeded — as structured JSON, with the notable actions also posted to a chat channel. When someone asks why a webhook changed last Thursday, you want an answer that isn't archaeology. Recording the dry-run flag matters too, so a rehearsal is never mistaken for the real thing.

Rate-limit awareness. Anything that walks four figures of repositories through a provider API will hit rate limits. Knowing your remaining budget before starting a long reconciliation is the difference between a clean run and a job that dies two-thirds through having done two-thirds of the work.

Account hygiene. Enforcing multi-factor authentication and pruning outside collaborators are the least interesting features here and probably the highest-value per line of code. They're also the checks nobody performs manually, because doing so means reading a members list — which is exactly the sort of task that only ever happens if something scheduled does it.

What this actually buys

Teams change a few lines of YAML in a pull request and get a working pipeline, and the platform team isn't in that path at all. That's the visible win, and it's the one people ask for.

The one worth more is that the estate becomes queryable. "Which repositories lack branch protection?" and "who owns this pipeline?" and "what has nobody touched in a year?" go from an afternoon of clicking to a command. You can't improve a property you can't measure across the whole estate, and a UI-shaped source of truth means you can never measure anything.

If I were starting this again, the order I'd build it in is: the registry, then validation, then dry run, then reconciliation, then reporting, and only then any gating. Every step in that order makes the next one safe. Done in the other direction, you get a tool that changes things nobody asked it to change — and once you've done that once, you'll never be allowed to run it again.

Rebuilding CI/CD Without Changing Platforms

Mon, 03 Aug 2026 00:00:00 GMT

When a CI/CD platform becomes painful, the instinct is to shop for a new one. Sometimes that's right. Often the pain has nothing to do with the product and everything to do with what has accumulated around it: build agents configured by hand years ago, pipelines assembled by clicking through a UI, a server nobody dares upgrade.

Migrating platforms doesn't fix any of that. It relocates it, and adds a migration to the bill.

I've written about the decision not to migrate and how you make that case credibly. This is the other half — what the rebuild actually consisted of, and which parts of it I'd do the same way again.

This post is about hosting the platform: agents, scaling, upgrades, and the things that quietly rot. The pipelines-as-code half of that list — how teams declare a pipeline instead of clicking one together — is its own post.

For a sense of scale, what this now carries is roughly 1,400 projects and around 2,400 build configurations, with a few dozen more archived. That's worth holding in mind through the rest of this, because most of the decisions below only start to matter somewhere in the high hundreds.

Two agent tiers, and why

The instinct is one kind of build agent, uniformly configured. That breaks on the first job that needs Docker inside Docker.

Nested container workloads need privileges a managed container platform won't grant you, and shouldn't. So there are two tiers, and the split is a rule rather than a preference:

VM agents for anything needing Docker-in-Docker, privileged operations, or a full machine. They cost more and start slower.
Container agents for everything else, which is the large majority of builds.

The container tier runs framework-specific images — one for Python, one for TypeScript, one for .NET — rather than a single image carrying every toolchain any team might want. A fat image is appealing for about a month, until the pull time starts dominating short builds and every team's dependency upgrade becomes everyone's problem. Narrow images fail independently, which is the property you want.

The VM tier is deliberately heterogeneous too: general-purpose x86 for most work, ARM instances where the workload benefits, compute-optimized shapes for the heavy jobs, and Mac metal for mobile builds, because iOS builds leave you no choice.

A single build image is a shared-fate decision disguised as a simplification.

Scale on the queue, not on CPU

Build capacity is bursty in a way that defeats ordinary autoscaling. Nothing for two hours, then dozens of jobs land at once because a release branch merged.

CPU utilization is the wrong signal here — it's a lagging proxy for demand. The build queue is the demand signal: it knows exactly how much work is waiting and what kind of agent each job needs. So agents provision in response to queue depth, and scale back down after an idle period.

Two things follow from this that are worth stating plainly. Idle cost approaches zero, because at three in the morning there are no agents running at all. And the scale-down timer is a real tuning decision, not a default to accept: too aggressive and you pay the cold-start cost on every commit during working hours; too lax and you're paying for idle agents all afternoon. An hour of idle tolerance turned out to be a reasonable balance, but it's the kind of number worth revisiting against your own commit patterns rather than inheriting from a blog post.

Build capacity also runs on spot, and which workload gets it is the decision that matters. Pull request inspection is where it points: that generates by far the deepest queue, every run is short, and every run is safe to repeat.

Which makes interruption handling much less interesting than it sounds, and deliberately so. When spot capacity gets reclaimed, the container tier absorbs the work — runners spin up there and the queue drains a little slower rather than stalling. That's a better answer than clever retry logic, because it requires no coordination and no new moving parts: the fallback is a capacity type that was already running everything else.

The cheapest way to handle a failure mode is to arrange for something else to already be able to do the work.

Don't let the CI system deploy itself

This is the piece I'd argue hardest for, and the one I see skipped most often.

The platform's infrastructure — the server, the agent fleets, the networking, the roles — is defined as code. The question is what applies that code. The tempting answer is the CI platform itself: it's right there, teams already know it, and it can reach everything.

Don't. If the system that rebuilds your pipeline system is your pipeline system, you have a circular dependency. It costs you nothing on an ordinary day and everything on the day the server won't start, which is precisely the day you need to deploy a change to it.

So that pipeline lives somewhere else entirely — in our case GitHub Actions, deliberately not the platform under management:

A pull request runs a plan against every environment and posts the diff. The diff is the review artifact; you approve a described change, not an intention.
Merging applies it, environment by environment, with concurrency pinned to one so two environments can never converge on the same state simultaneously.
Environments go in order, test first. The blast radius of a mistake is a test environment, and you learn about it before production.

The test for whether you've actually broken the dependency is uncomfortable and worth asking out loud: if the CI server were gone right now, could I rebuild it? If the answer routes through the CI server, you haven't broken anything — you've just written the circular dependency down.

There's a subtlety worth naming. Not everything that touches the platform has to avoid it. A reconciliation job that calls the server's API to keep configuration current legitimately depends on the server being up, because it has nothing to do when the server is down. Bootstrap and steady-state are different problems, and only bootstrap needs the separation.

The base image will drift unless something stops it

This is the section I'd most like people to take seriously, because I learned it the expensive way.

I once estimated two weeks to move a fleet from Amazon Linux 2 to Amazon Linux 2023 and spent five. The operating system migration was never the problem. The problem was that the applications running on it had been quietly held together by an old base image for years — runtimes several versions behind, OpenSSL moving from the 1.0.2 and 1.1.1 era to 3.0, Python 2 gone entirely. I hadn't estimated a migration. I'd estimated an unmeasured maintenance backlog.

Drift isn't an event. It's the default state of any long-lived image that nothing actively updates. So something has to actively update it:

A golden image is built by a pipeline and its identifier published to a parameter store, so there is exactly one answer to "what should agents be running?"
A scheduled reconciler reads that parameter, resolves the image and its snapshot, and rewrites the agent fleet configuration to match. Agents replaced after that point come up on the current image.
The schedule is staggered across environments — test on one day, then the others through the week. A bad image surfaces in test before it reaches anything that matters.

The staggering is the part that's easy to skip and shouldn't be. Refreshing everything at once converts a routine hygiene job into a simultaneous fleet-wide outage the one time the new image is broken.

Nothing about this is clever. It's a scheduled job that keeps a number in sync with another number. That's exactly why it works — and why the five-week version of my week-two estimate never has to happen again.

Scan the images, and mean it

The VM side of the fleet has drift handled. The container side has a different problem: an agent image is a supply chain, and it's one you're pulling into a system that holds deployment credentials.

Container agent images come out of a managed image-build pipeline into a registry with scanning enabled, and they're exercised in a test phase after the build rather than trusted on the strength of having built successfully. A green build says the Dockerfile is valid. It says nothing about whether the result works or what it now contains.

The part that matters is what happens on a critical finding: the image is removed before it is used for any deployment. Not flagged for triage, not added to a backlog with a due date — deleted, under a policy that doesn't negotiate about it.

A scanner whose findings you triage at leisure is a compliance artifact. A scanner that can delete an artifact before it ships is a control.

That distinction is worth being honest about, because scanning is one of the easiest things to install and one of the easiest to render meaningless. Almost everyone has scanning. Far fewer have a rule that stops a flagged image from being used, and the rule is the entire value.

Make the server disposable

The server was the thing nobody wanted to touch, which is how you end up years behind on versions.

The fix isn't making the server more robust. It's making it disposable, which means getting the state off it:

The database moves to a managed cluster — more than one instance, encrypted, with a real backup retention window and a defined maintenance window.
The data directory moves to shared encrypted network storage, so it survives the instance entirely.
Backups go to object storage on top of that, because a managed cluster protects you from hardware failure, not from a bad decision.

With state elsewhere, upgrading stops being surgery on a precious box and becomes replacing an instance. It runs in an autoscaling group behind a load balancer, replaced one node at a time, and — the important detail — the rollout waits on a success signal from the new instance and requires it before proceeding. A server that fails to come up blocks its own rollout rather than taking the platform down with it. The group also always tracks the newest launch template version, so an image change propagates on the next replacement instead of needing anyone to remember.

One honest note, because this gets oversold: that is not a highly available server. It's a single node, and there's a window during replacement when it isn't serving. The availability lives in the state tier — the database cluster and the shared storage — while the compute is deliberately cheap to throw away. For an internal build platform that's the right trade, and it's worth being precise about which property you actually bought.

Highly available and disposable are different goals. Disposable is usually the one that makes upgrades boring, and boring upgrades are how you stop drifting.

Credentials that expire, scoped by environment

The legacy setup's worst habit was credentials living in pipeline configuration. Two changes fixed the shape of it.

Credentials are issued, not stored. A pipeline declares that it needs cloud access and receives a short-lived, scoped credential at build time. Nothing durable sits in the pipeline definition, so nothing leaks out of it in a screenshot or a config export.

The agent pool is the boundary. Each environment has its own agent pool, and the pool a pipeline requests determines which environment's credentials it can obtain at all. A pipeline that asks for a test agent cannot get production credentials, because the association is structural rather than a naming convention someone has to respect. Environment isolation you can enforce beats environment isolation you have to remember — and it fails closed, which convention never does.

The direction to keep heading is federated identity, where the CI system proves who it is to the cloud provider and no long-lived key exists anywhere in the chain. Any credential with no expiry is a credential you will eventually find in a place you didn't put it.

What's still unfinished

One honest edge, and it's the one with our name on it.

Image builds aren't self-service yet. Agent images are built centrally, so a team wanting a new toolchain files a request and waits for us. The scanning and deletion policy above is exactly the sort of guardrail that makes handing that over safe — teams build their own images against a policy rather than asking permission, which is the pattern that already works for pipelines and repositories. The guardrail exists. The self-service hasn't been built on top of it yet.

What I'd keep

If I were doing this again on a different platform, four things would come along unchanged: two agent tiers with a hard rule about which is which, scaling on queue depth rather than a proxy metric, the platform's own deployment living outside the platform, and something scheduled that keeps the base image current whether or not anyone is paying attention.

None of that is specific to a product. All of it is what the legacy setup was missing — which is why replacing the product would have solved nothing.

Safe Rollouts for Stateful Cloud Infrastructure

Fri, 24 Jul 2026 00:00:00 GMT

There is a particular flavor of dread that only comes from stateful infrastructure. With a stateless service, a bad deploy is a shrug — roll back, redeploy, move on. With a database cluster that took hours to provision and holds the only copy of something important, a bad change is not an inconvenience. It is an incident with a recovery time measured in hours and a blast radius measured in trust.

So the discipline changes. For stateless systems, rollout speed is a feature. For stateful systems, recoverability is the feature, and speed is something you trade away on purpose. This post is the rollout chapter of a series that starts with config-as-data and path-filtered CI/CD — here I'm assuming you have those, and focusing on the principles that keep a stateful change from becoming a story people tell for years.

Principle 1: Optimize for Recoverability, Not Speed

For anything stateful, the defaults should lean toward "hard to destroy by accident." Concretely, that means retention policies and termination protection baked into the code, not left to a runbook:

# CDK — retain data-bearing resources even if removed from the stack
database.apply_removal_policy(RemovalPolicy.RETAIN)

# Pulumi — protect a critical resource from direct deletion
Database("primary", ..., opts=ResourceOptions(protect=True))

If recreation takes hours or days, "just redeploy" is not a recovery strategy — it is the incident. Make the safe default the lazy default, so an engineer under pressure has to go out of their way to do the dangerous thing.

Principle 2: Separate Validation from Deployment

Every change earns its way to production through two distinct phases:

Render/preview in CI, for each target environment — the machine proves the change is valid.
Deployment, executed only by pipeline automation after approval — never an ad hoc local deploy/up.

The path-filtered pipeline is what makes phase one cheap and phase two auditable.

Principle 3: Use Environment Promotion Discipline

Promote through environments in order, never in parallel:

development / testing
staging / acceptance
production

Each stage catches a different class of problem: testing surfaces logic and config errors, staging surfaces integration and data-shape surprises, and by production you want no surprises left. If you run more than one production footprint for resilience (see Principle 6), roll out to them one at a time — never to both at once.

Principle 4: Treat Config Changes as High Signal

When infrastructure is config-driven, the config diff is the change. That is a gift to reviewers — but only if they know to read it that way. Require a reviewer to check, on every diff:

capacity and sizing fields
network allocations
identity and sharing principals
feature-enablement flags

A one-line config change can trigger a resource replacement. The config-as-data post is about making these diffs legible; this principle is about making sure someone actually reads them.

Principle 5: Enforce Manual Gates for High-Risk Changes

Manual approval earns its keep when:

replacement of a stateful resource is on the table
a maintenance window is required
cross-team coordination is necessary

Automated quality gates and manual production gates are not in tension — the machine proves valid, the human decides safe to release now.

Principle 6: Decide Your Availability Target Before You Design the Rollout

For production, the biggest rollout decision is not how you deploy — it is how many production footprints you keep alive. That is really a disaster recovery (DR) decision, and it drives both your resilience and your bill.

The ideal setup: active-active across two failure domains. Run two independent production stacks in separate regions or availability zones, both serving traffic, both kept in lockstep. If one fails — an AZ outage, a bad regional change, a corrupted stack — the other already carries the load. You roll out to one footprint at a time, so a single bad release can never take down both. For a stateful platform that cannot tolerate downtime or data loss, this is the gold standard.

The trade-off: it roughly doubles your infrastructure cost and adds real operational weight. Data has to stay in sync, routing has to fail over cleanly (the connectivity post covers the redundant-path side of this), and every change now runs twice. For an expensive-to-provision platform — a large managed database engine, for example — the second footprint is often the single biggest line item on the bill.

So the honest choice is rarely "do the ideal thing." It is "how far down from the ideal can we responsibly go?" The options I actually weigh:

Availability need	Realistic setup	Cost vs. single stack
No downtime, no data loss	Active-active, staged one side at a time	~2x, highest complexity
Short recovery time acceptable	Active-passive: warm standby, promote on failure	~1.3–1.6x
Rebuild window acceptable	Single production + rehearsed backup/restore	~1x, cheapest

My rule of thumb: design the ideal on paper, then deliberately spend down from it based on what an outage on this specific system would actually cost the business. Sometimes active-active is obviously worth it. Just as often, active-passive with a rehearsed failover is the better return. The worst outcome is paying for a second footprint you have never actually failed over to — that is cost without the reliability you bought it for.

Principle 7: Rotate the Credentials You Depend On

A stateful platform you cannot afford to lose is also one whose credentials you cannot afford to leak or let go stale. Two habits keep this boring:

Store secrets in a secrets manager — never in state or config. IaC state files are the classic accidental leak; keep database credentials out of them entirely (see where state and secrets live).
Rotate on a schedule, and rehearse it. Rotation that has never been exercised is just another failover you are assuming works. Automate it, and run it through the same testing → staging → production promotion as any other change, so a rotation can never surprise production.

A Lightweight Risk Matrix

The point of the principles above is to make decisions consistent under pressure, when judgment is worst. A small matrix in the pull request template does most of that work:

Change Type	Risk	Approval Rule
Tag-only metadata updates	Low	Standard reviewer
Route and IAM changes	Medium	Senior reviewer + preview diff
Stateful sizing/replacement changes	High	Senior reviewer + manual release gate

It turns "is this scary?" from a gut call into a lookup — which is exactly what you want at 4pm on a Friday.

Final Takeaway

For stateful cloud infrastructure, safety comes from process design, not heroics:

conservative defaults that make the safe path the easy path
strict preview validation, separated from deployment
phased promotion, one footprint at a time
explicit production approvals matched to blast radius

Get the process right and incidents become rare and boring. That scales far better than being brilliant during the ones you could have prevented.

Network Connectivity Patterns for Managed Database Platforms

Sat, 18 Jul 2026 00:00:00 GMT

The hardest part of putting a managed database platform into a large organization is almost never the database. It is answering one deceptively simple question: how does everything that needs the database actually reach it? A dozen application teams, a couple of legacy accounts, an on-prem network still running the systems of record — each one needs a path in, and each path is a decision about ownership, latency, blast radius, and cost.

I have watched that question get answered badly in both directions. Route everything through one central hub and you get a tidy diagram and a single point of failure that no one team feels responsible for. Let every team build its own path and you get flexibility plus a slow drift into a routing mess nobody can reason about. The useful framing sits in between:

Transit path — for centralized ingress/egress, on-prem, and shared inspection.
Direct peering — for low-friction, east-west application access.

This post is the connectivity chapter of a series on running stateful platforms safely; it leans on the same config-as-data discipline for CIDRs and shares its DR thinking with safe rollouts.

Two Connectivity Models

flowchart LR
  subgraph Transit[Model A: Transit-Centric]
    T1[Consumer A] --> H((Transit Hub))
    T2[Consumer B] --> H
    OP[On-prem / VPN] --> H
    H --> DBa[(Database network)]
  end
  subgraph Direct[Model B: Direct Peering]
    C1[Consumer A] --> DBb[(Database network)]
    C2[Consumer B] --> DBb
  end

Model A: Transit-Centric

All traffic passes through a transit hub.

Pros: centralized routing controls, a natural place for shared inspection, easy global policy enforcement.

Cons: more route dependencies, a larger blast radius when the hub changes, and often a longer path for what should be a short app-to-database hop.

Model B: Direct Peering by Consumer Domain

Each consumer domain owns a direct peering to the database network.

Pros: clear ownership boundaries, simpler troubleshooting (one path, two ends), no central bottleneck.

Cons: more peerings to govern, and route hygiene now lives in each domain rather than one place.

Decision Criteria

Choose direct peering when:

consumer domains are mature enough to own their own routes
low-latency application flows are the primary use case
decentralized ownership is a feature, not a risk

Choose transit-centric when:

centralized traffic inspection is mandatory
consumer domains cannot safely manage network routes
a hub-and-spoke operating model already exists and works

In practice most mature setups end up hybrid, and deliberately so: direct peering carries the high-volume east-west application traffic, while the transit path is reserved for on-prem and shared ingress — the flows that genuinely benefit from a central chokepoint. The mistake is drifting into a hybrid by accident, with no clear rule for which flow belongs where.

A Migration That Paid for Itself

My own route to that hybrid was not a clean design — it was a correction. The first version I built routed everything through a central transit hub, because a hub is the easy thing to reason about: one place for routes, one place for policy, one diagram. It worked. It was also needlessly expensive.

A transit hub typically bills two ways: an hourly charge per attachment and a per-gigabyte data-processing charge on everything that flows through it. All that east-west application traffic to the database was paying a toll on a path that added nothing for it — those flows never needed central inspection, they just needed to reach the database. Moving the consumer traffic onto direct peerings, and leaving only on-prem and shared ingress on the hub, took a recurring line item down substantially with no loss of function.

The lesson I took: a transit hub is a great home for the traffic that benefits from centralization and a quietly expensive one for the traffic that doesn't. Route by need, not by habit — and revisit the bill once real traffic is flowing, because the cheapest topology on day one is rarely the cheapest at scale.

The Ideal Path vs. the One You Can Afford

The ideal connectivity design is fully redundant: every consumer reaches the database over at least two independent paths, in two failure domains, so no single peering, route table, or region can cut access. Pair that with an active-active database footprint and you have a platform that shrugs off most infrastructure failures.

That resilience is not free. Every extra path is another peering to create, another set of routes to keep clean, and — for active-active — a duplicate of the most expensive resource you run. The trade-offs I weigh in practice:

Redundant paths everywhere — highest resilience, but the most route-governance overhead and cost. Justified only when downtime on this flow is genuinely unacceptable.
Single primary path plus a tested failover — where most projects should live. You accept a short, rehearsed recovery step in exchange for far less standing cost and complexity.
Single path, no failover — fine for environments you can afford to lose for a while (testing, internal tooling), reckless for anything else.

My rule of thumb mirrors how I think about DR generally: design the ideal on paper, then deliberately spend down from it based on what an outage on this specific flow would actually cost. A redundant path you never fail over to is just complexity with a bill attached. (I develop the same idea, applied to the database itself, in the safe rollouts post.)

Routing Guardrails That Prevent Incidents

Whichever model you pick, the failure mode is the same — ambiguity about who owns which route. Most outages I have seen came from that, not from a missing cloud feature. The guardrails that hold it back:

Declare route ownership per domain, explicitly.
Document the allowed prefixes in both directions.
Enforce non-overlapping CIDR allocations (validate them as data — see the config-as-data guardrails).
Test bidirectional reachability before cutover, not after.
Keep DNS forwarding design explicit and versioned, not tribal knowledge.

A minimal connectivity matrix, kept in the repo, is worth more than any diagram:

From	To	Path	Owner
app-domain-a	database-net	direct peering	app-domain-a
on-prem	database-net	transit hub	network team

The Prerequisite That Blocks Everything

Before any of this works, the managed-database integration has to be enabled in every account that will consume it. It is easy to switch it on in the account that owns the database, prove connectivity there, and then watch a consumer account fail weeks later with an opaque error — because the integration was never enabled on that side. Make "enable the integration in this account" an explicit, checked step in account bootstrap, not tribal knowledge. When you run many accounts it belongs in the same config-as-data inventory that already tracks your CIDRs and route ownership.

A Two-Phase Rollout That Avoids the Route Race

The subtle trap with peering is ordering — and it gets worse when the database platform's control plane lives outside the cloud you are deploying from. Creating the peering is not instant: the request has to reconcile across that provider boundary, and until it reports active on both sides, any route that references it will fail. A change that creates the peering and writes the routes in a single shot will race that propagation and leave a network half-wired. Split it into two phases with a real barrier between:

Phase 1  reserve CIDRs, validate no overlaps
         create the connectivity primitive (peering / attachment)
         --- confirm the primitive is ACTIVE ---
Phase 2  apply routes in the producer network
         apply routes in each consumer network
         validate DNS + the connectivity matrix
         enable application traffic gradually

Make that barrier a poll, not a guessed sleep: wait until the peering actually reports active, then apply the routes. Treating peering creation and route updates as separate approvals also keeps the blast radius small — a bad route change can be rolled back without tearing down the peering underneath it.

Final Takeaway

Connectivity architecture is mostly an ownership problem expressed through routes. The cloud gives you transit gateways and peerings; whether they add up to a reliable platform or a fragile one comes down to explicit boundaries, validated CIDRs, and disciplined rollout ordering — not which primitive you picked.

Multi-Cloud, Done Deliberately: Strategy, and Choosing Between Terraform and Pulumi

Fri, 17 Jul 2026 00:00:00 GMT

Most "multi-cloud strategies" I've seen weren't strategies at all. They were the accumulated residue of an acquisition, a team that liked BigQuery, another that lived in AWS, and a CFO who read an article about vendor lock-in. Multi-cloud arrived; nobody chose it.

That's the honest starting point. Multi-cloud can be a genuine advantage or a self-inflicted tax, and the difference is whether you did it deliberately — with a reason, a boundary, and the automation to keep it from sprawling. This post is about both halves: deciding whether (and how) to go multi-cloud, and then picking the infrastructure-as-code tool that makes it survivable. Since the IaC landscape shifted hard over the last two years, I'll be specific about where things actually stand in 2026.

Part 1 — Is multi-cloud even the right call?

Start by being ruthless about why. There are good reasons and expensive-cargo-cult reasons.

Reasons that hold up:

Best-of-breed by workload. BigQuery for analytics, a specific AWS service for something else, Azure because your enterprise agreements and AD already live there. You're picking the best tool per job, not spreading one app across clouds.
Data residency and regulation. You genuinely need workloads in a region or jurisdiction where only one provider is a good fit.
Resilience against provider-level failure. Real, but rarer and more expensive than people think — true active-active across clouds is a serious engineering commitment, not a checkbox.
Mergers and acquisitions. You inherited another cloud and consolidation isn't worth it yet.
Negotiating leverage / avoiding lock-in. Legitimate, but it's a business hedge, not an architecture — and it has a running cost.

Reasons that usually don't survive scrutiny: "in case we ever want to switch," and "so we're not locked in" — when nobody has costed what that portability actually requires.

The uncomfortable truth: the more portable you force your architecture to be, the more you give up the managed services that made each cloud worth using. A lowest-common-denominator design that runs anywhere often runs worse everywhere. Portability is a real cost you pay in capability — spend it on purpose.

The patterns, honestly

Pattern	What it is	When it's worth it
Portfolio / siloed	Different apps live on different clouds; each is single-cloud internally	The common, sane default — best-of-breed without cross-cloud complexity
Best-of-breed composition	One system spans clouds to use a specific service (e.g. app on one, analytics on another)	When a specific managed service is genuinely differentiated
Portable / abstracted	Workloads built to run on any cloud (often on Kubernetes)	Strong regulatory or exit requirements — and you've accepted the capability tax
Active-active DR	Same workload live across clouds for provider-level resilience	Rarely; only when an outage cost genuinely exceeds the (large) engineering cost

What this looks like in practice

A deliberate best-of-breed split might look like this — illustrative, not a prescription:

Workload	Cloud	Why there
Analytics	GCP	BigQuery and the data stack around it
Identity	Azure	where the enterprise directory already lives
Application tier	AWS	the broadest service catalog
Primary database	a dedicated database cloud	placed where the engine runs best — kept separate from the app tier on purpose

Several clouds, but not chaos: each workload is placed where it runs best, and each stays coherent inside its own provider. That's the "best-of-breed" pattern working as intended — multi-cloud by deliberate selection, not by accident. The cost is that you now need one consistent way to provision and observe all of them, which is the whole reason the IaC choice below matters so much.

My default advice: aim for "portfolio" and reach for the heavier patterns only when a concrete requirement forces you there. Whatever you pick, the thing that keeps multi-cloud from rotting into chaos is a single, consistent way to define and change infrastructure — which is where IaC earns its keep.

Part 2 — Why IaC matters more in multi-cloud

In a single cloud you can almost get away with clicking around the console. Across clouds you can't: you now have two or three sets of primitives, IAM models, and networking quirks, and no human holds all of it. Infrastructure-as-code gives you one control plane, reviewable diffs, reproducible environments, and drift detection across every provider. In multi-cloud, IaC isn't a nice-to-have; it's the only thing standing between you and untracked, un-auditable sprawl.

The 2026 question isn't whether to use IaC — it's which tool. The realistic contenders are Terraform, its fork OpenTofu, and Pulumi. (Cloud-specific tools like AWS CDK or Bicep are excellent but single-cloud by design, so they're out for a genuinely multi-cloud estate.)

Part 3 — Terraform and OpenTofu: the fork you need to understand

You can't choose Terraform in 2026 without understanding what happened to it.

In August 2023, HashiCorp relicensed Terraform from the open-source MPL to the Business Source License (BUSL) — still source-available, but no longer OSI-approved, and it restricts building competing products. The community forked the last MPL version into OpenTofu, now a Linux Foundation / CNCF project. Then in April 2024 IBM announced it would acquire HashiCorp, a deal that closed on February 27, 2025 (~$6.4B). The license did not revert after the acquisition — Terraform remains BUSL.

Two years on, the fork has genuinely diverged — this is the part people miss:

	Terraform (BUSL, IBM/HashiCorp)	OpenTofu (MPL-2.0, CNCF)
License	Business Source License — source-available, restricted	MPL-2.0, fully open source
Leads on	Enterprise platform: HCP Terraform, Stacks, Sentinel policy, the provider registry, Infragraph	Open-source CLI features: native state encryption, provider `for_each`, `-exclude`, early variable evaluation, OCI registries
Governance	Vendor-controlled	Community / foundation
Best when	You want the managed enterprise platform and don't mind the license	You want open governance, no license risk, and state encryption in the CLI

Both still speak HCL, share the state format, and have the deepest provider ecosystem in the industry. Day to day they look nearly identical — but they are no longer the same tool, and "which Terraform" is now a real decision. For a licensing- or compliance-sensitive enterprise, OpenTofu is often the safer default; for teams already invested in HCP Terraform's platform features, Terraform proper still leads.

If you're starting fresh on HCL in 2026, I'd default to OpenTofu unless you specifically need HashiCorp's commercial platform — you get an open license, state encryption out of the box, and an easy path, without betting on a vendor-controlled license.

Part 4 — Pulumi: infrastructure in a language you already know

Pulumi takes the other road. Instead of a DSL, you define infrastructure in general-purpose languages — TypeScript, Python, Go, C#, Java, or YAML — against the same underlying cloud providers (GCP, Azure, AWS, and others), so it genuinely spans a multi-cloud estate. It's Apache-2.0 open source. State can live in Pulumi Cloud (managed, versioned, locked, with per-value secret encryption), or in a self-managed backend — for example an object-storage bucket for state, with a cloud secrets manager holding the secrets. You're not forced onto a hosted service to use it.

What that buys you:

Real language constructs — loops, conditionals, functions, classes, types — without HCL's expression gymnastics.
Lower onboarding cost. Engineers who already write Python or TypeScript can read, understand, and contribute to infrastructure without first learning a separate DSL. In a multi-cloud estate that's a genuine force-multiplier: the app and data engineers who need to touch infra can understand the setup because it's written in a language they already think in, so onboarding someone onto the infra is a matter of hours, not weeks spent learning HCL.
Testability — you can unit-test infrastructure with the frameworks you already use (pytest, Jest, Go's testing), mock provider calls, and run integration tests.
Embedding IaC in software — the Automation API lets you drive Pulumi programmatically, which is why it's a strong fit for building an internal developer platform.
Secrets and config — Pulumi ESC (Environments, Secrets, Configuration), GA since 2024, centralizes secrets across AWS Secrets Manager, Vault, Azure Key Vault, GCP Secret Manager, and more, with logged access.
A hedge on the HCL question — as of January 2026 Pulumi added native HCL support via a Terraform bridge, so it can interpret existing HCL. That lowers the switching cost if you're coming from Terraform.

The trade-off: general-purpose languages give you power and enough rope to build genuinely over-engineered infrastructure. HCL's constraints are sometimes a feature — they keep configuration boring and legible. Pulumi shines when your team is application-developer-heavy or you're building a platform; it can feel like overkill for a small, stable estate a couple of HCL modules would cover.

Part 5 — How I'd choose

There's no universal winner. The decision comes down to your team, your license tolerance, and whether infrastructure is a product you build on or a config you maintain.

flowchart TD
  A["Choosing an IaC tool for multi-cloud"] --> B{"Team writes app code<br/>daily and wants tests,<br/>loops, an IDP?"}
  B -->|"Yes"| C["Pulumi<br/>real languages, testable,<br/>Automation API + ESC"]
  B -->|"No — infra config,<br/>HCL is fine"| D{"Need HashiCorp's<br/>enterprise platform?<br/>(HCP, Stacks, Sentinel)"}
  D -->|"Yes"| E["Terraform (BUSL)"]
  D -->|"No — want open license<br/>+ state encryption"| F["OpenTofu (MPL-2.0)"]

A head-to-head, condensed:

Dimension	Terraform / OpenTofu	Pulumi
Language	HCL (declarative DSL)	TypeScript / Python / Go / C# / Java / YAML
License	BUSL (Terraform) · MPL-2.0 (OpenTofu)	Apache-2.0
Provider ecosystem	Largest in the industry	Broad (bridges Terraform providers)
Testing	Limited (plan checks, terratest)	Native unit/integration testing
Secrets	Vault / cloud stores; OpenTofu adds state encryption	ESC + per-value state encryption by default
Best fit	Ops/platform teams standardizing on config	App-dev teams, platform engineering, IDPs

Where I land: for a multi-cloud estate whose team lives in application code, I reach for Pulumi in a general-purpose language (Python or TypeScript), with a self-managed state backend and a dedicated secrets manager. Real languages let the data and application engineers contribute to infrastructure in the languages they already use — onboarding a developer onto the infra takes hours rather than weeks of learning a DSL, and they can genuinely understand how the setup works because it's written in a language they know. One tool spans every provider without a separate dialect per cloud to context-switch between. It's not the only right answer — but for a team that lives in code, it's the one that tends to stick.

Part 6 — The part that matters more than the tool

Here's what I'd tell anyone going multi-cloud: the tool is the smallest decision. I've watched teams agonize over Terraform-vs-Pulumi and then lose a weekend to a corrupted state file, or ship an IAM wildcard to three clouds at once. What actually determines whether multi-cloud IaC succeeds:

State strategy. Remote, locked, encrypted, and segmented — not one monolithic state file for your whole estate. Blast radius is a design decision.
Module/component boundaries per cloud. Resist the urge to build a grand homegrown abstraction that "unifies" the clouds. Share patterns and conventions, not a leaky lowest-common-denominator wrapper — those wrappers become the thing nobody can maintain.
Policy-as-code. Sentinel (Terraform) or OPA/Conftest (anywhere) to enforce least-privilege, tagging, and region rules before apply — across every provider.
Secrets, centralized. ESC or Vault, never in state you can read, never in the repo.
CI/CD with plan-review gates. Every change is a reviewed diff; nobody applies from a laptop.
Observability as code, and centralized. Telemetry can't live in separate cloud consoles per provider. Standardize on one observability plane — Datadog, for instance — and manage its monitors, dashboards, and SLOs as code too. The major observability platforms ship both a Pulumi provider and a Terraform provider, so the same review-and-apply flow that provisions the infrastructure also provisions what watches it, across every cloud.

Get those right and Terraform, OpenTofu, and Pulumi all work. Get them wrong and no tool saves you.

Part 7 — From image to running infra: the delivery pipeline

Choosing the IaC tool is only half of "automation." The other half is how images get built and how changes actually ship — and in a multi-cloud estate, that pipeline is what keeps things reproducible instead of hand-crafted.

Immutable images, built once, deployed as versions. Rather than mutating running servers, bake versioned images and replace instances wholesale. Two common ways to build those images:

Packer for golden images across clouds — one set of templates produces the AMI, the Azure image, the GCP image, and so on from a shared definition, which is exactly the leverage you want when you support several providers. (Worth knowing: Packer, like Terraform, is now BUSL-licensed — the same license consideration applies.)
Cloud-native image builders (EC2 Image Builder, Azure VM Image Builder, and equivalents) where a provider-managed pipeline fits a workload better than a portable one.
A configuration-management tool (Salt, Ansible, or similar) for the "what's actually inside the image" layer: packages, hardening, and config applied at build time, so a running instance is fully defined rather than hand-tweaked after boot.

That gives a clean separation of stages: build the image (Packer / image builder + config management) → version it → let the IaC tool reference the image ID → provision and deploy. Each stage is independently versioned and reviewable, which is what makes a change traceable across clouds.

CI/CD ties it together. The IaC runs live in CI — often a mix of self-hosted runners and a hosted CI service. Self-hosted runners earn their place for pipelines that need to sit inside the network or reach systems that shouldn't touch the public internet; the hosted service handles the rest. Either way the flow is the same: pulumi preview (or terraform plan) on the pull request — the reviewed diff — and apply on merge, pulling secrets from a secrets manager and state from the backend, with policy checks gating the apply. Image builds run on the same backbone, publishing new versioned images that later infra changes pick up.

The through-line: image build, configuration, and provisioning are three distinct, versioned, reviewable stages — not one big manual dance. That discipline is what lets a multi-cloud, multi-language estate stay reproducible.

Final thoughts

Go multi-cloud because a specific requirement makes it the better architecture — best-of-breed services, real regulatory needs, genuine resilience economics — not because "lock-in" sounds scary in the abstract. Then automate it with the tool that fits your team: OpenTofu if you want open, HCL-based IaC without license risk; Terraform if you're buying into HashiCorp's enterprise platform; Pulumi if your people think in code and you're building infrastructure as a product. And whichever you pick, spend your real effort on state, boundaries, policy, and secrets — the discipline outlives the tool.

What's your multi-cloud setup — deliberate, or accidental? I'd love to compare notes — find me on LinkedIn.

Path-Filtered CI/CD for Infrastructure Monorepos

Mon, 13 Jul 2026 00:00:00 GMT

An infrastructure monorepo is a lovely idea right up until the first time you change a README and watch three cloud pipelines spin up to tell you nothing changed. Multiply that by every pull request and you get the quiet tax of the naive monorepo: slow feedback, plans nobody reads because they're always green-for-the-wrong-reason, and eventually a team that stops trusting CI because it cries wolf on every commit.

The fix is not to split the repo. It is to run only the pipeline that owns the files you touched — path-filtered delivery chains. One repository, one place to review cross-cutting changes, but execution scoped tightly to what actually changed.

This is the delivery chapter of a series on running stateful infrastructure safely. It assumes the config-as-data layout (the path filters map onto those directories) and feeds directly into safe rollouts.

The Problem

In a monorepo with no path awareness:

a networking change triggers the database pipeline
an IAM tweak triggers unrelated application checks
slow, always-running plans bury the one failure that mattered in noise

The result is longer feedback loops and — worse — lower confidence in every green check.

The Design

Split pipelines by directory ownership. A representative mapping:

cloud-a/** → Cloud A IaC chain
cloud-b/** → Cloud B IaC chain
docs/** → docs checks only

Each chain owns its own runtime bootstrapping, lint/test commands, preview/synth step, and release/version policy. The repo stays unified; the execution stays scoped.

flowchart LR
  A[Pull Request] --> B{Changed Paths}
  B -->|cloud-a/**| C[Cloud A Chain]
  B -->|cloud-b/**| D[Cloud B Chain]
  B -->|docs/**| E[Docs Chain]
  C --> F[Preview or Synth]
  D --> G[Preview]
  F --> H[Manual Prod Gate]
  G --> H

What a Path Filter Actually Looks Like

The mechanism is boring, which is the point — every major CI system ships it. The shape, in a generic pipeline config:

jobs:
  cloud-a:
    trigger:
      paths: ["cloud-a/**"]
    steps:
      - install-deps
      - lint-and-test
      - render-plan        # synth / preview / equivalent
      - publish-artifact

  cloud-b:
    trigger:
      paths: ["cloud-b/**"]
    steps: [install-deps, lint-and-test, render-plan, publish-artifact]

On GitHub Actions that's on.push.paths; on GitLab it's rules: changes:; most other systems have a direct equivalent. Two guardrails worth setting from the start:

Make filters mutually exclusive. Overlapping globs (cloud-a/** and **/config) will double-trigger and quietly waste runners.
Decide what a shared-file change does. A change to a root lockfile or CI config arguably touches every chain — be explicit about whether it fans out to all of them or fails closed.

Why Manual Production Gates Still Matter

Automated checks should be strict. But for stateful infrastructure, the deploy step itself often deserves a human. This is not nostalgia for manual ops — it is matching the approval to the blast radius.

Use a manual gate when:

replacing the resource has a high blast radius
the change window is controlled
the deploy depends on coordination with another team

Automated quality gates and manual production gates are complementary: the machine proves the change is valid, the human decides it is safe to release now. I unpack that division in the safe rollouts post.

Suggested Stage Layout

For each infrastructure chain:

Install dependencies (from a committed lockfile — see failure modes).
Static checks and unit tests, including config guardrails.
Render the plan (synth, preview, or equivalent).
Publish the artifact and review the diff.
Manual approval for production.
Deploy by pipeline identity only — never a local laptop.

That last point matters more than it looks: a local deploy skips every gate above it and leaves no artifact to audit. The pipeline identity is the only principal that should hold deploy permissions in production.

Ownership Model

Path filters pay off only when each folder has a clear owner:

one team owns the pipeline and the folder it deploys
one review group approves that component's production releases
one changelog and version stream per component

Anything ambiguous here shows up later as a handoff nobody owns.

Failure Modes to Watch

Overlapping path rules that trigger duplicate chains.
Shared files with unclear ownership — the root config that belongs to everyone and no one.
Non-deterministic previews from a missing or stale lockfile. Pin dependencies or your "no-op" diffs will lie.
Hidden runtime dependencies on a specific engineer's machine.
A toolchain too old to know the resource types. Brand-new cloud services often ship L1-only — the raw, CloudFormation-level constructs — and only in recent releases of your IaC library. Pin a recent version (and, if you lint CloudFormation separately, a recent linter too). An older toolchain fails synth or validation on an "unknown resource type" it has simply never heard of — a confusing failure if you don't know to check versions first.

Fix these before you scale up contributors, not after.

Final Takeaway

Path filtering is one of the highest-leverage changes you can make to an infrastructure monorepo. It cuts noise, tightens feedback, and — most importantly — keeps delivery responsibility aligned with code ownership, so every green check means something again.

Config as Data: A Safer Pattern for Multi-Environment Infrastructure

Tue, 07 Jul 2026 00:00:00 GMT

Almost every infrastructure incident I have had to clean up traced back to the same root cause, and it was never an exotic cloud bug. It was a human editing stack code under pressure: a CIDR copied from the wrong environment, an availability zone hardcoded "just for now," a feature flag that defaulted to on because nobody wrote down that it shouldn't. The cloud primitive worked perfectly. The change did exactly what the code said. The code was just wrong in a way no review caught, because the risky value was buried in the middle of a hundred lines of resource wiring.

One pattern fixed most of that class of problem for me: treat configuration as data, and infrastructure code as a generic engine that consumes it. The stacks stop knowing anything about "production" or "testing." They know how to build a network or build a database, and they read every environment-specific number from a validated config file.

This post is the anchor for a small series on the practices that grew out of running this pattern in anger — it pairs closely with safe rollouts for stateful infrastructure and path-filtered CI/CD for infra monorepos. The whole point is that the pattern is the reusable part.

The Core Rule

Keep stack logic generic. Put every mutable, environment-specific value in a versioned config file. The dividing line is simple: if a value can differ between environments, it is data.

What lives in config:

CIDR blocks and subnet allocations
Availability zone choices
Environment toggles for optional resources
Principals and cross-account sharing targets
Sizing and capacity values

What stays in code:

Validation rules
Resource composition logic
Security defaults (retention, encryption, permission boundaries)
Naming and tagging conventions

The mental model is a pure function. Config is the input, the stack is the function, and the rendered plan is the output:

flowchart LR
  A[(config/<br/>environments.py<br/>capacity.py)] --> B[Generic stack code<br/>network / database / identity]
  B --> C[testing plan]
  B --> D[acceptance plan]
  B --> E[production plan]
  A -.validated first.-> V{{Guardrail tests}}
  V -.block on failure.-> B

Same function, different inputs, deterministic outputs. That single property is where all the benefits come from.

Why This Works

You gain three things that compound:

Predictability. A change diff is almost entirely config, so intent is obvious at a glance. "We grew the acceptance database by two nodes" reads as a two-line diff, not a hunt through stack logic.
Reusability. The same code path serves every environment, so testing exercises the exact code that runs in production.
Safety. Because config is just data, you can validate it before the provider ever sees it.

It also collapses code review into one question: did we change data, or did we change behavior? Data changes get scrutinized for values; behavior changes get scrutinized for blast radius. Reviewers stop having to do both at once on every line.

A Practical Structure

A layout that has held up well for me:

infra/
  config/
    environments.py     # per-env CIDRs, AZs, principals, toggles
    capacity.py         # per-env sizing, gated by an `enabled` flag
  stacks/
    network_stack.py
    database_stack.py
    identity_stack.py
  tests/
    test_config_guardrails.py

Stacks load config by environment key and never reach for a constant of their own:

env_cfg = ENVIRONMENTS[env_name]
network_cfg = env_cfg.network
capacity_cfg = CAPACITY[env_name]

# The stack composes resources from cfg — it never hardcodes an environment.
VpcNetwork(self, "network", cidr=network_cfg.cidr, azs=network_cfg.azs)

The rule I hold the line on: grep the stacks/ directory for an environment name and you should find zero hits. The moment if env == "production" appears in stack code, the pattern has started to leak.

Guardrails You Should Add Early

The payoff of config-as-data is that you can validate the data. Do it before synth/preview runs, as ordinary unit tests:

def test_subnets_within_supernet():
    for name, env in ENVIRONMENTS.items():
        assert env.network.cidr in env.network.supernet, (
            f"{name}: client subnet escapes its supernet"
        )

def test_no_overlapping_cidrs():
    seen = []
    for env in ENVIRONMENTS.values():
        assert not any(env.network.cidr.overlaps(s) for s in seen)
        seen.append(env.network.cidr)

A starter set of checks worth having on day one:

Every subnet sits within its environment's supernet.
No two environments overlap CIDRs. (This one has saved me more than once — it's the same discipline that keeps database connectivity sane across peerings.)
Every environment defines all required keys.
Account and tenant identifiers appear only in config, never in code.
Optional features require an explicit flag; nothing dangerous defaults on.

These run in milliseconds and catch bad inputs before any provider API call — which means bad config fails a pull request, not a deployment window.

CI/CD Integration

The guardrails only help if they run automatically. In every pull request:

Lint and unit tests.
Config guardrail tests.
Render plans/previews for the changed environments.
Block merge on any failure.

Because the config is the diff, reviewers get a plan that maps one-to-one to the values that changed. That fast, legible feedback loop is exactly what makes path-filtered pipelines worth building.

Where State and Secrets Live

Config-as-data answers what to deploy. Two adjacent questions decide how safely you can operate it.

State. Tools like Pulumi and Terraform keep a state file describing reality. You can self-host that backend on object storage you own, or use the vendor's managed service. Self-hosting keeps the data in your own account and gives you full control; the managed option hands you locking, history, and encryption without running one more piece of stateful infrastructure. Neither is wrong — self-host when control or data residency matters, reach for the managed service when you would rather not babysit a backend. Pick deliberately, because migrating state later is a chore.

Secrets. Whatever you choose, database credentials and other secrets do not belong in that state file, in config, or in the repo — the state file is the classic accidental leak. Keep secrets in a dedicated secrets manager, reference them by name, and rotate them on a schedule (more on rotation in safe rollouts).

Common Anti-Patterns

Even when time is tight, these are the ones that come back to bite:

A "quick fix" value hardcoded directly into stack logic.
Environment-name checks (if env == ...) scattered across files.
Feature switches with implicit defaults.
Copy-pasted config with no schema validation.

Every one of them reintroduces the exact drift the pattern exists to kill.

Final Takeaway

If you adopt one rule from this post, make it this: environment-specific values are data, never stack logic.

It speeds reviews, kills a whole category of drift, and makes multi-environment infrastructure easier to reason about the larger your platform grows. Everything else in this series is a consequence of taking that one rule seriously.

Making AI Coding Assistants Reliable: My Playbook

Thu, 04 Jun 2026 00:00:00 GMT

Over the last couple of years I've gone from occasionally accepting an autocomplete to running multi-file agents against real codebases every day. Along the way these tools have saved me genuine hours — and, on the days I got sloppy, handed me confident, subtly wrong code that cost me more hours than they saved. This post is the working discipline I've settled on to get the upside without the regret. It's the guide I'd give a version of myself two years ago.

My examples lean on the stack I use daily — Python (uv, pytest, click) and TypeScript (AWS CDK) — but every principle transfers. It's written for cloud engineers and early-career developers.

The goal is simple: make AI coding assistance reliable, secure, and low-regret. These assistants are spectacular accelerators and confident liars, often in the same minute. In my experience the difference between the days they make me faster and the days they quietly make me slower isn't the model — it's the discipline around it.

TL;DR

Defaults

Default model: Claude Sonnet 5
Default workflow: Ask → Plan → Agent → Review → Test
Rule: small diffs, explicit constraints, always add or verify tests

Model ladder (as of 2026 — revisit quarterly, this changes fast)

Quick tiny edits → Claude Haiku 4.5 or your provider's fast tier (e.g. Gemini Flash)
Most coding tasks → Claude Sonnet 5
Multi-file patch + lots of tests → GPT Codex (current tier)
Deep debugging / security review / hardest refactors → Claude Opus 4.8
Second opinion on architecture or algorithms → Gemini Pro (current tier)

1. What an AI Assistant Is (and Isn't)

Think of Copilot not as a replacement for a developer, but as a highly skilled, tireless intern with a photographic memory and no common sense.

What it's good at

Accelerating mechanical coding — boilerplate, wiring up DTOs and interfaces, repetitive edits across large blocks of code.
Agentic multi-file scaffolding — drafting entire feature skeletons: a new API endpoint, its service layer, and its database model in one pass.
Generating test matrices — edge-case matrices, pytest parameterization, and mock setups for legacy code that no human wants to write by hand.
Code forensics — explaining legacy modules, tracing call graphs, finding hidden side effects ("Which functions in this module write to S3?").
Constraint-based refactoring — safe refactors ("rewrite this without changing the public API") when you provide a "golden example" from the repo to mimic.
On-the-fly documentation — README updates, docstrings, and architectural summaries derived directly from code.

What it's not

An independent architect. It cannot make high-level design decisions or understand business requirements without explicit context and human steering.
A source of truth. It doesn't "know" your system state — it predicts from what it sees in your open tabs. Outdated tabs produce outdated advice.
A replacement for quality gates. It doesn't replace code review, integration tests, or threat modeling. AI-generated code needs more rigorous review, not less.
A security expert. It's a pattern mimic. It will happily suggest insecure patterns — hardcoded credentials, permissive IAM wildcards — because those patterns are everywhere in public training data.
An autopilot for high-risk code. IAM policies, encryption logic, and auth handlers require deep human verification plus specialized tooling (Snyk, Checkov, bandit).

The golden rule: trust the AI for the syntax, trust yourself for the logic and security. If you wouldn't merge a junior developer's code without reading it, don't merge the AI's.

2. Modes & Sessions: Choosing How It Thinks and Where It Works

Modern Copilot is no longer a chat box — it's a tiered workflow system. Picking the right mode (how it thinks) and session (where it works) is the difference between a clean PR and a messy refactor.

Modes — the "brain" selection

Ask (analyze & explore). For forensics ("trace the call graph of this function"), strategy ("brainstorm test approaches for this module"), and gut checks ("is this IAM policy too permissive?"). Avoid it when you want code written and applied — that's Agent mode.

Plan (architect & align). Produces a Markdown plan and checklist of files to touch and tests to add before any code changes. Best for multi-file work where you want to review the "Step 1, Step 2…" logic first, and for surfacing side effects early.

Agent (autonomous execution). Implements features spanning multiple layers, applies patterns consistently across folders, and self-heals: it runs tests and linters, reads the errors, and fixes its own code until green.

Guardrails I include in every agent prompt:

"No new dependencies."
"Limit scope to [specific folder]."
"Stop and ask for confirmation after each step."
"Run uv run pytest after changes."

Sessions — the "workspace" selection

New chat — start fresh for each ticket, or whenever the context feels polluted with old logs. Stale context measurably degrades reasoning.
Local (interactive) — short tasks where you watch every edit in real time. Best for small fixes and tests in the file you're already in.
Background (asynchronous) — delegated tasks of five-plus minutes ("write 20 integration tests"). The agent works in an isolated Git worktree while you keep coding; you review the diff and apply when done.
Cloud (remote / PR agent) — large refactors and repo-wide tasks. Runs on remote infrastructure, opens a draft pull request, and assigns you as reviewer.

The workflow I use for 90% of non-trivial tasks:

flowchart LR
  A["Ask<br/>understand the code"] --> B["Plan<br/>design and align"]
  B --> C["Background Agent<br/>execute in isolation"]
  C --> D["Review the diff"]
  D --> E["Run the tests"]
  E -->|"pass"| F["Apply and commit"]
  E -->|"fail"| C

3. Context Window: The AI's Working Memory

AI reliability isn't about model size anymore — it's about context engineering. Most tools now show real-time token usage (e.g. 15K / 128K); learn to read it.

Why it matters:

Too little context → the AI guesses your repo patterns: hallucinated internal APIs, generic boilerplate that doesn't fit your stack.
Too much noisy context → "context rot." Models get measurably less accurate when the window fills with irrelevant code, dead logs, and failed attempts.
The 80% rule → when usage climbs past roughly 80% of the limit, compact the conversation or start a new chat.

Practical context hygiene

1. Curate your workspace. The AI weights your active tab and open editors heavily. Keep open: the file you're editing, its test file (the "contract" for success), one golden example of the pattern you want, and config files only when relevant.

2. Use explicit references. Don't let it guess which files matter:

"Implement the logic in #UserService.py following the pattern in #AuthService.py."

3. Run the "inferred convention" check. Before letting it write 100 lines, verify its mental model:

"Before you begin coding, summarize the architectural conventions, naming patterns, and testing libraries you've inferred from my open files. If any convention is unclear, ask me instead of guessing."

4. Manage session history. Use /compact to summarize the conversation and prune dead context. Switching goals (CLI work → infrastructure work)? New chat. Mixing goals leads to logic leaks.

Think of the context window as working memory: if you'd be overwhelmed by 30 open tabs, so is the model.

4. Which Model to Use When

The current generation of models is optimized for agentic workflows and long-horizon coding, not just chat. Use a ladder to balance cost, speed, and accuracy — and expect this table to age; revisit whatever your tool offers every quarter.

Situation	My pick (2026)	Why
Default daily driver	Claude Sonnet 5	Near-frontier quality on coding and agentic work at mid-tier cost. Handles most planning and implementation.
Quick tasks / tiny edits	Claude Haiku 4.5 / fast tier	Surgical for scoped edits, boilerplate tests, renames, and docs — at a fraction of the cost.
Multi-file patch + heavy tests	GPT Codex (current tier)	Strong instruction-following for structured code output; usefully rigid about provided signatures.
Deep debugging / security review / huge refactors	Claude Opus 4.8	The heavy artillery: long-horizon autonomy, strongest first-try rate on complex tasks, 1M-token context for sprawling codebases.
Second opinion / logic check	Gemini Pro (current tier)	A genuinely different model family — useful cross-check on architectural trade-offs and math-heavy algorithms.

The escalation strategy

Don't sit on one model all day — switch based on the risk profile of the task:

flowchart TD
  A["Task arrives"] --> B{"Risk and complexity?"}
  B -->|"mechanical: renames,<br/>boilerplate, docs"| C["Downgrade<br/>Haiku 4.5 / fast tier"]
  B -->|"normal feature work,<br/>planning, tests"| D["Default<br/>Claude Sonnet 5"]
  B -->|"high ambiguity: security audit,<br/>15+ interdependent files,<br/>race conditions"| E["Upgrade<br/>Claude Opus 4.8"]
  D -->|"stuck or hallucinating<br/>an internal API"| F["Cross-check<br/>Codex or Gemini Pro"]

Tip: if your default model keeps hallucinating a specific internal library, try a different family — specialized code models are often more rigid (in a good way) about sticking to the signatures you provide.

5. The Contract Model: Why Assistants Become Reliable

Copilot gets "buggy" when it has to guess your architecture. The fix is a contract-first approach: bind the AI to the same truth sources humans follow.

5.1 System rules (the always-on layer)

Bake non-negotiables into .github/copilot-instructions.md (or AGENTS.md) so they attach to every request:

Dependency discipline — no new libraries unless explicitly requested.
Security hygiene — never log secrets or print tokens.
Atomic changes — small, incremental diffs; no drive-by refactors.
Test-driven execution — every logic change ships with a test or verification checklist.
Infrastructure — least privilege always; IAM wildcards (*) require a comment explaining why, or they get rejected in review.

5.2 Repo signals (defining the truth)

Keep these open or reference them with @ to ground the AI in your actual stack:

Category	Python (uv)	TypeScript (CDK)
Build & deps	`pyproject.toml`, `uv.lock`	`package.json`, `tsconfig.json`
Quality gates	`ruff.toml`, `.pre-commit-config.yaml`	`eslint.config.js`, `.prettierrc`
Test logic	`tests/`, `conftest.py`	`tests/`, `jest.config.js`
Conventions	`README.md`, `CONTRIBUTING.md`	`README.md`, `docs/architecture.md`

5.3 The task prompt (ticket-level contract)

When handing a ticket to an agent, include four pillars:

Anchor — specific file paths and a golden example to mimic.
Behavior — clear inputs and outputs ("the CLI takes --json and returns a flat object").
Constraints — explicit don'ts ("don't call boto3 directly; use the existing storage wrapper").
Acceptance — how to verify ("uv run pytest tests/unit passes").

The truth-check prompt, for when you suspect a hallucinated API:

"Stop. Check the definitions in #pyproject.toml and the actual module source. Are you using a real method or guessing? If guessing, ask me for the correct import."

6. The 4-Part Prompt: A.T.C.D.

Anchor, Task, Constraints, Done-means. Without an anchor the AI defaults to "generic internet style"; without exit criteria an agent doesn't know when to stop.

Copy-paste template:

# Context: You are acting as a senior engineer in this repository.

### ANCHOR
- Primary file: @src/cli/deploy.py
- Reference pattern: follow the argument parsing style in @src/cli/auth.py

### TASK
- Goal: [describe the feature or fix]
- Signature: add_record(id: str, data: dict) -> bool
- Business rules: [rule 1], [rule 2]

### CONSTRAINTS
- Dependencies: no new libraries — existing uv / npm packages only
- Security: no PII or secrets in logs; least-privilege IAM
- Style: follow the ruff / eslint config in the repo root
- Diff: minimal changes only; do not refactor unrelated logic

### DONE MEANS
- [ ] Logic implemented in [path]
- [ ] Unit tests added/updated in [path]
- [ ] Verification: uv run pytest
- [ ] Edge cases handled: empty input, timeout, 403

For complex tasks, add a chain-of-thought trigger at the bottom:

"Before writing any code, explain your reasoning and list the files you intend to modify. Wait for my GO to proceed."

7. Prompt Profiles: Choosing Reasoning Intensity

Match the instruction set to the task, like choosing a gear.

Generate (fast)

For atomic edits and one-liners. Ideal model: Haiku 4.5 / fast tier.

"Minimal diff. Change ONLY the logic inside this function. Do not refactor, do not add comments, do not rename variables. If the fix is more than 5 lines, stop and ask."

Safe (default)

For daily feature work. Ideal model: Sonnet 5. The rhythm: plan → you approve → implement one step → verify with tests. Plan twice, code once.

Forensics (legacy & infrastructure)

For high-stakes areas — legacy modules, complex CDK stacks, IAM logic. Ideal model: Opus 4.8 with thinking. The discovery prompt:

"Before suggesting any changes, analyze this module: 1) System map — key responsibilities. 2) Side effects — does this touch S3, a database, or external APIs? 3) Safe seams — where is the safest place to inject new logic? 4) Blast radius — if this fails, what breaks downstream? Return a bulleted report before proposing code."

The profile switcher rule: if a "fast" task starts getting complicated, abort and restart in Safe mode. Hallucinations cluster exactly where a complex problem meets a fast instruction set.

8. Prompt Cookbook: Battle-Tested Recipes

Module forensics

"Analyze this module for a new engineer. Return bullets covering: core mission, the 3 most critical functions and their callers, any AWS / database / filesystem / network touches, which env vars or config files dictate behavior, and the lowest-risk seams for injecting new logic."

The plan-act handoff

"Propose a 4-step implementation plan for [feature] following the patterns in #file. Constraints: no new dependencies, minimal diff, pytest verification. Provide the plan first — do not write code. Once I approve, implement step 1 only."

Pytest matrix & parameterization

"Given this function, propose a pytest test matrix covering happy paths, boundaries (max/min, empty strings, None), and error propagation (403s, timeouts). Write the tests with @pytest.mark.parametrize and fixtures from #conftest.py. Strictly no network calls — mock all AWS and external APIs."

Click CLI testing

"Write pytest tests using click.testing.CliRunner for [command]. Cover exit codes (0 vs non-zero), stdout/stderr, and the --dry-run and --json flags. For --json, assert the output dictionary schema."

CDK fine-grained assertions

"Write unit tests using the aws-cdk-lib assertions library for this construct. Verify resource properties in the synthesized template: encryption settings, retention policies, mandatory tags. Do not use snapshots — use Template.has_resource_properties for targeted assertions."

The safe refactor

"Refactor this function to reduce cyclomatic complexity. Constraints: zero behavior change, add type hints and docstrings, keep the public signature identical. Propose the refactor plus pytest tests that would protect existing behavior during the swap."

Facts-only documentation

"Update the docs for [feature]. Document ONLY behavior and flags explicitly present in the provided code. If unsure about a side effect, omit it. Include 3 usage examples and common failure modes with their error messages."

The self-critique pass

"Critique your own solution: 1) Security — hardcoded secrets, IAM wildcards? 2) Performance — O(n²) loops, excessive API calls? 3) Constraints — did you stick to #pyproject.toml dependencies? 4) Edge cases — network down, malformed input? Return a fix list before I merge."

9. VS Code Surfaces: What to Use When

Inline suggestions — the flow state. Micro-edits, repetitive wiring, next-edit predictions. Steer with a comment:

# Create an SQS queue with a 14-day DLQ and KMS encryption

Avoid inline for multi-file logic or anything security-sensitive.

Chat (Ask mode) — the researcher. @workspace /explain for repo architecture (in current VS Code, #codebase is the newer inline equivalent), stack-trace root-causing from #terminal, planning before touching code.

Agent & Plan modes — the executors. Plan researches and drafts a step-by-step TODO list; Agent applies edits, runs commands, and fixes its own errors.

Session types in practice:

Session	Best for	How it works
Local	Interactive tasks, real-time Keep/Undo review	Edits your working tree directly
Background	Long tasks (bulk test-writing, folder-wide refactors)	Isolated Git worktree; review and apply when done
Cloud	Repo-wide migrations, "document everything"	Remote infrastructure; opens a draft PR

Watch the context indicator. Red means the AI is about to start forgetting your early instructions — compact or start fresh.

10. The Agent Workflow, Step by Step

Step 1: Local agent (interactive scaffolding)

Ensure a clean Git state so you can diff everything.
Open Chat → Mode: Agent, with your default model.
Prompt with the A.T.C.D. structure.
Review with the Keep/Undo UI, cherry-picking suggestions.

Example: "Create a CLI skeleton using click with a command group and one subcommand. No new dependencies; use uv. Done means: pytest tests using CliRunner and the exact uv run command."

Step 2: Plan + background agent (delegated execution)

Commit first — create a clean baseline.
Switch to Plan mode and prompt for a roadmap.
Approve the plan, then "Continue in Background."
Keep coding while the agent works in its isolated worktree.
Review the completed diff and apply.

Note: background agents don't see unsaved changes — save everything before delegating.

Step 3: Cloud agent (PR-scale work)

For repo-wide changes and migrations: the agent runs remotely, creates a branch, opens a draft PR, and your CI runs against it. Review it exactly like a human's PR — because that's the contract that keeps quality up.

11. Instruction Files: Reducing Bugs Through Grounding

The most common cause of hallucination is a lack of grounding. Instruction files are the AI's law books.

What counts as an instruction file: anything that defines the rules of the repo — pyproject.toml and lockfiles (which libraries are allowed), linter configs (style and strictness), CI workflows (how code is actually tested and deployed), and CONTRIBUTING.md (conventions and philosophy).

Three habits:

The open-tab rule. Keep pyproject.toml open in a background tab when dependency decisions matter — open files carry more weight in the AI's reasoning.
Explicit grounding. "Write this test using the fixtures defined in @conftest.py." Don't assume it read your configs.
Forbidden invention. Models love suggesting popular libraries (requests, pandas) whether or not they're in your stack: "Strictly use only the dependencies listed in #pyproject.toml. Do not suggest new libraries."

The repo-level instruction file. Create .github/copilot-instructions.md (Copilot) or AGENTS.md (the emerging cross-tool standard) at the repo root with your hard rules: "We use uv for all Python tasks." "CDK tests use fine-grained assertions, never snapshots." "CLI output must support --json."

If a rule isn't in a file, it doesn't exist for the AI. Document your conventions to automate your conventions.

12. Safe Usage Rules (Security & Privacy)

AI-assisted coding is now a real attack vector — both for leaking your data and for pulling malicious code in.

Data privacy

Treat every prompt as a record that may be stored and logged.

No secrets. Never paste API keys, tokens, or credentials. Use placeholders like <MY_API_KEY>.
Redact PII. No customer names, emails, or production data — use synthetic data (test_user_123) for debugging.
Prompt hygiene. Don't describe proprietary algorithms in detail; use abstract descriptions. If you're doing this at work, know your employer's AI usage policy before pasting anything.

Infrastructure & IAM

AI agents lack common sense about permissions. They suggest wildcards (*) because wildcards always work — and always create holes.

Least privilege: S3 access means a policy for one bucket, not s3:*.
The "why" rule: any AI-suggested wildcard needs a manual comment justifying it.
Audit trail: tag AI-generated infrastructure code — # Generated with AI — reviewed by <name>.

Human-in-the-loop verification

Never blind-merge AI code. A meaningful share of AI suggestions still contains subtle vulnerabilities — SQL injection, insecure deserialization, over-broad permissions.

Security-critical code (auth, IAM, encryption, data handling) gets human senior review, full stop.
Run scanners in pre-commit: ruff, bandit, checkov.
Check for phantom dependencies. Verify every new import actually exists in your lockfile. Attackers register packages under names AI models commonly hallucinate ("slopsquatting") — an import that "looks right" can be a supply-chain attack.

13. Definition of Done for AI-Assisted Changes

Because AI is probabilistic, "done" needs a verification loop. My PR checklist:

[ ] Small diff, scoped to the ticket — no drive-by refactors.
[ ] Contract alignment — verified against pyproject.toml (deps) and conftest.py (test patterns).
[ ] No phantom dependencies — every new import exists in the lockfile.
[ ] Tests added/updated — happy path plus at least two edge cases.
[ ] Local verification passed — uv run pytest / npm test ran and passed.
[ ] Lint & type checks pass.
[ ] Facts-only docs — updated from the final code, not from intentions.
[ ] Security gut-check — no secrets, no unjustified wildcards, no PII in logs.
[ ] The explain-back test — can you explain why the AI chose this logic? No black-box code allowed.

Automate the tail end by ending agent prompts with:

"When finished, provide a verification report: 1) files changed, 2) the exact command you ran to test, 3) confirmation that no new libraries were added."

14. Going Deeper

Learn to verify, not just to prompt. The skills that matter most for auditing AI output:

Advanced pytest — fixtures, parametrize, monkeypatch. The AI writes test logic well; humans must design the test strategy.
Fine-grained CDK assertions — snapshots are brittle and AI defaults to them; targeted assertions make infrastructure tests survive refactors.
Secure-by-design IAM — you need to spot an over-privileged suggestion at a glance.
Legacy refactoring patterns — seams, the strangler-fig pattern, working effectively with untested code. Large AI refactors need stepwise human strategy.

References worth your time: the VS Code Copilot documentation and agents guide, plus Anthropic's prompt-engineering docs — most of it transfers directly to any coding assistant.

15. Expert-Level Pitfalls

Avoid vibe coding. Never accept code because it looks clean and the AI sounds confident. Run the tests. Confidence is a UI feature, not a correctness signal.
Combat context rot. Models get less sharp as the window fills. /compact often; new chat per ticket.
Use cross-repo awareness deliberately. @workspace can trace how service A's schema change breaks service B — ask for the blast radius before you change shared contracts.

Appendix: Quick Prompt Snippets

Diff control

Minimal diff: "Change ONLY the code inside the target function. No unrelated refactors, no reformatting, no import changes unless strictly required."

No new dependencies: "Use only libraries already in @pyproject.toml / @package.json. If a task requires a new one, stop and ask."

Ask before guessing: "If unsure about a repo convention or internal API, ask a clarifying question. I prefer a question over a hallucination."

Python & CLI

Type-strict: "Implement with full type hints; explicit return types everywhere."

Click hygiene: "Every @click.option gets a help string and a defined type, matching the existing help-text style."

Infrastructure

Least-privilege IAM: "Generate the IAM policy with no wildcards for actions or resources. Use specific ARNs."

CDK assertions: "Test with the assertions library, focusing has_resource_properties on encryption and public-access settings. No snapshots."

Quality & debugging

Manual verification checklist: "Give me a checklist to verify manually: happy-path test, an error state to trigger, and logs to monitor."

Log forensics: "Analyze this stack trace against #file. Give the most likely failing line and 3 ranked hypotheses."

Blast radius: "If I change the return type of get_user_auth in @auth.py, trace all downstream dependencies across @workspace and list every file needing a signature update."

This playbook is a snapshot of 2026. The models will change; the discipline — anchor, constrain, verify, never blind-merge — won't.

uv: One Tool to Rule Your Python Workflow

Sun, 11 Jan 2026 00:00:00 GMT

Every Python developer eventually accumulates the same tool zoo: pyenv to install Python versions, venv to isolate projects, pip to install packages, pip-tools to lock them, pipx to run CLI tools, and maybe Poetry to hold the whole thing together. Each tool is fine. The pile is not — six tools, six config styles, six ways for CI to differ from your laptop.

uv, from Astral (the makers of ruff), replaces the pile with one tool, written in Rust, and it is fast — dependency resolution and installs are routinely 10–100× quicker than pip with a warm cache. This post covers what it replaces, how to start, and how to migrate incrementally. (It pairs well with my tour of what's new in Python 3.13 and 3.14.)

What uv actually is

One binary that can:

Install and manage Python versions — no more pyenv, no more system-Python roulette.
Create and manage isolated environments per project, automatically.
Manage dependencies with a universal lockfile — add, remove, lock, sync.
Run tools in isolation — linters and formatters without polluting your project env.
Speak pip — a uv pip interface for incremental adoption with zero workflow change.

The kitchen analogy. If your development environment is a kitchen, the classic setup has a separate gadget for every job — one machine to chop, one to time, one to weigh. uv is the multifunction appliance: one interface, every job, and it happens to work faster than the gadgets it replaced.

flowchart TD
  A["pyenv<br/>Python versions"] --> U["uv<br/>one fast binary"]
  B["venv<br/>environments"] --> U
  C["pip<br/>packages"] --> U
  D["pip-tools<br/>lockfiles"] --> U
  E["pipx<br/>isolated CLI tools"] --> U
  F["Poetry<br/>project management"] -->|"mostly"| U

Quick start

# Install and pin a Python version for your project
uv python install 3.14
uv python pin 3.14

# Initialize a project, add dependencies, lock, sync, test
uv init myproj && cd myproj
uv add fastapi uvicorn
uv lock
uv sync
uv run pytest -q

# Run tools without touching your project environment
uvx ruff check .
uv tool install black
uv tool list

# Or use pip-style commands for incremental adoption
uv venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
uv pip freeze > requirements.lock.txt
uv pip sync requirements.lock.txt

Two commands do most of the daily work: uv add <package> (updates pyproject.toml, resolves, locks, and syncs in one step) and uv run <command> (runs anything inside the project environment, creating it on demand — you may never type source .venv/bin/activate again).

The replacement cheat sheet

Traditional tool	What it did	uv equivalent
pyenv	Install/switch Python versions	`uv python install`, `uv python pin`
venv / virtualenv	Create isolated environments	`uv venv` (or implicit via `uv run`/`uv sync`)
pip	Install and uninstall packages	`uv pip install`, `uv pip uninstall`, or `uv add`/`uv remove`
pip-tools	Lock and synchronize dependencies	`uv lock`, `uv sync`, `uv pip sync`
pipx	Run CLI tools in isolation	`uvx`, `uv tool install`
Poetry (mostly)	Project deps, lockfiles, scripts, publishing	`uv init`, `uv add`, `uv lock`, `uv run`, `uv build`, `uv publish`

A few notes on the edges:

pipx → uvx is the easiest win: uvx ruff check . downloads, isolates, caches, and runs in one shot.
Poetry "mostly": uv covers dependency management, lockfiles, scripts, workspaces, building, and publishing. If you rely on Poetry plugins, check for equivalents before switching.
The lockfile (uv.lock) is cross-platform and universal — one lockfile resolves for macOS, Linux, and Windows simultaneously, which ends a classic CI-versus-laptop drift.

Who gets what out of it

Platform / infrastructure teams: reproducible, predictable builds — the lockfile plus pinned Python version means "works on my machine" finally means something.
DevOps / CI: fewer tools to install in pipelines, and dramatically faster installs. A cold uv sync frequently turns a minutes-long pip step into seconds.
SREs: isolated tool execution (uvx) means diagnostic tools never contaminate an app environment.
Developers: one mental model, one config file, no environment-activation rituals.

An incremental migration path

You don't need a big-bang migration. This is the sequence I recommend:

Pilot — pick one small project. uv init, uv add its dependencies, run its tests with uv run pytest. You'll learn the model in an hour.
Replace pipx first — switch your global tools (ruff, black, pre-commit) to uvx/uv tool install. Zero risk, immediate speed.
Adopt pip-compatibly — keep your existing requirements.txt workflow, but run it through uv pip install/uv pip sync. Same files, much faster, no team retraining.
Move to lockfile workflows — when the team is comfortable, migrate to pyproject.toml + uv lock + uv sync as the source of truth, and update CI to uv sync --frozen.
Pin Python per project — uv python pin ends the "which Python is this using?" class of bugs.

Caveats

uv is developed at a rapid pace — pin the uv version in CI (uv self version locally, a pinned installer version in pipelines) so a tool upgrade never surprises a deploy.
Some Poetry-specific workflows (custom plugins, some dynamic-versioning setups) need rework rather than translation.
Corporate proxies and private indexes work well (UV_DEFAULT_INDEX / UV_INDEX, or [[tool.uv.index]] in pyproject.toml), but test them in your pilot before rolling out broadly.

The bottom line: uv consolidates pyenv, venv, pip, pip-tools, pipx, and most of Poetry into a single fast, coherent utility. Of everything that's happened in Python tooling in the past few years, this is the change with the highest payoff-to-effort ratio — the pilot costs you an afternoon, and it's the rare tool that's simultaneously simpler and faster than what it replaces.

For the full documentation, see the official uv repository and docs.

What's Actually New in Python 3.13 and 3.14

Sat, 10 Jan 2026 00:00:00 GMT

Python 3.13 (October 2024) and 3.14 (October 2025) are the two most consequential releases the language has had in years. Between them, they touch the thing people have complained about for two decades — the Global Interpreter Lock — and quietly modernize half of the daily developer experience: the REPL, error messages, type annotations, string templating, debugging, and compression.

This post is a plain-language tour of what changed and what it means in practice. If you also want to modernize your tooling, read the companion post on uv, the tool that replaces pyenv, pip, venv, and pipx.

The ten things worth knowing

Python 3.13 shipped a redesigned interactive shell — colors, multi-line editing, sane shortcuts — and even friendlier error messages with "did you mean…?" suggestions.
An experimental 3.13 build removes the GIL (free-threading), enabling real parallelism across CPU cores; in 3.14 free-threading became officially supported.
JIT compilation landed in 3.13 as an experimental, off-by-default foundation for future speedups.
3.14 makes type annotations lazy (deferred evaluation), improving startup time and killing a whole class of annotation workarounds.
Subinterpreters — multiple isolated Python interpreters in one process — are now usable from Python code, giving you another concurrency option.
Template strings (t-strings) in 3.14 make dynamic content — HTML, SQL — safer to build than f-strings.
The incremental garbage collector shipped in 3.14.0 to reduce pause times in large, long-running applications — but it was reverted in 3.14.5 (June 2026) back to the 3.13-style generational GC after memory-pressure regressions, with possible reintroduction in a later release. (Accurate at publication; corrected since.)
Zstandard (zstd) compression is now in the standard library — faster and tighter than gzip for most workloads.
Live-process debugging: 3.14 lets you attach a debugger to a running Python process — including, carefully, in production.
Both releases removed long-deprecated modules — the "dead batteries" cleanup — so legacy code needs a checkup before upgrading.

Python 3.13: quality of life, plus two experiments

The new REPL and better errors

The interactive shell finally caught up with the decade: multi-line editing that works, color-coded output and tracebacks, exit that just exits. Error messages continued their glow-up — misspell a keyword argument or a module attribute and Python now suggests what you probably meant. Small thing, huge cumulative time savings, especially for people learning the language.

Free-threading: the GIL becomes optional

The Global Interpreter Lock has always meant one thread executes Python bytecode at a time — multithreaded Python could wait in parallel but not compute in parallel. Python 3.13 shipped a separate experimental build (python3.13t) with the GIL removed.

The bridge analogy. Picture a bridge that only ever allowed one vehicle to cross at a time, no matter how many lanes it had. Free-threading opens all the lanes. But some older vehicles — C extensions written assuming the GIL — aren't certified for the new bridge yet, so adoption needs care.

In 3.13 this was strictly experimental: single-threaded code ran noticeably slower on the free-threaded build, and many C extensions weren't compatible. In 3.14, free-threading was promoted to officially supported (PEP 779) — still a separate build, but no longer a science project: the single-threaded penalty shrank substantially and the major scientific and web libraries have been landing support. If you have genuinely CPU-bound multithreaded workloads, 3.14's free-threaded build is now worth a real evaluation.

JIT compilation: the foundation is laid

3.13 added an experimental just-in-time compiler (off by default, enabled at build time). Gains are modest so far — this release was about landing the architecture, not the speed.

The barista analogy. A new barista makes your regular order by reading the recipe every time. After a hundred repetitions, they've internalized it and it's faster. That's the JIT: it watches which code paths run hot and compiles them to faster machine code as the program runs.

Expect this to compound over the next several releases rather than deliver a big bang.

Housekeeping

3.13 completed the "dead batteries" removal (PEP 594) — nineteen crusty stdlib modules (cgi, telnetlib, pipes, …) are gone. It also made frame locals behave predictably (PEP 667), which matters if you use debuggers or clever locals() tricks.

Python 3.14: the features release

Deferred annotations: type hints stop costing you at startup

Until now, Python evaluated every type annotation eagerly at import time — whether or not anything ever looked at them.

The buffet analogy. Imagine preparing a full buffet every morning regardless of whether anyone eats. Deferred annotations (PEP 649/749) cook the dish only when someone orders it: annotations are stored lazily and evaluated on access.

Practical wins: faster imports, no more from __future__ import annotations dance, forward references that just work, and the end of quoting-your-own-class-name in method signatures.

Subinterpreters: concurrency without shared chaos

PEP 734 exposes multiple isolated interpreters inside a single process to pure Python code (concurrent.interpreters), each with its own GIL.

The office analogy. A large building where each team works in its own soundproof room — no interference, shared infrastructure. That's subinterpreters: isolation like multiprocessing, but cheaper, in one process.

Together with free-threading, Python now has a genuine menu of concurrency options: asyncio for I/O-bound, free-threading for shared-memory parallelism, subinterpreters for isolated parallelism, multiprocessing when you want OS-level separation.

Template strings: f-strings that don't blend everything together

PEP 750 adds t-strings: t"Hello {name}" produces a template object instead of a final string, so the consuming code can see — and safely escape — each interpolated value before assembly.

The smoothie vs. LEGO analogy. An f-string is a smoothie: once blended, you can't separate the ingredients, so if one of them was user input headed for SQL or HTML, it's already too late. A t-string is LEGO: every block stays visible and inspectable until you decide how to snap them together.

This is the foundation for injection-safe HTML templating and SQL building in libraries — expect frameworks to adopt it broadly.

Debugging, memory, and compression

Attach to a live process (PEP 768): a zero-overhead-when-idle interface lets debuggers safely attach to a running Python program — pdb can now connect to a live PID. For diagnosing a wedged production process, this is gold.
Incremental garbage collection: 3.14.0 introduced an incremental collector for shorter GC pauses and smoother latency on large heaps. Update: this was reverted in 3.14.5 (June 2026) — memory-pressure regressions led the core team to restore the 3.13-style generational collector, with the incremental approach possibly returning in a future release. This post was accurate when published in January 2026; on current 3.14 (3.14.5+) you're back on the generational GC. See the discussion on reverting the incremental GC.
compression.zstd (PEP 784): Zstandard in the stdlib — dramatically faster than gzip at comparable or better ratios, with tarfile/zipfile/shutil integration.
The REPL got syntax highlighting, and error messages got another round of polish.

How I'd adopt this

Upgrade path: if you're on 3.11/3.12, target 3.13 or 3.14 directly — the upgrade friction is mostly deprecated-module removals. Run your test suite with -W error::DeprecationWarning first.
Try the new REPL and error messages — no action needed, they're the default.
Deferred annotations: on 3.14, delete from __future__ import annotations and un-quote forward references as you touch files.
Free-threading: evaluate the 3.14 free-threaded build on a CPU-bound workload in a branch. Check your C-extension dependencies for compatibility first.
t-strings: adopt where you build HTML or SQL by hand; watch your web framework's release notes for native support.
zstd: swap in for gzip anywhere you compress logs, artifacts, or datasets.

Gotchas to check before upgrading

Code importing removed "dead battery" modules (cgi, telnetlib, imghdr, …) breaks on 3.13 — most have drop-in PyPI replacements.
Tools that poke at frame locals (custom debuggers, profilers) may need updates for the PEP 667 semantics.
Custom GC tuning (gc.set_threshold calls) deserves a re-benchmark after any upgrade. Note that the incremental collector added in 3.14.0 was reverted in 3.14.5, so current 3.14 is back on the generational collector — if you tuned for the brief incremental window, re-check against generational behavior.
The free-threaded build requires compatible wheels — a dependency without a cp314t wheel means compiling from source or waiting.

Both releases reward the same habit: upgrade early in a branch, run the tests loudly, and read the "What's New" documents — they're genuinely well-written.

Riddam Jain — Engineering

Self-Service With Guardrails: Repositories and Pipelines as Data

Two registries, one source of truth

Validate before you touch anything

Reconcile, don't create

Dry run is a feature, not a flag

Compliance as a report before it's a gate

Retirement is part of the lifecycle

The unglamorous parts that make it usable

What this actually buys

Rebuilding CI/CD Without Changing Platforms

Two agent tiers, and why

Scale on the queue, not on CPU

Don't let the CI system deploy itself

The base image will drift unless something stops it

Scan the images, and mean it

Make the server disposable

Credentials that expire, scoped by environment

What's still unfinished

What I'd keep

Safe Rollouts for Stateful Cloud Infrastructure

Principle 1: Optimize for Recoverability, Not Speed

Principle 2: Separate Validation from Deployment

Principle 3: Use Environment Promotion Discipline

Principle 4: Treat Config Changes as High Signal

Principle 5: Enforce Manual Gates for High-Risk Changes

Principle 6: Decide Your Availability Target Before You Design the Rollout

Principle 7: Rotate the Credentials You Depend On

A Lightweight Risk Matrix

Further Reading

Final Takeaway

Network Connectivity Patterns for Managed Database Platforms

Two Connectivity Models

Model A: Transit-Centric

Model B: Direct Peering by Consumer Domain

Decision Criteria

A Migration That Paid for Itself

The Ideal Path vs. the One You Can Afford

Routing Guardrails That Prevent Incidents

The Prerequisite That Blocks Everything

A Two-Phase Rollout That Avoids the Route Race

Further Reading

Final Takeaway

Multi-Cloud, Done Deliberately: Strategy, and Choosing Between Terraform and Pulumi

Part 1 — Is multi-cloud even the right call?

The patterns, honestly

What this looks like in practice

Part 2 — Why IaC matters more in multi-cloud

Part 3 — Terraform and OpenTofu: the fork you need to understand

Part 4 — Pulumi: infrastructure in a language you already know

Part 5 — How I'd choose

Part 6 — The part that matters more than the tool

Part 7 — From image to running infra: the delivery pipeline

Final thoughts

Path-Filtered CI/CD for Infrastructure Monorepos

The Problem

The Design

What a Path Filter Actually Looks Like

Why Manual Production Gates Still Matter

Suggested Stage Layout

Ownership Model

Failure Modes to Watch

Further Reading

Final Takeaway

Config as Data: A Safer Pattern for Multi-Environment Infrastructure

The Core Rule

Why This Works

A Practical Structure

Guardrails You Should Add Early

CI/CD Integration

Where State and Secrets Live

Common Anti-Patterns

Further Reading

Final Takeaway

Making AI Coding Assistants Reliable: My Playbook

TL;DR

1. What an AI Assistant Is (and Isn't)

What it's good at

What it's not

2. Modes & Sessions: Choosing How It Thinks and Where It Works