Stop Pinning Everything: Quantifying Upgrade Risk in Durable Workflows
All durable execution engines give the same versioning advice: pin your runs. Temporal pins runs to worker builds. Vercel's Workflow SDK pins each run to the deployment that started it. Azure Durable Functions recommends deploying side by side and draining old runs. The methods vary, but the idea is the same: treat every code change as risky for every in-flight run, and keep the old version running until all runs finish.
If your workflows finish in seconds, pinning just means a bit of wasted capacity. But for workflows that last weeks, like subscription lifecycles, compliance timers, drip campaigns, or human-in-the-loop approvals, pinning can prevent a critical bug fix from ever reaching a stuck run. You find the bug, write the fix, deploy it, but the runs that need it most never get it. That’s not a safety guarantee; it’s a liability that never goes away.
We’ve published a paper that replaces this overly cautious approach with a measurable number: the Workflow Upgrade Risk (WUR) model. This model uses a static diff and existing engine telemetry to score the risk of upgrading each in-flight run from V1 to V2. There’s no need for dry-runs, sandboxes, or shadow execution. In our tests, it classified 64,920 runs with zero false negatives and can score 50,000 in-flight runs in just 0.72 seconds on a single core.
Below we will cover:
Why deploying over in-flight durable workflows is dangerous in the first place.
Why the current mitigations (pinning, patch gates, side-by-side) are uniformly pessimistic
How the WUR model turns the event log into an analytical asset
The evaluation: soundness, precision, calibration, and speed
Let’s get started.
The determinism contract and its corollary
Here’s a quick refresher on how durable execution works. Engines like Temporal, Azure Durable Functions, Restate, and the Workflow SDK keep an append-only log of events, such as step invocations, results, timers, and external signals. If a process needs to resume after a crash, suspension, or redeployment, the engine re-executes the orchestration function from the start and uses the recorded results for completed steps. The run doesn’t repeat side effects; it just replays decisions. This works because of a determinism contract: if the event history is the same, the orchestration code must always make the same decisions. There’s an important point that often gets overlooked: the event log only makes sense in the context of the code that created it. If you deploy V2 while runs started under V1 are still active, you can run into three types of failures:
Replay divergence. V2 reaches a different step than the log expects because a step was inserted, removed, or reordered. Best case, the engine throws a hard nondeterminism error. Worst case, it silently substitutes the wrong recorded result into the wrong call site.
Rehydration failure. A recorded payload no longer satisfies the shape V2's downstream code consumes. A field was removed, renamed, or its type narrowed, and the run faults mid-replay.
Behavioural drift. Replay succeeds, but the steps not yet executed behave differently under V2. The run completes with semantics neither version's author intended.
The rules that cause these failures are stricter than you might expect. In engines that use ordinal identity, each site is identified by a pair: the global invocation order and the name. This sequence is shared across steps, timers, and hooks. If you add a step at the end of a workflow, every recorded event still matches up, so nothing breaks. But if you add a step at the beginning, every following ordinal shifts, and replay diverges right away for any run that has already started. The same change can have opposite effects, and you won’t notice unless you check each run.
If you replay the same recorded log against two versions, adding a step at the end keeps all the ordinals matched. But adding a step at the beginning shifts every ordinal after it, causing replay to diverge on the first event.
Why pin-by-default is the wrong default
The state of the art here is operational, not analytical. Temporal offers patched() markers and worker build IDs. The Workflow SDK pins by default and offers a deploymentId: "latest" escape hatch that explicitly delegates the compatibility judgment to you. Restate's docs enumerate the unsafe change classes and then leave the per-run decision to human judgment.
These methods are reliable, but they’re also overly cautious. They assume every change is incompatible with every in-flight run, even though most changes aren’t. For example, editing the body of a step doesn’t affect replay. Adding a step at the end doesn’t break anything. Consuming a new field only causes problems for runs whose recorded payload is missing that field, and you can check that in your database right now.
This cautious approach adds up over time. Temporal’s patched() helps with mixed-version execution, but patches pile up, and you can only remove a patch when you’re sure no in-flight run can still use the old branch. That’s a probability question about your fleet, but no one calculates it today. Meanwhile, every pin-by-default deployment keeps old versions running, building up what the paper calls forgone-fix exposure: the chance that a stuck run hits the exact bug V2 was meant to fix.
Here’s the question no one was answering: given the difference between V1 and V2, and the telemetry your engine already saves, what’s the chance that upgrading a specific in-flight run will fail?
Turning the durability guarantee into a risk model
Our main insight is that event-sourced workflow engines are especially well-suited for analyzing change risk. In most distributed systems, risk models have to guess interaction patterns from samples. But here, the protocol saves complete histories, every path taken and every payload produced, because of its durability guarantee. You’ve already paid for the storage, and the WUR model turns that into a valuable analytical tool.
There are two separate tracks: the static track classifies the diff without running any code, and the telemetry track rebuilds each run’s state without looking at the source. At no point does the system replay a log against V2.
Risk decomposes into two components that differ in kind:
Backward risk: can the recorded history rehydrate under V2? This is computed exactly, not estimated. The run's prefix is fully known, so the model checks whether the prefix maps to a valid path in V2's step graph and validates every recorded payload against V2's actual demands. In the exact regime, backward risk is a verdict in {0, 1}, and the paper proves a certification theorem: a zero backward-risk verdict guarantees that replay under V2 succeeds and suspends at the matched frontier with the same substituted values. Not “probably fine.” Proven safe.
Forward risk: will the remainder of the run traverse changed code? This part is genuinely probabilistic, so the model estimates it honestly. Complete path histories yield an empirical Markov chain over the step graph; hitting probabilities from the run's current frontier to each changed site, weighted by a severity estimated from telemetry (retry logs give you an empirical idempotency measure, deployment history calibrates severity per change class), compose into a forward score.
The model uses Bayesian estimation, so every WUR score comes with a credible interval. If you don’t have much telemetry, the model won’t give you a confident but wrong answer—it will just give you a wide interval. That’s useful on its own: it tells you there isn’t enough data to safely approve this migration.
Two design choices make a big difference:
No code execution, ever. The pivotal substitution is that payload compatibility is not a property of code paths you must exercise. It is a measure-theoretic property of the observed payload distribution against a static contract. If V2 newly destructures field x, the incompatibility rate is exactly the recorded absence rate of x. Data validated against a schema, no sandbox required.
Precision spent only where the diff spends risk. The differential-demand lemma says that if a consumer slice is unchanged between versions, history itself is the compatibility proof: the run already consumed that payload with that exact code and did not fault. Contract inference is confined to the handful of access paths a release actually changes.
Migrate, review, pin
The scores support a three-way policy. Runs with zero backward risk and very low forward risk can move to V2. Runs with some but uncertain forward risk go to review, ranked by expected loss. Runs with any backward risk are pinned, but now pinning is a measured cost, not the default. You can calculate expected drain time from the same Markov chain, so if a release leaves runs stuck behind a 90-day sleep, you’ll see that risk up front, not as a surprise later.
A worked example from the paper makes the granularity concrete. A linear-order workflow (validate, charge, sleep 24h, notify, archive) ships a V2 in which archive now consumes charge.promoCode, a field present in 56.7% of historical runs. Four in-flight runs, one change, three different verdicts:
Run (state at suspension) | Backward | Forward | WUR | Verdict |
r1: before charge | 0.00 | 0.43 | 0.43 | review |
r2: charged, promoCode recorded | 0.00 | 0.00 | 0.00 | migrate |
r3: charged, promoCode absent | 0.00 | 1.00 | 1.00 | pin |
r4: complete prefix, promo present | 0.00 | 0.00 | 0.00 | migrate |
One change, four runs, three different outcomes. r2 and r3 are at the same point, but the verdict is different because the model checks each run’s recorded state.
r2 and r3 are at the same point under the same change, but the difference comes from their recorded payloads, which the model checks. Only r1, which needs a payload that doesn’t exist yet, has real probabilistic risk (0.43 is the measured absence rate). So instead of pinning every run, this rollout only leaves one run stranded.
There’s another layer to consider, since production fleets aren’t just collections of independent runs. Workflows can signal each other, create child runs, and share resources. Any real-world split between migrate and pin creates a window where connected runs are running different code but sharing channels. The model builds a coupling graph from tokens already in the logs, tracks how failures can spread, and calculates the best migrate/pin split using a minimum s-t cut. Saga partners only cross versions when the costs make sense, just like a careful human operator would, but now it’s computed, not guessed.
The numbers
We validated the model three ways: a controlled scenario suite, a mutation study, and a hand-translated corpus of real workflows from the Workflow SDK's own repository (45 workflow functions spanning sagas, fan-out, human-in-the-loop approval, parent-child hierarchies, webhooks, and durable agents).
The headline results:
Metric | Result |
False negatives (synthetic, 64,920 verdicts) | 0 |
False negatives (real corpus, 18,160 verdicts) | 0 |
Precision, synthetic mutations | 0.815 |
Precision, real-workflow corpus | 0.933 |
Calibration error, distributional regime | 0.016 ECE, 0.92 correlation |
Backward risk per run | 9 to 15 µs |
50,000-run fleet classification | 0.72 s, one core |
A recall of exactly 1.0 is what the certification theorem requires, and the evaluation is clear about what that means: it checks the implementation against verified replay behaviour, not against failures outside the model’s scope. Every false positive has a clear cause, usually from conservative predicate matching, and each cause can be targeted for annotation. For example, predicate-equivalence hints alone would remove 86% of the synthetic false positives. Most importantly, the model only errs in one direction. A false positive just pins a run that was actually safe, costing you some drain time. A false negative would corrupt state. The design chooses the first to make the second impossible, and then measures what that caution costs.
The calibration result is important for teams that don’t keep full payloads. Just knowing how often a field is present gives a good risk estimate, and you can get useful predictions with just tens of completed runs, not thousands.
Why we built this
This isn’t just theory. Platformatic ICC already uses version-aware traffic control and skew protection for Node.js fleets, and our predictive autoscaler (which we covered in our previous deep dive) is based on the same idea: infrastructure decisions should be driven by real measurements from your system, not by static thresholds or worst-case guesses.
Autoscaling and upgrade management face the same problem today. Reactive scalers treat every metric snapshot separately, and pin-by-default versioning treats every deploy as extremely risky. Both approaches waste money because they lack a real model. The solution is the same for both: use the telemetry you already have to make informed decisions instead of guessing.
For durable workflows, the costs are clear. Every pinned run means you’re paying for old-version capacity and still exposed to bugs you’ve already fixed. The WUR model puts a price on both, for each run, before you deploy. It also lets you split releases, since risk is provably sub-additive across changes. This means you can break up a release into smaller deploys, each within a risk budget. The common wisdom that small, frequent deploys are safer is now backed by theory.
What's next
The reference implementation is a single-file analyzer that only needs run listings and event logs. It comes with the paper, the full experiment suite, and the hand-translated corpus. The most valuable experiment is one we can’t do alone: validating the model against real production deployment histories by replaying past releases with known outcomes. That telemetry is with platform operators, which is exactly where the model is meant to be used.
If you run a durable execution platform or manage long-lived workflows and are tired of choosing between stranded runs and risky deploys, we’d love to hear from you.
Drop us a note at hello@platformatic.dev or reach out on LinkedIn.
Thank you, and happy building!
