An Executable RPC Retry Budget: Bound Amplification Before Backoff

A client issues 100 logical reads during a complete upstream outage. Its wrapper allows three attempts, so it can produce 300 physical requests. Adding exponential backoff spreads those requests across time but leaves that maximum unchanged. The same failure that reduced useful capacity has created an extra source of work.

The distinction between a logical operation and a physical attempt is the starting point. One caller wants one answer before a deadline. The transport may make several attempts to obtain that answer. Capacity planning must count the attempts; user-facing completion metrics should count the operation. A system that silently merges the two can appear to recover because a greater fraction of attempts succeeds while each useful answer consumes more resources.

This article builds a deterministic retry controller for explicitly replayable reads. It computes real schedules, honors server pushback, and limits aggregate additional attempts. Its inputs are scripted durations and HTTP outcomes, so the experiment is reproducible without a live provider. The controller is an application policy, not an implementation of a JSON-RPC standard retry rule.

Derive an amplification bound before choosing a delay

Let O be the number of admitted original operations and R the number of additional attempts. Give the process initial credit b and earn one additional attempt for every ten originals. Then, ignoring discarded credit at the bucket cap, R is at most b plus floor(O/10). Physical work is O+R. With b=2 and O=100, the bound is 112 attempts, versus 300 with three attempts available independently to every caller.

The bound does not promise a successful answer. During a complete outage, all 112 attempts can fail. Its purpose is to constrain how much the client contributes to the outage. During partial failure, a small subset of operations can use the remaining credit to recover. Which operations receive credit is a scheduling policy; the sequential experiment below intentionally does not claim fairness among concurrent users.

Store credit in integer tenths. An original adds one unit; a retry costs ten. This avoids a floating-point balance accumulating small rounding differences over a long process lifetime. Bucket capacity is four retries, while initial balance is two. Idle capacity cannot accumulate without limit because every addition is capped. The cap limits bursts, and the refill ratio limits sustained amplification; these are separate controls.

A process-local budget also has a fleet boundary. Twenty processes each starting with two credits can collectively create forty initial retries. Repeated restarts replenish that burst. If the desired boundary is an entire application fleet, use shared accounting or allocate explicit per-process shares and include the restart allowance in the bound. A local mathematical guarantee cannot silently become a global one.

Eligibility precedes backoff

An HTTP outcome alone does not establish whether an operation is safe to replay. The example requires a safe decision from the caller and only considers 429, 503, and 504 transient. That allowlist is intentionally narrow. A real adapter should classify documented provider errors and transport failures separately rather than retry every non-success response. Here status 200 is a scripted success supplied by that classifier; a real HTTP 200 can contain a JSON-RPC error or malformed body and cannot be accepted on status alone.

For a contract simulation, a valid execution revert can be the requested answer. Invalid parameters require a different input. An authentication failure requires a credential decision. Repeating any of those because the HTTP wrapper used POST does not create progress. Conversely, POST does not make a side-effect-free read unsafe merely because it shares a transport method with transaction submission.

Transaction submission deserves its own reconciliation workflow. A timed-out submission can have reached the node. Preserve the signed bytes and inspect the transaction identity; do not construct another transaction as an incidental retry. Amazon's retry discussion supports the narrower principle that timeouts can leave side effects uncertain and retries can increase load. The specific credit algorithm and examples here are independently derived policy.

The attempt cap still matters after adding the aggregate bucket. A single operation cannot consume all the available credit through an indefinitely repeating loop. The implementation permits at most three attempts, including the original. It also has one 1,500 ms deadline, a 50 ms response reserve, and a minimum 20 ms useful attempt window. Those numbers are laboratory inputs, not recommended universal limits.

Backoff schedules an eligible retry

For retry index k starting at zero, compute an exponential ceiling min(800,100×2^k) milliseconds. Draw a delay uniformly from zero up to that ceiling. The demonstration injects a random function returning 0.5, producing 50 ms followed by 100 ms. Production randomness would disperse callers; fixing it here makes the arithmetic inspectable and the assertions repeatable.

Server pushback supplies a lower bound. RFC 6585 §4 allows a 429 response to include Retry-After. RFC 9110 §10.2.3 defines seconds and HTTP-date forms. The effective delay is the larger of jitter and the parsed server delay. A two-second pushback with only 990 ms left ends the operation; it does not become a shorter delay merely because the caller is impatient.

The parser accepts decimal seconds and an IMF-style date shape. It treats malformed input as absent policy information, clamps past dates to zero, and treats delays above a day as unusable for this interactive controller. Returning infinity causes deadline refusal rather than an immediate retry. This is a deliberate teaching profile, not a complete HTTP-date parser: it omits the two obsolete date forms that RFC 9110 requires HTTP recipients to accept. A production HTTP adapter should use a compliant HTTP-date parser, while retaining the deadline and overflow refusal policy.

Convert an accepted HTTP date into a duration using wall time at receipt, then schedule using elapsed time. Do not keep comparing an absolute wall-clock deadline while the system clock can change. The simulation has a fixed wall-time origin solely to test date conversion. Real requests need a monotonic operation clock and a separately obtained wall-time observation for the header.

Run the controller and its counterexamples

Save the following as retry.cjs and run node retry.cjs with Node.js 24.22.1. It uses only built-in modules and makes no network requests. Each scripted outcome provides a duration and status; the algorithm chooses whether and when another attempt exists. The successful schedule is therefore computed, rather than embedded as a canned final result.

'use strict';
const assert = require('node:assert/strict');
function retryAfter(value, wallNow) {
  if (value == null) return 0;
  const s = value.trim();
  if (/^\d+$/.test(s)) {
    const n = BigInt(s);
    return n > 86400n ? Infinity : Number(n) * 1000;
  }
  if (!/^[A-Z][a-z]{2}, \d{2} [A-Z][a-z]{2} \d{4} \d{2}:\d{2}:\d{2} GMT$/.test(s)) return 0;
  const n = Date.parse(s);
  return Number.isFinite(n) ? Math.max(0, n - wallNow) : 0;
}
class Credits {
  constructor(initial = 2, cap = 4) { this.units = initial * 10; this.cap = cap * 10; }
  original() { this.units = Math.min(this.cap, this.units + 1); }
  take() { if (this.units < 10) return false; this.units -= 10; return true; }
}
function run(script, {credits, deadline = 1500, rng = () => 0.5, safe = true}) {
  let now = 0, attempts = 0;
  const events = [];
  credits.original();
  while (true) {
    const available = deadline - now - 50;
    if (available < 20) return {reason:'deadline', attempts, now, events};
    const reply = script[Math.min(attempts, script.length - 1)];
    attempts++;
    if (reply.ms > available) return {reason:'attempt_timeout', attempts, now:now + available, events};
    now += reply.ms;
    events.push({attempt:attempts, at:now, status:reply.status});
    if (reply.status === 200) return {reason:'ok', attempts, now, events};
    if (!safe || ![429, 503, 504].includes(reply.status)) return {reason:'terminal', attempts, now, events};
    if (attempts === 3) return {reason:'attempt_cap', attempts, now, events};
    const cap = Math.min(800, 100 * 2 ** (attempts - 1));
    const jitter = Math.floor(rng() * cap);
    assert(jitter >= 0 && jitter < cap);
    const delay = Math.max(jitter, retryAfter(reply.after, Date.UTC(2026, 0, 1) + now));
    if (delay + 20 + 50 > deadline - now) return {reason:'deadline', attempts, now, events};
    if (!credits.take()) return {reason:'retry_budget', attempts, now, events};
    now += delay;
  }
}
const good = run([{status:503, ms:40},{status:429, ms:30, after:'1'},{status:200, ms:20}], {credits:new Credits()});
assert.equal(good.reason, 'ok'); assert.equal(good.now, 1140); assert.equal(good.attempts, 3);
console.log('recovery', JSON.stringify(good));
const delayed = run([{status:429, ms:10, after:'2'}], {credits:new Credits(), deadline:1000});
assert.equal(delayed.attempts, 1); assert.equal(delayed.reason, 'deadline');
assert.equal(run([{status:400, ms:1}],{credits:new Credits()}).attempts, 1);
assert.equal(run([{status:503, ms:1}],{credits:new Credits(), safe:false}).attempts, 1);
const credits = new Credits(); let physical = 0;
for (let i=0;i<100;i++) physical += run([{status:503, ms:1}],{credits}).attempts;
assert.equal(physical,112);
assert(physical <= 100 + 2 + Math.floor(100 / 10));
assert.equal(retryAfter('Thu, 01 Jan 2026 00:00:02 GMT',Date.UTC(2026,0,1)),2000);
assert.equal(retryAfter('999999999999999999999',0),Infinity);
assert.equal(retryAfter('-1',0),0);
console.log('pushback',delayed.reason,'attempts',delayed.attempts);
console.log('outage originals=100 physical=' + physical + ' unbudgeted_max=300');
console.log('date, overflow, terminal and unsafe-operation assertions passed');

The actual local output was:

recovery {"reason":"ok","attempts":3,"now":1140,"events":[{"attempt":1,"at":40,"status":503},{"attempt":2,"at":120,"status":429},{"attempt":3,"at":1140,"status":200}]}
pushback deadline attempts 1
outage originals=100 physical=112 unbudgeted_max=300
date, overflow, terminal and unsafe-operation assertions passed

The successful example begins with a 40 ms 503. Its first jitter adds 50 ms, so the second attempt starts at time 90 and finishes at 120. That response asks for one second of delay. Although the second jitter is only 100 ms, pushback wins. The final 20 ms success finishes at 1,140 ms, leaving the response reserve inside the original deadline. The event timestamps distinguish attempt completion from dispatch and make a mistaken “add Retry-After to jitter” implementation visible.

The failure example receives a two-second pushback after ten milliseconds. It stops after one attempt. A controller that ignores pushback, or truncates it to fit, would send an additional request and fail this test. The terminal-status and unsafe-operation assertions catch a wrapper that retries because a status looks temporary without first establishing operation eligibility.

The outage case shares one credit bucket across 100 independent operations. It produces 112 attempts. If each operation accidentally constructed its own bucket, the same fixture would permit 300. This is the important negative case: the assertion checks a cross-operation invariant that cannot be established by inspecting one retry loop. The large-seconds test also prevents numeric overflow from converting a long server delay into immediate retry eligibility.

Integrate the budget at one owner

The synchronous controller is an executable policy model. A production implementation must replace simulated time with cancellable timers and actual request completion. Reserve credit immediately before dispatching an additional attempt, after deadline feasibility is known. Otherwise a request can consume retry allowance and then discover that there is no time to use it. If cancellation occurs after reservation, choose explicitly whether the reservation is spent; a refund is safe only if dispatch did not occur.

The model spends a credit when a retry is scheduled and never refunds it. That conservative choice simplifies the bound. A more efficient implementation may hold a reservation until the delay ends and refund an undispatched attempt, but reservation ownership must survive cancellation races. The budget operation needs atomicity across callers; this single JavaScript loop demonstrates the accounting, not a distributed compare-and-update protocol.

Retry ownership must include hidden layers. If an SDK performs three attempts and a gateway repeats that SDK operation three times, nine upstream attempts can result before considering any outer application wrapper. Count attempts at the lowest layer you control, disable redundant loops where possible, and expose actual upstream attempt totals. Merely naming the outer counter “retry budget” cannot constrain work hidden below it.

A rolling window and a credit bucket answer different questions. The bucket limits additional work relative to recent original admission, subject to bounded stored credit. A time window can report observed amplification over a specific interval, including retries from operations that started earlier. During a traffic stop, that reported ratio may spike even while the bucket remains correct. Preserve operation start, attempt dispatch, and denial reasons so this boundary is explainable.

Do not use incoming retries themselves to earn new retry credit. Only an admitted original logical operation earns the fractional refill. Otherwise the budget becomes self-financing: retries generate allowance for further retries. Likewise, avoid earning credit for work rejected before admission, because an attacker could create budget without consuming the original capacity that the ratio assumes.

Finally, evaluate the policy with useful completion and physical work together. Record logical success before deadline, original count, additional attempts, refusal by deadline, refusal by credit, and server-pushback deferral. The experiment proves the arithmetic and branch behavior for its finite scripts. It does not measure Internet latency, provider support, cancellation effectiveness, or recovery probability. Those require a local transport fixture and later controlled deployment measurements, while the same amplification invariant remains available as a testable contract.