Ten steps at 99% is 90%: retrying multi-step jobs
Part 4 of 10: Reliable background jobs in Node.js, from zero. By Sandro Dezerio, senior full-stack engineer. The signup scenario is made up. It is the same one used in Part 1, Part 2 and Part 3. The failure rates come from a simulation with stated assumptions, not from real services. A job with ten steps, where each step works 99% of the time, finishes about 90% of the time. Nobody looks at a step that works 99% of the time and calls it unreliable. That is why the problem hides. Each step looks fine on its own, and the job as a whole fails about once in every ten runs. In Part 3, every job was a single call: send one welcome email. Real jobs are rarely that small. This post looks at a job with several steps, shows what goes wrong when you retry it as one piece, and fixes it with the withRetry wrapper from Part 3, unchanged. Picture the signup job some time later. Sending the welcome…
