Runs wey you fit trust

Nobody go notice one automation until e send something two times

Den everybody go notice. Di four mechanisms wey dey dis page dey exist because di failures of dis kind of software na always di same four failures, and design fit avoid all of dem, no be by care.

Simulate am before you publish am

Push one real record through one workflow wey dem switch off, and read back exactly wetin for happen: di path wey e take, every value for every step, di wording of every message, and di request wey for go to each external system.

Nothing go comot from di building. No e-mail go send, no invoice go raise, no charge go attempt, no row go write. Dem go record external calls as intentions, with di payload wey for send, so you go fit see whether di mapping correct before e become somebody problem.

Simulation na how you go answer di question wey dey stop people from switching automation on: wetin e go really do?

Simulation · order 88 214Dem no send anything

1Check addressE match. E for write 6 fields.
?ConditionOrder pass 1 000 zł? No — na 289,00 zł. E take di short path.
3InvoiceE for do POST to di invoicing system. Di payload dey show in full.
!Dem catch am hereDem map VAT rate from di wrong field. Dem find am inside simulation instead of inside accountant inbox.

Wetin dem send: nothingWetin dem charge: nothing

One run dey walk di version wey e enter on

If you publish change to one workflow, e no go touch di runs wey already dey inside am. Dem pin each run to di version wey e start on and e go finish dat version, no matter how long e take and no matter how much you rewrite for di meantime.

So you fit restructure one workflow while ten thousand people dey halfway inside am, and every one of dem go complete di flow wey dem promise dem. Di other way — di design where dem materialise di whole downstream path at entry — na di reason why to edit one live automation dey feel dangerous for most tools. Here e no be anything.

New runs go get di new version from di moment you publish. Dem go write which version one run dey on for di run itself, so if you ask about behaviour for March, answer go dey for September.

To edit while e dey livev4 → v5

1Bifor2 918 runs dey in flight, all of dem on v4. Some dey wait two days.
2Publish v511:40Dem reorder two steps, dem rewrite one message.
3AftaDat 2 918 go finish v4. Everything wey start from 11:40 go run v5.
§Nobodyreceive half of one workflow and half of anoda.

Dem record version on every runDem keep old versions

Retries wey no fit send two times

One step na pure function of im inputs, and im effects commit togeda with di run own move to di next step. No window dey where di message don go out but di run no know am.

Na di whole trick be dat, and na structural property e be, no be promise. If worker die mid-step — deploy, crash, or machine wey vanish — di run go stay exactly where e dey. Anoda worker go pick am up and re-run di step. Because di effect and di advance commit as one, na either both happen or none happen.

Outbound calls to systems wey support idempotency keys dey carry one, wey dem derive from di run and di step, so di oda side sef go recognise di repeat.

Worker die mid-sendRecovered

1Step 5 start09:14:06Send di shipping notice.
✕Worker die09:14:07Deploy. Nothing don commit.
2Picked up09:14:31Anoda worker see say di run still dey step 5.
✓Sent once09:14:32Di customer receive one e-mail, no be two.

Effects commit join di advanceIdempotency key for each run and step

Log wey dem write make person fit read am

Every run, every step, di branch wey e take and di value wey decide am. Timings per step. Di request and response of each external call, with credentials redacted. Di version wey di run dey on. Where di run dey wait and until when.

When somebody ask wetin happen to di order wey dem place on di fourteenth, answer dey, and to find am na one minute e take, no be whole afternoon of matching three systems' logs.

Runs wey fail gather for one place with di reason grouped, so “forty-one failures overnight” go resolve to “one supplier return 503 for nine minutes” instead of forty-one separate investigations.

Overnight, grouped41 failures

1One causeSupplier API return 503 between 02:41 and 02:50.
2Wetin happenAll 41 dem retry; 39 succeed on di second attempt.
3Left overTwo stop and dey wait for person. Dem list both with di row wey fail.
§No be forty-oneinvestigations. Na one, plus decision about two records.

Credentials redactedVersion recorded

Wetin we no dey claim

We no dey publish uptime figure for marketing page, because number wey no get measurement window and definition of “up” na decoration. If you need one contractually, ask and we go talk about wetin we fit actually commit to.

We sef no fit make anoda person API reliable. If provider down, workflow wey depend on am go wait, retry and eventually stop with clear reason. Wetin we fit promise na say di failure dey visible, say e no spoil anything, and say to resume no go send di message second time.

On waits: run wey dey wait for two days na wake_at on di run, no be job wey dey sit for queue. Na why workflow fit hold hundred thousand runs inside three-week wait without any of am costing anything, and why di number of runs wey dey wait now no be something you go need think about.