Handle retries and failures

You’ll make a webhook-delivery task retry temporary failures with backoff, give up on permanent ones, stop waiting on hung calls, and replay what failed once the cause is fixed.

Prerequisites

• A running worker (see Run workers).
• A result backend, to read failures with AsyncResult.

1. Choose which errors retry

import 'dart:async';
import 'dart:io';

import 'package:gisila_queue/gisila_queue.dart';
import 'package:http/http.dart' as http;

final deliver = app.task<int>(
  'webhooks.deliver',
  (ctx) async {
    final url = ctx.arg(0) as String;
    final http.Response res;
    try {
      res = await http.post(Uri.parse(url), body: ctx.arg(1) as String);
    } on SocketException catch (e) {
      ctx.retry(cause: e);
    } on TimeoutException catch (e) {
      ctx.retry(cause: e);
    }
    if (res.statusCode >= 500) ctx.retry(cause: 'HTTP ${res.statusCode}');
    if (res.statusCode == 410) throw const RejectTask(reason: 'subscriber gone');
    return res.statusCode;
  },
  maxRetries: 6,
  retryBackoff: const Duration(seconds: 5),     // ~5 s, 10 s, 20 s, 40 s, …
  retryBackoffMax: const Duration(minutes: 15),
  timeLimit: const Duration(seconds: 30),
);

• ctx.retry() re-enqueues with backoff while retries remain; it never returns.
• Any other exception (a 4xx you didn’t handle, a bug) fails immediately because autoRetry is off.
• RejectTask() stores revoked and doesn’t dead-letter: nothing to fix.
• timeLimit turns a hung call into a failure after 30 s.

2. Or retry everything

For simple idempotent tasks:

app.task<void>('cache.warm', warmCache, maxRetries: 3, autoRetry: true);

3. Enqueue through the TaskRef

await deliver.delay([subscriberUrl, payloadJson]);

delay/applyAsync carry maxRetries: 6 in the message. If you enqueue with app.send('webhooks.deliver', ...), pass maxRetries: 6 yourself.

4. React to terminal failures

In the producer:

final r = await deliver.delay([url, body]);
try {
  await r.get(timeout: const Duration(minutes: 30));
} on TaskFailureException catch (e) {
  print('gave up: ${e.message}'); // e.g. "retries exhausted (6): HTTP 503"
}

In the worker, without waiting on anything:

final worker = app.worker(concurrency: 16)
  ..onFailure = (msg, result) => alerts.send('${msg.task} ${msg.id}: ${result.error}');

5. Inspect the dead-letter queue

Failed messages land on dead_letter with the reason:

dart run gisila_queue:gisila_queue inspect --namespace=shop --queues=dead_letter

6. Replay after the fix

final moved = await app.replayDeadLetters(limit: 500);
print('replayed $moved');

Each message goes back to its original queue with retry counters reset.

Verify it works

Point the task at an endpoint that returns 503 and watch the worker log:

[gisila_queue] WARNING task webhooks.deliver ↻ retry 1/6 in 0:00:03.712000 id=...
[gisila_queue] WARNING task webhooks.deliver ↻ retry 2/6 in 0:00:08.115000 id=...
...
[gisila_queue] ERROR   task webhooks.deliver ✗ retries exhausted id=...
[gisila_queue] WARNING task webhooks.deliver id=... dead-lettered to "dead_letter"

Next steps

• Retries and failure handling for when to use which tool.
• Retries, acks and dead letters for the exact rules.