Queue
A queue runs slow work in the background, so the request can answer at once.
Some tasks take too long to do during a request, like sending an email, calling a slow API, making a report or resizing an image. You put these tasks on a queue as jobs. A worker process then takes each job and runs it.
The dox_queue package gives you jobs, retries, delays and a worker. It has a redis driver and a sync driver. The database driver is in the dox_queue_database package.
Setup
A new app from dox create already has the queue. In other apps, run:
The default driver is database. Add --driver=redis or --driver=sync to pick another one. The command adds the package and writes the config, the provider, bin/queue.dart and a sample job. It never overwrites a file, so it is safe to run twice.
With database, it also adds dox_queue_database and the migration for the jobs tables, so run dox add orm first. With redis, it adds the REDIS_* keys.
The sections below show the files it writes. Check them, and change the driver if you want. To set it up by hand, run dart pub add dox_queue and write the same files.
Configuration
The queue config picks the driver and lists your jobs:
import 'package:dox_queue/dox_queue.dart';
import '../generated/jobs.g.dart';
QueueConfig queueConfig = QueueConfig(
/// sync, database or redis
driver: 'database',
/// every @Job class, from lib/generated/jobs.g.dart
jobs: jobRegistry,
);
jobRegistry holds every job class of your app. dart run build_runner build writes it, because the job builder is in dox_builder, which your app already runs. See All options for the other settings.
Dox has three drivers:
sync:dispatch()runs the job at once and waits for it. There is no worker, no retry and no delay. Use it for tests and local work.database: jobs are rows in thejobstable. It needsdox_queue_database, and Postgres, MySQL 8+, MariaDB 10.6+ or SQLite.redis: jobs are kept in Redis.
Keep retryAfter bigger than your longest job
retryAfter (default 90 seconds) is how Dox finds the job of a crashed worker.
A job that runs longer than retryAfter looks crashed, so a second worker runs it again.
The provider
The provider sets up the queue in every isolate. Put it in lib/providers/, so the generated container finds it:
import 'package:dox_queue/dox_queue.dart';
import '../config/queue.dart';
/// Sets up the queue. Dox finds every DoxProvider class,
/// see lib/generated/container.g.dart.
class AppQueueProvider extends QueueProvider {
AppQueueProvider() : super(queueConfig);
}
The worker file
The worker runs from bin/queue.dart. It starts the providers of your app (database, cache, queue), but not the HTTP server:
import 'dart:io';
import 'package:dox_queue/dox_queue.dart';
import 'package:my_app/config/app.dart';
/// run queued jobs: `dart run bin/queue.dart [work] [flags]`.
/// the `dox` CLI runs this file for its queue commands.
Future<void> main(List<String> args) async {
exitCode = await QueueWorker(app: appConfig).run(args);
}
Database driver
The database driver, job batches and dispatching after the commit are in the dox_queue_database package. If you use only Redis or sync and none of these, skip this step.
Then make your queue provider extend DatabaseQueueProvider in place of QueueProvider. This turns the package on. Without it, QueueConfig(driver: 'database') throws an ArgumentError that names the package.
import 'package:dox_queue_database/dox_queue_database.dart';
import '../config/queue.dart';
class AppQueueProvider extends DatabaseQueueProvider {
AppQueueProvider() : super(queueConfig);
}
The database driver needs the jobs and failed_jobs tables. Batches need the job_batches table, with every driver. dox queue:table writes a migration in db/migration that makes all three:
The migration looks like this, so you can also write it by hand:
// ignore_for_file: file_names
import 'package:dox_migration/dox_migration.dart';
import 'package:dox_queue_database/dox_queue_database.dart';
/// The jobs, failed_jobs and job_batches tables of dox_queue_database.
class CreateQueueTables extends Migration {
@override
Future<void> up(Schema schema) async {
await QueueTables.up(schema);
await QueueTables.upBatches(schema);
}
@override
Future<void> down(Schema schema) async {
await QueueTables.downBatches(schema);
await QueueTables.down(schema);
}
}
To keep the jobs in another database connection of config/database.dart, use QueueConfig(connection: 'postgres').
Redis driver
Set the driver to redis and give the Redis server:
QueueConfig queueConfig = QueueConfig(
driver: 'redis',
jobs: jobRegistry,
redis: RedisOptions(host: env.redisHost, port: 6379),
);
You can also read the Redis server from the env with QueueConfig.redisFromEnv():
QueueConfig queueConfig = QueueConfig(
driver: 'redis',
jobs: jobRegistry,
redis: QueueConfig.redisFromEnv(),
);
It reads REDIS_URL, like rediss://user:password@host:6380/0, where rediss turns on TLS. If REDIS_URL is not set, it reads REDIS_HOST, REDIS_PORT, REDIS_USERNAME, REDIS_PASSWORD and REDIS_TLS. A key that is not set keeps the default of RedisOptions.
Creating jobs
Generating job classes
dox make:job makes a new job file in lib/app/jobs/:
Class structure
A job is a class with @Job() that extends DoxJob. The worker calls its handle() method:
import 'package:dox_core/dox_core.dart';
import 'package:dox_queue/dox_queue.dart';
@Job()
class SendWelcomeEmail extends DoxJob {
SendWelcomeEmail({required this.userId});
final int userId;
@override
Future<void> handle() async {
MailService mail = Dox().ioc.get<MailService>();
await mail.sendWelcome(userId);
}
}
Run dart run build_runner build after you add or change a job. It writes lib/generated/jobs.g.dart, which knows how to save and load each job.
Job data
The parameters of the unnamed constructor are saved with the job. Each parameter needs a field or getter with the same name. These types work:
int,double,num,String,boolDateTime- enums (saved by name)
ListandMap<String, ...>of these, also nesteddynamicandObject(the value must be JSON)- nullable versions of all of them
Any other type stops the build with a clear message.
Save an id, not an object
Do not save a User. Save the userId, and load the user in handle().
The user may change between dispatch and run, so the id gives you fresh data.
Using services in a job
A job is made again from its saved data, so it can not get services in its constructor. Get them in handle() from the Dox container:
The worker runs registerContainer, so every @Inject() class of your app is there.
Setting the number of tries
By default a job runs once. If you want the worker to try again when handle() throws, set tries. Use backoff to wait between the tries:
/// run up to 3 times
@override
int get tries => 3;
/// wait 10 seconds before the 2nd try, 60 seconds before the 3rd
@override
List<Duration> get backoff => const <Duration>[
Duration(seconds: 10),
Duration(seconds: 60),
];
The last backoff entry is used for all later tries. With no backoff, the retry happens at once.
There are three more settings for retries:
retryUntilretries until a time, in place of a number of tries. It is read once, when the job is dispatched.maxExceptionsfails the job after this many tries that threw an error. A timeout does not count.failOnTimeoutfails the job at once when a try times out, with no retry.
Setting a timeout
timeout is the max run time of one try. The default is the worker --timeout, which is 60 seconds:
Choosing the queue
Jobs go on the default queue. You can give a job its own queue, so a worker can run it first or apart:
Handling failed jobs
failed() runs once, after the last try failed. Use it to tell someone, or to clean up:
/// runs once, after the last try failed
@override
Future<void> failed(Object error, StackTrace stack) async {
// tell someone, or clean up
}
Encrypting job data
If a job holds private data, add the ShouldBeEncrypted mixin. Then a person who can read Redis or the jobs table can not read the data:
The data is encrypted with the app key (AppConfig.appKey), so the worker needs the same key, or one of AppConfig.previousKeys. The job name, the queue and the id stay plain text.
Renaming a job class
The class name is saved with each job. So when you rename a class, queued jobs with the old name can not be found. Keep the old name with name:
Dispatching jobs
Call dispatch() to put a job on the queue. It returns the job id:
Queue.dispatch() does the same:
Delayed dispatching
Sometimes a job should wait before it runs. Use delay. You can also pick another queue here:
To wait until a fixed time, use delayUntil. Give delay or delayUntil, not both:
Conditional dispatching
dispatchIf dispatches only when the condition is true, and dispatchUnless only when it is false. They return null when the job was not dispatched:
await SendWelcomeEmail(userId: 1).dispatchIf(user.wantsEmail);
await SendWelcomeEmail(userId: 1).dispatchUnless(user.isBlocked);
Synchronous dispatching
dispatchSync runs the job now, in this process, with no queue:
Dispatching after the response
Sometimes a job is short, but the client should not wait for it. dispatchAfterResponse runs the job in this process after the answer is sent:
Do not await it, because it returns nothing. The job does not go on the queue, so it has no job id and no retry. If the job throws, its failed() runs and the error goes to the exception handler, never to the client. If the server stops before the job runs, the job is lost.
Dispatching after the commit
If you dispatch a job inside a database transaction, it is pushed at once. Then a worker can run it before the transaction commits, or after it rolls back. So the job reads data that is not saved yet, or is never saved.
Use afterCommit: true to wait for the commit. This needs dox_queue_database:
await QueryBuilder.transaction(() async {
User user = await User().create(<String, dynamic>{'email': email});
await SendWelcomeEmail(userId: user.id).dispatch(afterCommit: true);
});
- The job is pushed after the outermost transaction commits.
- If the transaction rolls back, the job is not pushed.
- Outside a transaction, the job is pushed at once.
dispatch()still returns the job id at once.
To do this for every dispatch, set QueueConfig(afterCommit: true). Then afterCommit: false on one dispatch pushes that job at once. Chains and batches take afterCommit too. See Dispatch after the commit details for special cases.
Queueing an email
You do not need a job class to send an email later. With dox_mailer, call Mail.queue():
The mail jobs are part of dox_mailer, so the queue provider and a worker run them.
Running the worker
Start a worker to run the jobs:
Use --queue to pick the queues. The first one goes first. Use --concurrency to run more jobs at once:
--concurrency runs jobs as async tasks on one thread. It helps for jobs that wait, like email, HTTP or database calls. For CPU heavy jobs, run more worker processes. See Worker flags for all flags.
The dox CLI runs the same file, with the same flags: dox queue:work, dox queue:failed, dox queue:retry <id|all>, dox queue:forget <id>, dox queue:flush, dox queue:clear [queue], dox queue:restart and dox queue:prune-failed.
Stopping the worker
Press Ctrl+C, or send SIGTERM (docker stop does it). The worker takes no new job, waits for the running jobs, then stops. A second signal stops it at once.
Restarting after a deploy
The worker keeps your code in memory. So restart it after each deploy, or it runs the old code:
This tells every running worker to stop after its current jobs, and your process manager starts them again with the new code. It works through the app cache, so the workers and the command must use the same cache store, like file on one machine, or redis. The memory store does not work for it.
Running the worker in production
In production, run the binary, not dart run. dox build makes build/bundle/bin/queue, next to the server. It takes the same commands and flags:
Run the worker as its own process, next to the web server. For example, with Docker Compose and the image of the project Dockerfile:
services:
web:
build: .
command: /app/bin/server
worker:
build: .
command: /app/bin/queue work --concurrency=4
You can also use systemd, Supervisor or a second Fly.io process. Set it to restart when it stops.
Running jobs in the web server
A small app can skip the worker process. Set inProcess: true, and the first isolate of the web server runs jobs too:
When the server stops, it waits for the running jobs. Only the web server runs this worker, not the scheduler process (bin/schedule.dart).
Heavy jobs slow down your API
The jobs share the isolate with web requests, so a long or CPU heavy job makes requests on that isolate slow. Use a worker process when your jobs are heavy or many.
Dealing with failed jobs
When handle() throws, the worker waits the backoff time and tries again, up to tries times. After the last try, the job goes to the failed jobs, and failed() runs. A timeout counts as a failed try.
A timeout does not stop the code
Dart can not stop a running Future. After a timeout the try counts as failed, but handle() keeps running in the background until it ends.
So write jobs that are safe to run twice.
If you want the worker to wait for the old run, start it with --timeout-grace=SECONDS (or set workerTimeoutGrace for the in-process worker). Keep timeout plus the grace time below retryAfter.
When a worker crashes, its job stays reserved. After retryAfter, another worker takes it, and this counts as a try.
Retrying failed jobs
Failed jobs are kept until you retry or delete them. List them with failed, and put one or all of them back on their queue with retry:
dart run bin/queue.dart failed
dart run bin/queue.dart retry 6f1c0e...
dart run bin/queue.dart retry all
Deleting failed jobs
forget deletes one failed job, and flush deletes all of them:
prune-failed deletes failed jobs older than --hours=N (default 24):
Controlling the job from handle()
Sometimes the job itself knows what should happen next. For example, an API is busy, so the job should try again later. Add the InteractsWithQueue mixin, then call one of these in handle():
| Call | What it does |
|---|---|
attempts() |
how many times a worker took the job. The first run gives 1 |
release(delay) |
put the job back on its queue. It runs again after delay |
fail(error) |
fail the job now, with no retry. failed() runs |
delete() |
delete the job, with no retry and no failed() |
@Job()
class SendReport extends DoxJob with InteractsWithQueue {
SendReport({required this.reportId});
final int reportId;
@override
int get tries => 5;
@override
Future<void> handle() async {
ReportApi api = Dox().ioc.get<ReportApi>();
if (await api.isBusy()) {
release(Duration(seconds: 30));
return;
}
if (!await api.hasReport(reportId)) {
delete();
return;
}
await api.send(reportId);
}
}
release() counts as a try. Because the default tries is 1, set tries above 1 (or retryUntil) when a job calls release(). If not, the job fails on its next pickup.
Clearing jobs from a queue
clear deletes all jobs of a queue, also the running ones. The default queue is default:
Job middleware
Job middleware wraps handle() of a job. Use it for rules that many jobs share, like a rate limit. Add the HasMiddleware mixin and list the middleware. The first one runs first:
@Job()
class SendReport extends DoxJob with HasMiddleware {
SendReport({required this.reportId});
final int reportId;
@override
int get tries => 5;
@override
List<JobMiddleware> get middleware => <JobMiddleware>[
RateLimited('report-api', 10, Duration(minutes: 1)),
];
// ...
}
The built-in middleware keeps its counts and locks in the default cache store. So all workers must use the same store: file on one machine, or redis for several servers. The memory store is per process, so the limit would not work.
A middleware that puts the job back counts as a try. So give these jobs tries above 1, or a retryUntil.
Rate limiting
RateLimited(key, max, per) runs at most max jobs of key in each per window. The other jobs go back on the queue until the window ends:
Preventing job overlaps
WithoutOverlapping(key) runs only one job of key at a time:
WithoutOverlapping(
'report-$reportId',
releaseAfter: Duration(seconds: 30),
expireAfter: Duration(minutes: 10), // longer than the longest report run
),
releaseAfteris when the other jobs go back on the queue. The default is zero, so at once. WithreleaseAfter: null, they are dropped.expireAfterfrees the lock after this time, also while a job still runs. The default is no expiry. Set it longer than the longest run of the job, so a crash can not hold the lock forever.- The key has no class name in it, so job classes with the same key share one lock. Put the class name in the key if they should not.
Throttling exceptions
ThrottlesExceptions(key, maxAttempts:, decay:, retryAfter:) is for jobs that call a service that is down. After maxAttempts errors (default 10), jobs of key wait until decay is over (default 10 minutes). A job that threw goes back after retryAfter (default zero).
Writing your own middleware
Extend JobMiddleware. Call next() to go on. If you do not call it, the job is skipped and counts as done. Call release(job, delay) to put the job back on the queue:
class SkipAtNight extends JobMiddleware {
@override
Future<void> handle(DoxJob job, JobNext next) async {
if (DateTime.now().hour < 6) {
release(job, Duration(hours: 1));
return;
}
await next();
}
}
Unique jobs
Some jobs should wait in the queue only once, like "rebuild the search index of product 5". Add the ShouldBeUnique mixin. While a job with the same class and uniqueId waits or runs, dispatch() pushes nothing and returns '':
@Job()
class UpdateSearchIndex extends DoxJob with ShouldBeUnique {
UpdateSearchIndex({required this.productId});
final int productId;
@override
String get uniqueId => '$productId';
@override
Duration? get uniqueFor => Duration(hours: 1);
// ...
}
uniqueIdis the key next to the class. The default is empty, so only one job of the class at a time.uniqueForends the lock after a time, also when the job never ran. You should set it, because the lock of a crashed worker never ends without it.- The worker frees the lock when the job is done, deleted or failed for good. A release or a retry keeps it.
- The lock is in the default cache store, so the web app and the workers must use the same store, like
redis.
If a new job may be queued while the old one runs, use ShouldBeUniqueUntilProcessing in place of ShouldBeUnique. It frees the lock when a worker starts the job.
Job chaining
A chain runs jobs one after the other. The next job is pushed only when the job before it is done. If a job fails for good, the rest of the chain is dropped and the catch job runs:
await Queue.chain(<DoxJob>[
ResizeImage(imageId: 1),
UploadImage(imageId: 1),
TellUser(imageId: 1),
]).catchError(TellAdmin(imageId: 1)).dispatch(queue: 'images');
The rest of the chain is saved in the job that runs now, so the chain works across worker processes and restarts. dispatch() returns the id of the first job. queue and connection count for every job, but delay counts for the first job only.
Each job carries the data of the jobs after it. So keep chains short, and save ids in the jobs, not big data. Give chain jobs more than one try, and make them safe to run twice.
Handling chain failures
A function can not be saved in the queue. So the catch handler is a job too, of a class in jobRegistry. Add the CatchesError mixin to read the error:
@Job()
class TellAdmin extends DoxJob with CatchesError {
TellAdmin({required this.imageId});
final int imageId;
@override
Future<void> handle() async {
await alertAdmin('image $imageId failed: $caughtError');
}
}
The catch job runs once, in the worker where the job failed, with no queue and no retry. An error in it is logged. For slow work, dispatch a new job from it.
queue:retry of the failed job brings the chain back. When the job is then done, the chain goes on.
Job batching
A batch runs many jobs and acts when they are all done. Batches need dox_queue_database and the job_batches table, with every driver. Without the package, Queue.batch() throws a StateError.
Batch batch = await Queue.batch(<DoxJob>[
ImportRows(from: 0),
ImportRows(from: 1000),
ImportRows(from: 2000),
])
.name('import')
.then(ImportDone())
.catchError(ImportFailed())
.whenComplete(CleanUpImport())
.dispatch(queue: 'imports');
print(batch.id);
The batch is saved in the job_batches table of the QueueConfig.connection database, also with the Redis and sync drivers.
Batch callbacks
These callbacks run at the steps of a batch:
| Callback | Runs |
|---|---|
before(job) |
once, at dispatch(), before the jobs are pushed |
progress(job) |
each time a job is done. With allowFailures(), also when one fails |
then(job) |
once, when every job is done |
catchError(job) |
once, when the first job fails for good |
whenComplete(job) |
once, when every job ran, done or failed |
A function can not be saved in the queue, so each callback is a job of a class in jobRegistry. It runs once, with no queue and no retry. An error in a callback is logged. Add Batchable to read the batch, and CatchesError to read the error:
@Job()
class ImportFailed extends DoxJob with Batchable, CatchesError {
@override
Future<void> handle() async {
Batch? batch = await this.batch();
await alertAdmin('import ${batch?.name} failed: $caughtError');
}
}
Defining batchable jobs
Add the Batchable mixin to a job to read its batch. batchId is the id, and batch() reads the batch as it is now. For example, a job can skip its work when the batch was cancelled:
@Job()
class ImportRows extends DoxJob with Batchable {
ImportRows({required this.from});
final int from;
@override
Future<void> handle() async {
Batch? batch = await this.batch();
if (batch?.cancelled ?? false) return;
await importRows(from);
}
}
Inspecting batches
A Batch has these fields:
| Field | What it holds |
|---|---|
id, name |
the id and the name |
totalJobs |
how many jobs the batch has |
pendingJobs |
how many jobs are not done. It also counts failed jobs, because they can be retried |
failedJobs |
how many jobs failed |
processedJobs |
how many jobs are done. Failed jobs are not counted |
progress |
the percent of done jobs, from 0 to 100 |
finished |
true when every job ran, or the batch was cancelled |
cancelled |
true after cancel(), or after a first failure |
A Batch shows the state at the time it was read. To read it again, use Queue.findBatch(id). It gives null when there is no such batch.
Cancelling batches
The first job that fails for good cancels the batch. If you want the batch to go on, add allowFailures(). You can also cancel a batch yourself with cancel(). This does not stop the jobs: they still run and count, so a Batchable job should read cancelled to skip its work.
Adding jobs to a batch
add(jobs) adds jobs to a batch, for example from a job of the batch. They go on the queue and connection of the batch. It throws a StateError when the batch is finished, cancelled or deleted.
Chains in a batch
Use chain(jobs) to put a chain in a batch. Its jobs run one after the other, and the batch counts the chain as one job:
await Queue.batch(<DoxJob>[ImportUsers()])
.chain(<DoxJob>[ImportOrders(), SumOrders()])
.dispatch();
When a job of the chain fails for good, the rest of the chain is dropped, and the batch counts one failed job.
Pruning batches
Old batch rows stay in the table until you delete them. prune-batches deletes batches that finished, or were cancelled, more than --hours ago (default 24). Add --unfinished to also delete batches that never finished and were made more than that many hours ago:
dart run bin/queue.dart prune-batches --hours=48
dart run bin/queue.dart prune-batches --hours=48 --unfinished=168
In code, use Queue.pruneBatches(before, unfinishedBefore:).
Queue events
Sometimes you want to log or count jobs. Listen to the steps of a job:
Queue.before((QueueEvent event) => print('start ${event.name} ${event.id}'));
Queue.after((QueueEvent event) => print('done ${event.name}'));
Queue.failing((QueueEvent event) => alert(event.name, event.error));
| Hook | When |
|---|---|
Queue.queued |
dispatch() pushed a job |
Queue.before |
a worker starts a job |
Queue.after |
a worker finished a job and deleted it |
Queue.released |
a worker put a job back (release() or a retry) |
Queue.exceptionOccurred |
handle() threw and the job will be tried again |
Queue.failing |
a job failed for good |
Queue.looping |
a worker is about to take the next job |
Queue.stopping |
a worker stops |
Listeners are per isolate. So register them where the app starts, like in a provider. Then the web app and each worker have them. A listener can be async, and a listener that throws is only logged. See Event details for the fields of QueueEvent.
Batch events
A batch has its own events. In them, event.batch is the batch after the step:
Queue.batchProgress((QueueEvent event) {
Batch batch = event.batch!;
print('${batch.name}: ${batch.progress}%');
});
Queue.batchFailed((QueueEvent event) => alert(event.batch!.id, event.error));
| Hook | When | Where it fires |
|---|---|---|
Queue.batchDispatched |
the batch was saved, before its jobs are pushed | the process that called dispatch() |
Queue.batchProgress |
a job (or a chain) of the batch is done. With allowFailures(), also when one failed |
the worker that counted it |
Queue.batchFailed |
the first job failed for good. event.error is its error |
the worker that counted it |
Queue.batchCancelled |
the batch was cancelled: cancel(), the first failure without allowFailures(), or a push that threw |
the process that cancelled it |
Queue.batchFinished |
every job ran, done or failed | the worker that counted the last job |
Each event fires once per step, also with many workers. When one failure cancels the batch, batchFailed fires first, then batchCancelled.
Queue connections
Sometimes one app needs more than one queue store. The config is the main connection. Add more in connections, by name:
QueueConfig queueConfig = QueueConfig(
driver: 'redis',
jobs: jobRegistry,
connections: <String, QueueConfig>{
'database': QueueConfig(driver: 'database'),
},
);
Pick a connection when you dispatch. With no connection, the job goes to the main connection. An unknown name throws an ArgumentError:
Run a worker for each connection. Every command of bin/queue.dart takes --connection:
dart run bin/queue.dart work
dart run bin/queue.dart work --connection=database
dart run bin/queue.dart failed --connection=database
- An entry uses only
driver,redis,prefix,connection(the dox_orm one),retryAfter,serializeSyncandfailedAtTimestamp. Thejobsof the main config count for all connections. - The in-process worker (
inProcess) runs the jobs of the main connection only. - Two
databaseconnections on the same dox_orm connection share thejobsandfailed_jobstables. So give each one its own dox_orm connection.
Using the queue without Dox
dox_queue also works in a plain Dart app. Add build_runner and dox_builder, so it writes lib/generated/jobs.g.dart:
Write your jobs as in Creating jobs, and run dart run build_runner build. Then set up the queue and run a worker yourself:
import 'dart:async';
import 'dart:io';
import 'package:dox_core/dox_core.dart';
import 'package:dox_queue/dox_queue.dart';
import 'package:my_app/generated/jobs.g.dart';
import 'package:my_app/jobs/resize_image.dart';
Future<void> main() async {
// the worker logs with the Dox logger, so it needs an AppConfig
Dox().initialize(AppConfig(appKey: 'not-used'));
Queue.configure(QueueConfig(
driver: 'redis',
jobs: jobRegistry,
redis: RedisOptions(host: '127.0.0.1', port: 6379),
));
await ResizeImage(path: 'photo.png').dispatch();
Worker worker = Worker(
driver: Queue.driver,
jobs: jobRegistry,
options: WorkerOptions(queues: <String>['default'], concurrency: 4),
);
// Ctrl+C: finish the running jobs, then stop
StreamSubscription<ProcessSignal> signal =
ProcessSignal.sigint.watch().listen((_) => worker.stop());
await worker.run();
await signal.cancel();
await Queue.close();
}
WorkerOptionstakes the same settings as the flags ofwork:queues,concurrency,once,tries,timeout,sleepandstopWhenEmpty.- The
databasedriver needsdox_queue_database. There are no providers here, so callQueueDatabase.register()beforeQueue.configure(). It uses the defaultdox_ormconnection, so set that up first withSqlQueryBuilder.initializeWithDatabaseConfig(...). - No services are in the Dox container, so make the services of a job in
handle().
Testing
Running jobs in tests
Use the sync driver. dispatch() then runs the job at once, so you can check the result right after:
import 'package:dox_queue/dox_queue.dart';
import 'package:my_app/app/jobs/send_welcome_email.job.dart';
import 'package:my_app/generated/jobs.g.dart';
import 'package:test/test.dart';
void main() {
setUp(() {
Queue.configure(QueueConfig(driver: 'sync', jobs: jobRegistry));
});
test('sends the welcome email', () async {
await SendWelcomeEmail(userId: 1).dispatch();
// check the email was sent
});
}
A job that uses Dox() (for example Dox().ioc or Logger) needs the app config. Call Dox().initialize(appConfig) in setUpAll.
The sync driver runs the job object as it is. So a job that is not in jobRegistry, or a field that is not JSON, works here and fails later in the worker. Set serializeSync: true to save the job and read it back before it runs, the way a worker gets it. Then dispatch() throws for such a job:
To test a job alone, call handle():
Faking the queue
If you only want to check that your code dispatched a job, without running it, use Queue.fake(). It records every dispatch() in a list, and needs no Queue.configure():
late QueueFake queue;
setUp(() {
queue = Queue.fake();
});
tearDown(Queue.restore);
test('signing up queues the welcome email', () async {
await signUp(userId: 1);
queue.assertPushed<SendWelcomeEmail>();
queue.assertPushed<SendWelcomeEmail>(
(SendWelcomeEmail job) => job.userId == 1,
);
queue.assertPushedTimes<SendWelcomeEmail>(1);
queue.assertNotPushed<SendInvoice>();
});
| Method | Checks |
|---|---|
assertPushed<T>([test]) |
a T job was pushed. With test, one of them passes it |
assertPushedTimes<T>(n) |
exactly n T jobs were pushed |
assertNotPushed<T>() |
no T job was pushed |
assertNothingPushed() |
no job was pushed at all |
assertChained(<Type>[A, B]) |
a chain of exactly these job classes, in this order, was dispatched |
assertBatched([test]) |
a batch was dispatched. With test, one of them passes it. test gets a PendingBatch |
assertBatchedChain(<Type>[A, B]) |
a batch has a chain of exactly these job classes, in this order |
assertBatchCount(n) |
exactly n batches were dispatched |
To read the jobs, use queue.pushed. Each item has the job, the queue name and the delay. queue.jobsOf<T>() gives the T jobs, queue.clear() forgets them, and Queue.restore() stops the fake.
The fake does not encode the job. So a job that is missing from QueueConfig.jobs, or has data that is not JSON, passes a faked test but fails for real.
Testing the queue itself
To test the real queue, use the database driver of dox_queue_database with SQLite :memory:. Then run a worker that stops when the queue is empty:
await Worker(
driver: Queue.driver,
jobs: jobRegistry,
options: WorkerOptions(stopWhenEmpty: true),
).run();
Details
This section is for special cases. Most apps do not need it.
All options
| Option | Default | What it does |
|---|---|---|
driver |
sync |
sync, database (needs dox_queue_database) or redis |
jobs |
empty | jobRegistry from lib/generated/jobs.g.dart |
redis |
127.0.0.1:6379 | RedisOptions(host: ..., port: ..., password: ...) for the redis driver. QueueConfig.redisFromEnv() reads it from the env |
prefix |
dox_queue: |
the start of every Redis key |
connection |
default | the dox_orm connection name, for dox_queue_database |
retryAfter |
90 seconds | when a worker does not finish a job in this time, another worker can take it |
inProcess |
false |
run a worker inside the web server, see Running jobs in the web server |
workerQueues |
['default'] |
the queues of the in-process worker |
workerConcurrency |
1 |
jobs at once in the in-process worker |
workerTimeoutGrace |
none | the --timeout-grace wait of the in-process worker |
serializeSync |
false |
the sync driver saves the job and reads it back before it runs |
connections |
none | more queue connections by name, see Queue connections |
afterCommit |
false |
push jobs that are dispatched inside a transaction only after it commits (needs dox_queue_database) |
failedAtTimestamp |
false |
the failed_at column is a timestamp, not a number, see The failed_at column |
Worker flags
| Flag | Default | What it does |
|---|---|---|
--queue=high,default |
default |
the queues to work. The first one goes first |
--concurrency=4 |
1 |
jobs at once in this process |
--once |
off | run one job, then stop |
--tries=3 |
1 |
tries of a job that does not set tries |
--timeout=60 |
60 |
max run time in seconds of a job that does not set timeout |
--sleep=3 |
3 |
seconds to wait when the queues are empty |
--stop-when-empty |
off | stop when the queues are empty |
--timeout-grace=10 |
none | after a timeout, wait up to this many seconds for the old run to end before the next try |
--backoff=1,5,10 |
none | seconds to wait before each retry of a job that does not set backoff |
--max-jobs=1000 |
off | stop after this many jobs |
--max-time=3600 |
off | stop after this many seconds |
--memory=256 |
off | stop when the process uses more than this many MB |
--connection=name |
main | the queue connection to use |
With --max-jobs, --max-time and --memory, the worker stops after its running jobs, so let your process manager start it again.
Worker details
- While the worker waits for
--timeout-grace, it takes no other job. Iftimeoutplus the grace time is aboveretryAfter, another worker can take the job while this one waits. - When a crashed worker's job has no tries left, it fails with a message that the worker did not finish.
restartwrites a time to the app cache, and each worker reads it every 3 seconds.- With
InteractsWithQueue, the worker does the call whenhandle()ends. When you call more than one, the last one wins. A call also wins over an error thathandle()throws after it. Whenhandle()times out, the call is dropped. - With
dispatchSync(),dispatchAfterResponse()and thesyncdriver there is no queue. Thereattempts()is 1,release()anddelete()do nothing, andfail()only runsfailed(). - In a test file,
package:testhas its ownfail(). Inside a job class withInteractsWithQueue, writethis.fail(...)to make it clear which one you call. - Job middleware also runs with the
syncdriver,dispatchSync()anddispatchAfterResponse(). There is no queue there, so a release logs a warning andhandle()does not run. dispatchAfterResponse()works becauseQueueProviderturns onAfterResponse. Outside a request, the job runs soon.
Dispatch after the commit details
afterCommitneedsdox_queue_database. Without it, a dispatch withafterCommiton throws aStateError.- A job that is not in the registry, or data that is not JSON, throws at
dispatch(), not at the commit. - When the push fails after the commit, the
QueryBuilder.transaction()call throws that error. The data stays saved. - It waits only for a transaction on the default dox_orm connection, opened with no
connection:name. A transaction opened by name pushes the job at once. - The
syncdriver runs the job after the commit. Asyncchain runs as a whole after the commit, and gets a new id. - A batch saves its
job_batchesrow and pushes its jobs after the commit. A rollback drops it all. Before the commit, the returnedBatchis not saved yet. dispatchSync()anddispatchAfterResponse()do not wait for a commit.Queue.fake()keeps the job when you calldispatch(), also when the transaction later rolls back.- It works through
QueryBuilder.afterCommit()of dox_orm.
Unique job details
clearfrees the locks of the jobs it deletes.retryof a failed job does not take the lock.dispatchSync()ignores uniqueness.- With
afterCommitinside a transaction, the lock is taken at the commit.dispatch()returns the id at once, also when the job is then skipped. Queue.fake()keeps its own locks and needs no cache.
Chain details
- A job that is released or retried keeps the chain. A job that ends with
delete(), or that a middleware skips, counts as done, so the chain goes on. - The
syncdriver runs the jobs now, one after the other. When one throws, the catch job runs anddispatch()throws the error. - A
ShouldBeEncryptedjob stays encrypted inside the job before it. - A skipped unique job stops the chain, and the catch job runs with a
StateError. When the first job is skipped,dispatch()returns''and nothing runs. - A next job that can not be read, for example because its class was removed, stops the chain. The catch job runs with the error.
- The worker pushes the next job before it deletes the job that is done. So a crash between the two can run the next job twice.
- When the push of the next job fails, the try of the current job counts as failed, and the job is tried again.
- The unique lock of a job is freed before the next job is pushed. So a next job with the same unique id is not skipped.
Batch details
- A failed job that you retry with
queue:retryis not counted again when it fails again. When it is done,pendingJobsandfailedJobsboth go down by one, socatchErrorcan run again on a later failure. - An empty batch throws an
ArgumentError, because it would never finish. thenandwhenCompleterun once, also with many workers, because each count changes in one atomic step.- A skipped unique job is taken out of the counts. When it was the last open job,
thenandwhenCompleterun. In a chain of a batch, a skipped first job takes the chain out of the counts, and a skipped later job fails the chain. - When a push fails in
dispatch()oradd(), the jobs that were not pushed are taken out of the counts. The batch is cancelled and the error is thrown. The jobs that were pushed still run. - A job that runs twice, for example when it runs longer than
retryAfter, can be counted twice.failedJobsnever goes below 0. - The batch is marked finished before
thenandwhenCompleterun, soadd()from a callback throws aStateError.add()takes jobs only, not chains. - When a worker crashes after a job is done and before it counted the job, the batch keeps that job as pending, and
thennever runs. Delete such batches withprune-batches --unfinished. - The
syncdriver runs the jobs atdispatch()and counts them. A job that throws counts as failed, anddispatch()does not throw. - In
Queue.fake(), chains are inqueue.chainsand batches inqueue.batches, and the chains of a batch inPendingBatch.chains. Nothing is saved or run. In a faked batch,add()adds to the batch,cancel()does nothing andQueue.findBatch()givesnull.
Event details
A listener gets a QueueEvent:
| Field | What it holds |
|---|---|
type |
the kind of event |
job |
the job. null for looping and stopping, and when the worker can not read the job |
name |
the job name of the registry |
id |
the job id |
queue |
the queue name |
attempts |
the try number. 0 for queued, looping and stopping |
error, stack |
the error and its stack trace, for exceptionOccurred and failing. batchFailed has error only |
delay |
the wait, for queued and released |
batch |
the Batch, for the batch events. null for other events |
- The
syncdriver,dispatchSync()anddispatchAfterResponse()fire no job events. Thesyncdriver does fire the batch events, all in the process that calleddispatch(). Queue.fake()fires no events.Queue.forgetListeners()removes all listeners, for example between tests.
Fake details
dispatchSync()anddispatchAfterResponse()still run the job under the fake. Their jobs are inqueue.dispatchedSyncandqueue.dispatchedAfterResponse, not inqueue.pushed.Queue.fake()keeps the connection name inPushedJob.connection.- The fake also catches
Mail.queue()on the default queue, but only whenMail.fake()is off. It does not catch a custom queue set withMail.useQueue(). See Mail.
How the drivers store jobs
- Database: jobs are rows in
jobs, failed jobs are rows infailed_jobs. On Postgres, MySQL and MariaDB a worker takes a job withSELECT ... FOR UPDATE SKIP LOCKED, so many workers do not wait on each other. On SQLite a worker takes a job with oneUPDATE ... RETURNING. SQLite runs one write at a time, so this is safe for small apps. - Redis: each queue has a list of ready jobs, a sorted set of delayed jobs and a sorted set of reserved jobs. A Lua script takes a job, so two workers never get the same one. Failed jobs are in the hash
dox_queue:failed. failed_jobs.connectionhas the name of the queue connection, or NULL for the main one. A table without this column works too, but then the name is not saved, andfailed --connection=Xlists all failed jobs of the table. A worker checks for the column once, so restart the workers after you add it:
The failed_at column
failed_jobs.failed_at is the time the job failed. By default it is a big integer with unix time in milliseconds. If you want a UTC timestamp, pass failedAtTimestamp: true to QueueTables.up in the migration, and set it in the queue config too:
QueueConfig queueConfig = QueueConfig(
driver: 'database',
jobs: jobRegistry,
failedAtTimestamp: true,
);
FailedJob.failedAt is the same DateTime with both types. Each entry of connections has its own failedAtTimestamp.
The setting and the column must match
A worker reads the column type once, when it first uses failed_jobs. If the setting does not match the column, fail, failed, retry, forget and prune-failed throw a StateError, and nothing is written. A job that fails all its tries is NOT moved to failed_jobs, and its failed() does not run. It stays in jobs and runs again after retryAfter, until the setting matches.
If the type can not be read (for example, the connection dropped), the worker tries again on the next call. With the setting off, it logs a warning and goes on. With the setting on, the call throws.
| Database | Column | How the time is kept |
|---|---|---|
| SQLite | TIMESTAMP |
ISO text in UTC, with milliseconds |
| Postgres | TIMESTAMP (without time zone) |
the UTC time, with milliseconds |
| MySQL, MariaDB | TIMESTAMP |
whole seconds, in UTC. The queue writes it with FROM_UNIXTIME() and reads it with UNIX_TIMESTAMP(), so the session time zone does not change the time. A MySQL and MariaDB TIMESTAMP can not hold a time after 2038-01-19 |
To change an existing integer column to a timestamp:
- Stop all workers, on every server. Stop the app too when it has
inProcess: true. A worker that still runs writes numbers into the timestamp column, and on SQLite this gives no error. - Run the SQL for your database below, in a new migration (
await schema.driver.execute('...');for each line) or in a database client. - Set
failedAtTimestamp: truein the queue config. - Start the workers again.
SQLite (3.35 or newer). The new column allows NULL, because SQLite can not add a NOT NULL column without a default. The queue never writes NULL there:
ALTER TABLE failed_jobs ADD COLUMN failed_at_utc TIMESTAMP;
UPDATE failed_jobs SET failed_at_utc = strftime('%Y-%m-%dT%H:%M:%fZ', failed_at / 1000.0, 'unixepoch');
ALTER TABLE failed_jobs DROP COLUMN failed_at;
ALTER TABLE failed_jobs RENAME COLUMN failed_at_utc TO failed_at;
Postgres:
ALTER TABLE failed_jobs ALTER COLUMN failed_at TYPE TIMESTAMP USING to_timestamp(failed_at / 1000.0) AT TIME ZONE 'UTC';
MySQL and MariaDB: