outloq

Why our scheduler stopped asking “is it time yet?”

Scheduled posts used to be found by checking the database every minute. Now each post carries its own timer. What changed, and the outage that showed us why.

The Outloq team3 min read

When you schedule a video for Thursday at 18:00, Outloq has one promise to keep: at 18:00 the video goes to TikTok, Instagram, YouTube and Facebook. Nothing before, nothing late, never twice.

There are two ways to keep that promise. You can keep asking “is anything due?” over and over. Or you can set an alarm for each post and let the alarm wake you. We started with the first and moved to the second.

Before: one loop, every minute

Our first scheduler was a single job that ran once a minute. Each run asked the database for every post whose time had passed and that was still waiting, then handed each one to the publishing worker.

Before: polling every minuteA clock job runs every minute and queries the database for due posts, then queues each one for the publishing worker. The database is queried 1,440 times a day, whether or not anything is due.Every-minute job* * * * *Database“Anything due?”Publish workersends the videoquerydue postsOne day, one bar per 15 minutes (each stands for 15 queries):00:0024:001,440 queries a day, even with nothing scheduled
The every-minute loop. Simple and reliable, but it talks to the database all day long.

This design is easy to reason about and hard to break. If the server restarts, the next tick picks up whatever was missed. Many schedulers work exactly like this.

It has two costs, and they only show up later:

  • Posts go out up to a minute late. A post due at 18:00:05 waits for the 18:01 tick.
  • The database never rests. Our database (Neon's serverless Postgres) switches itself off after five quiet minutes and bills only for the time it is awake. A query every 60 seconds means it is never quiet for five minutes, so it never switches off.

After: every post gets its own alarm

Now, when you schedule or move a post, Outloq sets a timer for that exact moment in our job queue (BullMQ, running on Redis). The database is only touched when something actually happens.

After: one timer per postScheduling a post saves it in the database and sets a delayed timer in Redis for its exact time. When the timer fires, the API claims the post only if it is still scheduled and due, then queues it for the publish worker.You scheduleThu 18:00Databasepost saved, onceTimer in Redisdelay = until 18:0018:00:00 · timer firesClaim the postonly if still scheduled and due(cancelled or moved? do nothing)Publish workersends the video within ~1 secondThe database is touched when you act and when a post is due. Otherwise it sleeps.
One timer per post. A post that's cancelled or moved simply finds nothing to claim when its old timer fires.

A few details make this safe:

  • Timers can be wrong; the claim can't. If you move a post from 18:00 to 20:00, the 18:00 timer still fires. It asks the database for “this post, if it's still scheduled and due now” and gets nothing back, so nothing is sent. The 20:00 timer does the real work.
  • Never twice. The claim changes the post from scheduled to publishing in one step. A second timer for the same post finds it already taken.
  • A safety sweep, four times a day. Redis can lose data, and a server can be down at the wrong moment. So at 03:00, 09:00, 15:00 and 21:00 UTC, and whenever the API starts, a sweep publishes anything overdue and re-creates every future timer. Every five minutes the API also checks (in Redis, not the database) whether Redis has been wiped; if so, it runs the sweep at once.

What changed, in numbers

Every-minute loopOne timer per post
Database queries just to look for due posts1,440 a day4 a day (the sweeps)
How late a post can go outup to 60 secondsabout 1 second
Can the database switch off when idle?NeverYes, between bursts of real use
Database compute at the smallest size, per month~180 hours (always on)only the hours people actually use it
Database awake time over one dayTop bar: with polling, the database is awake all 24 hours. Bottom bar: with timers, it is awake only during short bursts when people use the app, when posts are due, and for the four sweeps.Beforeawake 24 hoursAfter00:0012:0024:00Green: someone is using Outloq, a post is due, or a sweep runs. Grey: the database is off.
An illustration of one day, not measured data. The real pattern depends on when people post.

The day the old way caught up with us

We built the timer version in September and tested it on our staging servers. Production was still running the every-minute loop when, in October, sign-in stopped working on both staging and the live site.

The cause wasn't a bug in a new release. Our database plan included a fixed number of compute hours a month, shared by production and staging. Production's database had been awake around the clock: the old loop queried it every minute, our hosting's health check queried it every few seconds, and idle connections were never closed. Together they used up the month's hours early, and the database refused new connections, so nobody could sign in.

The timer version fixes all three: no per-minute query, a health check that only checks the API and Redis (the database has its own check for when we want it), and idle connections that close after 30 seconds. We moved to a paid database plan with a spending alert the same day, and production now runs the timer version too.

A cheap query, run forever, isn't cheap. Anything that runs on a clock should ask whether it needs to.

What you'll notice

  • Scheduled posts go out on the second, not up to a minute later.
  • Moving or cancelling a post just before its time is safe: the old timer finds nothing to send.
  • If something does go wrong on our side, the sweep catches overdue posts within six hours at most, and usually at the next restart.

Posting at the right moment is the whole point of a scheduler. Now it's also the only time we wake the database to do it.

  • scheduling
  • infrastructure
  • how outloq works

Keep reading