outloq

How we send a 500 MB video using a few megabytes at a time

Uploading your video to YouTube used to mean holding all of it in memory first. Now it flows through in 8 MB pieces, so we publish more posts at once.

The Outloq team3 min read

When it's time for your post, our publishing worker has to get your video from our storage to each platform. Instagram and Facebook make that easy: we hand them a private, short-lived link and they fetch the video themselves.

YouTube doesn't offer that. We have to send the bytes. Our first version did the obvious thing: download the whole video, then upload it in one go. For a phone video of a few hundred megabytes, that meant the worker held the entire file in memory, and in practice several copies of it while the request was being built and sent.

The problem: the whole video, all at once

Before: the whole video in memoryStorage sends the whole 500 MB video to the worker, which holds it in memory as about 2 GB, then sends it to YouTube.Storageyour videoWorker memory~2 GBfor a 500 MB videoYouTubeone big uploadTwo uploads at once could need 4 GB, so the worker sent only 2 posts at a time.
The old way. The whole file has to arrive before any of it leaves.

Memory, not processing power, became the limit. To stay safe on our server, the worker sent just two posts at a time. If ten people scheduled for 18:00, most of them waited in line.

The fix: a conveyor belt, not a bucket

YouTube's upload API can take a video in pieces. So instead of filling a bucket and then pouring it out, we built a conveyor belt:

  1. Ask storage how big the video is, without downloading it.
  2. Read the first 8 MB, send it to YouTube, let it go.
  3. Read the next 8 MB, send it, and so on until the last piece.

At any moment the worker holds about one piece, however large the video is.

After: one piece at a timeThe worker reads one 8 MB piece from storage and sends it to YouTube, then the next. Only one piece is in memory at a time.Storagereads by byte rangeWorker memory8 MBone piece at a timeYouTuberesumable uploadMemory stays small whatever the size of the video.
The new way. Pieces flow through; nothing waits for the whole file.

A bonus: a dropped connection no longer starts over

Big uploads sometimes hit a network hiccup. Before, a failure at 90% meant downloading and uploading the whole video again. Now, when a piece fails, we ask YouTube “how much do you have?” and continue from exactly that byte.

Resuming after a dropped piecePieces 1 to 3 arrive; piece 4 drops partway. Outloq asks YouTube how much it has and continues from that byte, not from the start.drop“How much do you have?” → continue from that byte0 MB500 MB
If a piece fails, the upload picks up where YouTube left off. Repeated failures stop with a clear reason.

What we measured

We tested a 500 MB video against a local stand-in for our storage and for YouTube's upload API, and recorded how much extra memory the worker needed.

Before: one upload
≈ 2,000 MB
Now: one upload
≈ 120 MB
Now: five uploads at once
≈ 300 MB

Five uploads at once now need less memory than one upload did before. So the worker sends five posts at once instead of two, and we can raise that with a setting rather than a bigger server.

The same idea, everywhere we send bytes

  • YouTube: 8 MB pieces through the resumable upload, resuming on failure.
  • TikTok (when we upload rather than TikTok fetching the video): 10 MB chunks, each read just before it's sent.
  • Facebook (only if Meta can't fetch our link): the file streams straight through in one request, never held whole.
  • Instagram and Facebook normally: unchanged. They fetch the video from a private link themselves, which was already the lightest option.

All of these read through the same small piece of code, which asks storage for an exact byte range and checks it got exactly that back. The links stay private, read-only and short-lived (one hour, enough for a slow upload to finish), and nothing about how your videos are stored changed.

Big files don't need big memory. They need to keep moving.

What you'll notice

  • When lots of posts are due at the same minute, they go out together instead of queueing two by two.
  • Large YouTube videos are more reliable: a network blip mid-upload no longer means starting over.
  • Nothing else. Your videos look and post exactly as before.
  • uploads
  • infrastructure
  • how outloq works

Keep reading