Small objects on spinning disks waste IOPS. On one cluster I look after, everything under 4 KB needed to end up on NVMe instead.

New uploads were the easy half. A Lua script on the RGW request path sets Request.HTTP.StorageClass from the incoming Content-Length, so anything written from that point on lands in the right place. That script is a post of its own.

The backlog was the hard half. A few thousand buckets, millions of objects in most of them, all written before the rule existed.

And I was not allowed to touch the buckets. They belong to tenants. A migration like this is an operator concern, and nobody on the other side should be able to tell it happened. That rules out a lifecycle policy. It also rules out copying objects onto themselves. More on both below.

So the job split in two. This post is the finding half.

What I ended up with

My first instinct was the obvious one, and it died immediately:

aws s3api list-objects-v2 --bucket mybucket \
  --query 'Contents[?Size<`4096`].Key'

The AWS CLI paginates transparently and applies --query after collecting every page. So the filter meant to shrink the output runs once the whole listing is already in RAM. On a bucket with a few million objects the process just grows until the kernel takes it out.

Fine โ€” stream it to a file and post-process. That fails differently. One scratch file per bucket, times a few thousand buckets, and suddenly I am managing disk on the admin host. Then writing my own sort and dedup on top of it.

What I wanted was a pipeline where nothing accumulates anywhere. radosgw-admin emits index entries as it walks the shards, jq --stream filters them one event at a time, and redis-cli --pipe pushes the survivors straight into a set. Redis holds the queue and deduplicates for free.

Those two turned out to fit together neatly. One filters without building a tree, the other consumes without building a list. Constant memory at both ends, no scratch space, and it is still just bash.

The pipeline

#!/usr/bin/env bash
set -euo pipefail

BUCKET="${1:?usage: $0 <bucket>}"
MAX_SIZE=4096
TARGET_CLASS=SMALL

radosgw-admin bi list --bucket "$BUCKET" \
| jq -n --stream -j --arg bucket "$BUCKET" \
      --argjson max "$MAX_SIZE" --arg class "$TARGET_CLASS" '
    fromstream(1 | truncate_stream(inputs))
    | select(.type == "plain")
    | .entry
    | select(
        .exists == true
        and .meta.category == 1
        and .meta.size < $max
        and .meta.storage_class != $class
      )
    | .name
    | "*3\r\n$4\r\nSADD\r\n"
      + "$\($bucket | utf8bytelength)\r\n\($bucket)\r\n"
      + "$\(utf8bytelength)\r\n\(.)\r\n"
  ' \
| redis-cli --pipe
flowchart LR
  A[bucket index shards] -->|cls_rgw omap reads| B[radosgw-admin bi list]
  B -->|JSON array, streamed| C[jq --stream]
  C -->|RESP SADD commands| D[redis-cli --pipe]
  D --> E[(Redis set per bucket)]

Out the other end comes a Redis set per bucket, holding exactly the keys that match. I can count it, look at it, and hand it to as many workers as the cluster will tolerate.

Why not lifecycle, and why not copy

Lifecycle transition is the textbook answer and it is a perfectly good one โ€” on a single bucket. RGW supports the ObjectSizeLessThan filter, so a transition rule expresses the whole requirement declaratively and you are done.

At a few thousand buckets it falls apart. Every bucket needs the policy applied, and that writes a visible property to somebody else’s bucket. It shows up in their GetBucketLifecycleConfiguration. It collides with rules they set themselves. And when the migration is finished I get to remove several thousand policies again.

Copying each object onto itself with a new storage class is worse.

Watch out

On a versioned bucket, copy-in-place creates a new version of every object it touches. Storage goes up, not down. It also resets Last-Modified and fires a bucket notification per object, which means a synthetic write event for every object in the cluster.

That is what “touching the bucket” means to me here: changing anything the bucket’s owner or their tooling can observe. Which left the S3 API out entirely, for the search as much as for the move.

Reading the bucket index

Every bucket has an index. It is a set of RADOS objects in the <zone>.rgw.buckets.index pool, each holding an omap โ€” a sorted key/value map in RocksDB. One entry per object, split across shards.

RGW uses it to answer LIST. Nothing stops you reading it directly:

radosgw-admin bi list --bucket mybucket

The manual page calls it “List raw bucket index entries”. Raw is the operative word. One entry:

{
  "type": "plain",
  "idx": "logs/tiny.txt",
  "entry": {
    "name": "logs/tiny.txt",
    "exists": true,
    "meta": {
      "category": 1,
      "size": 512,
      "mtime": "2026-05-01T10:00:00.000000Z",
      "etag": "d41d8cd98f00b204e9800998ecf8427e",
      "storage_class": "",
      "owner": "acct1",
      "content_type": "text/plain",
      "accounted_size": 512
    },
    "flags": 0,
    "versioned_epoch": 0
  }
}

Size, mtime, etag, owner, storage class. Everything a size-based decision needs, and not one S3 request to get it.

Note

The index carries only those fields. No user metadata (x-amz-meta-*), no object tags, no ACLs โ€” those live in the object’s own xattrs. If the requirement had been “every object tagged project=alpha”, none of this would have helped.

Streaming, not parsing

One catch. bi list emits a single JSON array containing every entry, and jq '.[]' parses that whole array before emitting anything. Straight back to the OOM I started with.

jq --stream turns the document into a sequence of [path, value] events rather than building a tree:

radosgw-admin bi list --bucket mybucket \
  | jq -n --stream -c 'fromstream(1 | truncate_stream(inputs))'

truncate_stream drops the array index from each path, so every object gets rebuilt on its own and emitted one at a time. Memory stays flat regardless of how many entries go past.

The filter

Four clauses, each earning its place:

ClauseWhy it is there
.type == "plain"The index also holds instance and olh entries on versioned buckets. Those are not objects.
.exists == truefalse marks entries pending cleanup. The object is already gone.
.meta.category == 11 is Main โ€” a real object. 3 is multipart metadata, 0 is a delete marker.
.meta.size < $maxThe actual requirement.

Take the key from .entry.name, never .idx. The two are identical for plain entries, which is exactly why the mistake survives testing on a bucket that happens not to be versioned. For instance and OLH entries .idx is the raw index key, carrying a 0x80 prefix byte.

Two of these cost me more time than the rest of the pipeline put together:

Watch out

null passes a size filter. An olh entry has no meta object, so .entry.meta.size is null โ€” and in jq, null < 4096 is true. Without the type check, every OLH pointer lands in the queue.

Multipart metadata is tiny. category: 3 entries run 50โ€“100 bytes, so every one of them passes a “smaller than 4 KB” test. On a bucket with a lot of multipart activity they can outnumber the real matches.

One version note. On current stable releases storage_class comes back as an empty string for default placement, while a change in main dumps "STANDARD" instead. Test != "SMALL" and you are right on both. Test == "STANDARD" and you silently match nothing on the older behaviour.

Talking to Redis properly

A set makes a good queue here: deduplicated, cheap to drain, easy to inspect. It also lives off-box, so the machine doing the scanning never accumulates anything โ€” which was the whole point.

redis-cli --pipe describes itself as “Transfer raw Redis protocol from stdin to server”. I ignored the word raw at first and emitted plain SADD mybucket some/key lines. They appear to work, because Redis also accepts inline commands. Inline arguments are split on spaces.

Caution

SADD mybucket reports/Q3 summary.pdf is a valid SADD with two members: reports/Q3 and summary.pdf. SADD is variadic, so there is no error and nothing wrong with the exit status. The queue is simply wrong, and stays wrong until somebody notices.

RESP arrays are length-prefixed and therefore binary-safe:

*3\r\n$4\r\nSADD\r\n$8\r\nmybucket\r\n$22\r\nreports/Q3 summary.pdf\r\n

In jq that means -j for raw output with no added newline, and utf8bytelength for the lengths.

Important

utf8bytelength, not length. RESP counts bytes, so the first key with a non-ASCII character in it would desynchronise the whole stream.

Across the fleet

Keying the set by bucket name means the rest composes for free:

radosgw-admin bucket list | jq -r '.[]' | while read -r bucket; do
  scan_bucket "$bucket"
done

radosgw-admin bucket list returns every bucket in the zone as a JSON array of names.

Do not run a few thousand buckets as one loop on one host, though. Partition the list across a handful of workers, and track finished buckets in a second Redis set so the run survives being stopped. That last part turned a multi-day job into one I could interrupt whenever the cluster got busy.

radosgw-admin bucket stats --bucket B is worth a call before committing to a full index walk. It returns object counts and sizes without touching the index, which rules out empty buckets immediately. size_actual / num_objects also gives a mean object size, good enough to skip a bucket that is obviously all large objects.

Note

bucket stats reports per category, aggregated across all storage classes, so it cannot tell you how much of a bucket is already migrated. Count that yourself as the workers drain each set.

What it costs

bi list reads every entry in every index shard. Index pools sit on SSD or NVMe because they are latency-sensitive, and this is a long sequential read against them while they are still serving live traffic.

Caution

Never run this during a reshard. Entries move underneath you and the output is inconsistent.

Run it off-peak. --shard-id walks one shard at a time if you want to spread the load out further, and --max-entries caps how much comes back per listing operation.

It is a point-in-time snapshot rather than a transaction. Objects written during the walk may or may not show up, and objects deleted during it may still appear. For a migration queue that is fine, as long as the worker tolerates a key that has since vanished.

Handing off to the mover

Splitting discovery from the move was deliberate. Discovery is read-only and restartable; moving data is neither. A mover that dies halfway through loses nothing.

SPOP with a count feeds workers without them colliding, since it reads and removes in one operation:

while mapfile -t batch < <(redis-cli --raw SPOP "$BUCKET" 100) \
      && ((${#batch[@]})); do
  for key in "${batch[@]}"; do
    migrate_object "$BUCKET" "$key"
  done
done

Note

redis-cli --raw separates values with newlines, so a key containing a newline breaks this loop. S3 permits those. It is fine for most real buckets, but if you cannot rule it out, drain the set from a proper client library rather than bash.

The target storage class has to exist before anything can move into it:

radosgw-admin zonegroup placement add \
  --rgw-zonegroup=default --placement-id=default-placement \
  --storage-class=SMALL

radosgw-admin zone placement add \
  --rgw-zone=default --placement-id=default-placement \
  --storage-class=SMALL --data-pool=default.rgw.buckets.data.nvme

Which leaves migrate_object, and that is where it got interesting.

It cannot be aws s3 cp, for every reason above. The command with the right shape is radosgw-admin object rewrite. It never goes near the S3 request path. Reading the implementation, it also preserves the object’s original mtime rather than stamping a new one, and rewrites in place instead of creating a version. Exactly the invisible-to-tenants behaviour I needed.

Important

RGWRados::rewrite_obj erases the stored storage-class attribute and rewrites into the bucket’s own placement rule, and there is no --storage-class option on the command. Out of the box, object rewrite moves objects to the bucket default โ€” the opposite of what I wanted.

So stock radosgw-admin hands you a mover with every property you need and no way to aim it. We ended up adding the missing option ourselves, which is its own post.

Closing

Bucket listing is the wrong tool once a bucket gets large. It is an API built for pagination, not for analysis. And if the migration has to stay invisible to the people who own the data, the whole S3 layer is wrong. Everything it offers leaves a mark.

The index already holds what I was filtering on. radosgw-admin bi list hands it over with none of that in the way, and once nothing in the pipeline buffers, bucket size stops mattering.

Two follow-ups: the Lua script that routes new uploads, and teaching object rewrite a target storage class so the backlog can actually be drained.