[improve][pip] PIP-486: Scalable Topic Key-Shared Consumption

[merlimat]

Motivation

PIP-468 and
PIP-483 give scalable topics a DAG of
range segments whose steady-state ordered-consumption model is one consumer per segment. Several
situations need key-shared (per-message-key) ordered consumption instead: draining a sealed-segment
backlog after a scale-up, consolidating many low-throughput topics, and getting consumer parallelism
beyond the max-segments ceiling.

Today, key-shared dispatch couples the producer's batching mode to the consumer's subscription mode
(the producer must disable batching or use the key-based batcher). This PIP removes that coupling for
scalable topics.

Modifications

Adds the design document pip/pip-486.md. In summary:

• Entry-bucketing: a second, independent 16-bit hash (hashB, the low half of the same 32-bit key
  hash whose high half drives segment routing) divides each segment into a configurable number of
  buckets; the producer keeps each batch within one bucket and stamps the bucket's hashB range in the
  outer MessageMetadata.
• Routing by range: the broker dispatches a whole entry to the consumer that owns that bucket — no
  per-key hashing, no decompression, one entry to exactly one consumer.
• Per-topic bucket budget, divided across segments (a split halves a segment's buckets); bucket
  count is immutable per segment, changed only by a controller-driven "rebucket rollover" (a no-op
  split reusing the PIP-468 seal/redirect flow).
• Controller-driven bucket→consumer assignment; reassignment reuses the existing Key_Shared
  blocked-hash handling, tracking pending state per bucket. No shared-entry ack machinery.

This is a documentation-only PR (the design proposal). Implementation will follow in separate PRs.
A discussion thread will be started on dev@pulsar.apache.org.

[lhotari]

Rather than fixing the solution to equal-sized hash buckets, it would be useful to support splitting a hot shard on arbitrary boundaries based on key statistics. Message keys are often derived from business entities (tenant, account, or device IDs) whose traffic is heavily skewed—a handful of keys can dominate throughput while most contribute little. Equal-sized buckets can't isolate such hot keys, so adapting split boundaries to the observed distribution would balance load far more effectively. This wouldn't have to be implemented from the start, but it would be great if the design could be kept open for extension in this area.