Validate legacy IVFFlat per-list codes size against ids size

[palios-taey]

Summary

The legacy IvFl/IvFL on-disk inverted-list formats in read_index_up serialize per-list ids and codes as two independent READVECTOR calls with no cross-check between their lengths. A malformed file can produce a list where codes[i].size() does not equal ids[i].size() * code_size.

This matters at the CPU → GPU clone path: GpuIndexIVFFlat::copyFrom → IVFBase::copyInvertedListsFrom → IVFBase::addEncodedVectorsToList_ derives its memcpy length from list_size (which derives from ids.size()) and copies that many bytes out of the codes buffer. When codes[i] is shorter than ids[i].size() * code_size, the memcpy reads past the heap allocation.

The current ilar/IwFl reader (read_InvertedLists_up) already validates this invariant via the mul_no_overflow + FAISS_THROW_IF_NOT_FMT pattern. This PR brings the legacy IvFl/IvFL branches in read_index_up to parity with that validation.

Reproduction (pre-patch)

A malformed legacy IVFFlat file with d=128, nlist=1, ids[0].size()=1, codes[0].size()=511 bytes driven through the public API path faiss::read_index() → faiss::gpu::index_cpu_to_gpu() produces (host AddressSanitizer):

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x... at pc 0x...
READ of size 512 at 0x... thread T0
    #0 memcpy
    #1 faiss::gpu::IVFBase::addEncodedVectorsToList_
    #2 faiss::gpu::IVFBase::copyInvertedListsFrom
    #3 faiss::gpu::GpuIndexIVFFlat::copyFrom
    #4 faiss::gpu::ToGpuCloner::clone_Index
    #5 faiss::gpu::index_cpu_to_gpu

0x... is located 0 bytes after 511-byte region
allocated by thread T0 here:
    ... faiss::MaybeOwnedVector<unsigned char>::resize
    ... faiss::read_index_up

The read is 512 bytes from a 511-byte heap allocation — boundary-minimal 1-byte over-read. The same shape fires deterministically across seeds with codes sizes 64..511 bytes (per-seed mapping available on request).

After the patch

The same file now produces a clean throw at parse time, before the GPU clone path is reached:

Error in faiss::read_index_up(IOReader*, int)::<lambda(size_t)> at faiss/impl/index_read.cpp:1843:
Legacy IVFFlat inverted list 0: codes size 511 bytes does not match ids size 1 * code_size 512 = 512 bytes

The memcpy site in IVFBase::addEncodedVectorsToList_ is never reached.

Scope

The patched lane affects consumers who load a legacy IvFl/IvFL-format .index file and clone it to GPU via faiss::gpu::index_cpu_to_gpu. Pure CPU-side consumers of legacy files that don't exercise the GPU clone path were not demonstrated to hit this exact sink; this PR is the parity fix at the reader. The sink (IVFBase::addEncodedVectorsToList_(idx_t, const void*, const idx_t*, idx_t)) takes the codes as const void* with no length and physically cannot validate its own input — the reader is the only layer that knows both the buffer and its true size, which is why the fix lives there.

Verification

Built and verified on the actual target hardware: NVIDIA GB10 (Blackwell, aarch64), driver 580.95.05, CUDA 13.0. Build flags -fsanitize=address,undefined -fno-omit-frame-pointer -g -O1. Harness invokes faiss::read_index(IOReader*) then faiss::gpu::index_cpu_to_gpu(provider, device, index, opts), deterministic 3× repro of the original failure. Pre-patch: all 7 adversarial seeds (codes sizes 64, 128, 256, 320, 384, 511 bytes against expected 512; and one with ids[0].size()=4, codes=512 bytes against expected 2048) fire heap-buffer-overflow READ. Post-patch: all 7 fire the new FAISS_THROW_IF_NOT_FMT at parse time and the GPU clone path is never reached.

Affected file

faiss/impl/index_read.cpp — 27 insertions, 0 removals. No interface change, no on-disk format change. Well-formed legacy files (where codes.size() == ids.size() * code_size, which the writer preserves) are unaffected.

Follow-up (out of scope here)

The deeper fix — deprecating the legacy IvFl/IvFL format in favor of the validated current IwFl format — is a maintainer policy call rather than something to bundle in this PR. Happy to file a separate discussion issue if useful.

🤖 Generated with Claude Code

[meta-cla]

Hi @palios-taey!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!