Projects / MVR-Data

MVR-Data

The Overview chapter includes:

MVR-Data is a package format and reference library for multi-vector retrieval data. It keeps original multimodal objects, vector representations, and long-form relevance judgments in explicit Arrow tables so publishers and consumers share the same identities and physical schemas.

Package model

Every package is a self-contained directory with a manifest.json, zero or more Arrow IPC File shards for each table binding, and a checksums.sha256 file. A package has exactly one physical kind:

  • Raw packages store ordered components for each base and query object. Component payloads are content-addressed files below assets/.
  • Embedded packages store one or more ordered vectors for each base and query object. Dimension and numeric dtype are fixed package-wide.
  • Both kinds store ground truth as long-form query-object judgments with a relevance level, split, judgment source, and annotation pool.

Raw and Embedded packages remain independent. When they describe the same logical collection, matching objects use the same object_id; consumers join the two representations by ID rather than row position.

Available now

The repository currently provides a header-only C++20 interface under include/mvr_data/ and a compiled mvr_data Python package under python/:

  • canonical Raw, Embedded, and ground-truth Arrow schema factories;
  • conversion between manifest dtype names and Arrow numeric types;
  • convention-based JSON Manifest loading into strongly typed metadata;
  • package readers that stream validated RecordBatches or materialize a table;
  • sorted whole-package SHA-256 checksum generation and verification;
  • safe package-relative path validation and symlink-aware file resolution;
  • the CMake interface target MVRData::mvr_data;
  • same-named Python classes and methods implemented by a native pybind11 extension;
  • native PyArrow schemas, dtypes, readers, and tables exchanged through Arrow's C Data and C Stream capsule protocols;
  • inline Python type stubs and a py.typed marker.

MVR-Data itself is header-only, but Apache Arrow and LibRHash are compiled dependencies. The current CMake project fetches pinned dependency sources and builds them locally.

Current boundaries

The native API is a compact reference implementation, not a distributed data engine. It reads existing packages but does not yet provide a public writer or full semantic validator. In particular, opening a package validates Manifest shape, vector settings, shard path spelling, and canonical schemas; it does not by itself verify checksums.sha256, object-ID uniqueness, ground-truth foreign keys, or every format-level row constraint.

The current C++ and Python APIs are read-only: neither provides a public package writer, CLI, or full semantic validator. Python package installation builds a platform-specific extension from source; a prebuilt package index release is not published yet. Some retained Python tests continue to describe the future writer and CLI behavior rather than the active binding.

Documentation map