Python Binding

The mvr_data package is a thin pybind11 layer over the C++20 reference implementation. It does not shell out or reimplement format behavior in Python. Public classes and methods keep the C++ names, while Arrow values are presented as native PyArrow objects.

Create the build environment

The repository includes a reproducible Conda environment with Python 3.12, GCC/G++ through cxx-compiler, CMake, Ninja, pybind11, scikit-build-core, PyArrow 24, and pytest:

git clone https://github.com/WittenYeh/MVR-Data.git
cd MVR-Data
conda env create -f environment.yml
conda activate mvr-data

The package metadata supports Python 3.10 or newer and PyArrow 14 or newer. The checked-in environment pins the versions used for the project's clean build and compatibility tests, including a modern C++ runtime for the native module.

Build and install

Build a platform-specific wheel from the checkout and install it into the active environment:

python -m pip install . --no-build-isolation

The first build fetches the pinned Apache Arrow, nlohmann/json, and LibRHash sources through CMake and compiles them locally. --no-build-isolation reuses the compiler-facing pybind11 and scikit-build-core packages from the Conda environment. PyArrow remains a runtime dependency.

For an in-tree development build with both test suites enabled:

cmake -S . -B build -G Ninja \
  -DMVR_DATA_BUILD_PYTHON=ON \
  -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure
PYTHONPATH="$PWD/build/python" python -m pytest -q tests/test_python_bindings.py

Same-named public surface

Importing mvr_data exposes PackageKind, TableRole, TableInfo, VectorConfig, Manifest, Schema, DataReader, and Checksum. Each corresponds directly to the C++ type with that name, and methods such as Manifest.load, DataReader.open, DataReader.get_batched_scanner, DataReader.read_table, Checksum.refresh, and Checksum.verify retain their C++ method names.

The source-linked Schema API, Manifest API, Reader API, and Integrity API define the shared behavioral contracts. Python signatures and return annotations are kept in the package's inline type stubs.

Python accepts str, os.PathLike[str], and pathlib.Path wherever the C++ API accepts std::filesystem::path. C++ std::optional results become either their value or None, table shard vectors become list[str], and Manifest extensions become ordinary nested Python dictionaries and values.

Arrow interoperability

The binding maps Arrow values without a Python-side schema model:

Schemas and dtypes cross the extension boundary through the Arrow C Data PyCapsule protocol. Batch readers use the Arrow C Stream protocol. This avoids serialization between the statically linked C++ core and the PyArrow runtime, and allows Arrow buffers to be shared without a serialization copy.

PyArrow types can also be passed back into the native API:

import pyarrow as pa

import mvr_data

dtype = mvr_data.Schema.parse_vector_dtype("float32")
assert dtype == pa.float32()

schema = mvr_data.Schema.make_embedded_object_schema(
    dimension=128,
    dtype=pa.float32(),
)
assert isinstance(schema, pa.Schema)

Stream or materialize a table

Open a package with the same factory and table-role names used in C++:

from pathlib import Path

import mvr_data

package = Path("/data/example")
mvr_data.Checksum.verify(package)

reader = mvr_data.DataReader.open(package)
base_role = mvr_data.TableRole.base()

for batch in reader.get_batched_scanner(base_role):
    consume(batch)

get_batched_scanner returns a pyarrow.RecordBatchReader. It preserves Manifest shard order and keeps the C++ reader's lazy, one-shard-at-a-time I/O behavior. Errors discovered while advancing the stream are raised by the PyArrow reader.

Use read_table only when the complete table fits in memory:

base = reader.read_table(mvr_data.TableRole.base())
print(base.schema)
print(base.num_rows)

The result is a pyarrow.Table, so downstream PyArrow operations work without an MVR-Data-specific adapter.

Read typed Manifest metadata

Manifest accessors use the same names as their C++ counterparts:

manifest = reader.manifest()

print(manifest.data_name(), manifest.data_version())
print(manifest.table_info(mvr_data.TableRole.query()).shards)

if manifest.package_kind() == mvr_data.PackageKind.embedded:
    vector = manifest.vector_config()
    assert vector is not None
    print(vector.dimension, vector.dtype, vector.scoring)

TableInfo.schema and VectorConfig.dtype are PyArrow values. Optional description, license, source, vector configuration, and extensions accessors return None when their Manifest fields are absent.

Python exceptions

Non-OK Arrow statuses become familiar Python exception categories:

Potentially blocking Manifest, package, shard, checksum, and filesystem operations release the Python GIL while the C++ work runs. Error messages retain the original Arrow status text for diagnosis.

Static typing

The wheel includes mvr_data/__init__.pyi and a py.typed marker. Type checkers therefore see concrete PyArrow return types, optional Manifest fields, path-like inputs, and the same-named class methods without separate stub installation.

The current package remains read-only. A package writer, CLI, and full semantic validator are outside this binding's public surface.