JingsongLi opened a new pull request, #10776:
URL: https://github.com/apache/paimon/pull/10776
### Purpose
Add `scan().export_lance()` to stream multimodal scan results into an
independent Lance dataset, honoring filters, projections, limits, and
snapshot/tag selection. Scalar BLOB payloads use native Lance Blob storage for
lazy reads. ARRAY/MAP BLOB values remain binary; MAPs are exported as ordered
key/value lists.
Use `pylance>=13,<14` on Python 3.10+ to preserve Blob NULL values across
batches. Standalone exports use Lance format 2.1. Keep Paimon's internal file
writers on their existing 2.0 default, preserve explicit version overrides, and
retain the older Lance dependency ranges for Python 3.8/3.9.
Document the export API, target storage options, native Blob reads, and
unsupported video-frame/VARIANT selections.
### Tests
With Python 3.13, PyArrow 22.0.0, and pylance 13.0.0: **44 passed**.
```shell
cd paimon-python
python -m pytest -q \
pypaimon/tests/lance_utils_test.py \
pypaimon/tests/multimodal_lance_test.py \
pypaimon/tests/reader_primary_key_test.py::PkReaderTest::test_pk_lance_reader \
pypaimon/tests/reader_primary_key_test.py::PkReaderTest::test_pk_lance_reader_with_filter
\
pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest::test_vector_dedicated_format_write_read_lance
\
pypaimon/tests/hdfs_native_test.py::HdfsNativeWriteFormatTest::test_write_lance_delegates_to_lance_specified
```
With pylance 0.39.0: `python -m pytest -q
pypaimon/tests/lance_utils_test.py` — **9 passed**.
The tests cover native Blob seek/read, NULL/empty values across batches,
source-independent payloads, snapshot/schema consistency, failure cleanup and
atomic publication, and compatible internal file versions. Also verified that
Lance 13 files written as 2.0 can be read by Lance 0.39.
Changed Python files pass Flake8 with `paimon-python/dev/cfg.ini`; `git diff
--check` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
