This is an automated email from the ASF dual-hosted git repository.
XiaoHongbo-Hope pushed a change to branch release-2.1
in repository https://gitbox.apache.org/repos/asf/paimon.git
from 6bfd8418dc [core] Optimize tag reads during snapshot expiration (#9967)
new b99f5b9955 [e2e] Strip any trailing Spark error log from spark-sql
output before comparison (#9971)
new 6bc31fa883 [eslib] Do not require build-time es-index options when
reading persisted indexes (#9969)
new 239321dd82 [format] Validate the avro block size before creating the
writer (#9961)
new e56ffee19c [flink] Fix GlobalFullCompactionSinkWrite re-reading
snapshots on every checkpoint after a full compaction (#9959)
new 8360a5614d [python] Support batch vector search with Ray (#9950)
new 23f5644052 [python] Consolidate composite writer lifecycle and file
output (#9948)
new 7f236491ed [python] Reuse Parquet fragment metadata for single-file
reads (#9881)
new c5d59d0ff0 [core] Add adaptive BTree posting list encoding (#9981)
new b4e2801e8e [python] Fix flaky Daft-on-Ray CI failures (#9975)
new 6679a73de3 [python] Avoid redundant blob stream opens on index cache
hits (#9974)
new 6dbd235225 [python] Fix MAP and MULTISET type JSON serialization
(#9682)
new 306120334c [python] Reduce LeRobot DataLoader worker startup time
(#9888)
new 8ce380d29b [release] Align Spark 4.0 UT parent version with 2.1
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci-python.yml | 1 +
docs/docs/pypaimon/multimodal-search.md | 29 +-
.../paimon/globalindex/btree/BTreeFileFooter.java | 13 +-
.../globalindex/btree/BTreeGlobalIndexer.java | 3 +-
.../globalindex/btree/BTreeIndexOptions.java | 9 +
.../paimon/globalindex/btree/BTreeIndexReader.java | 56 +++-
.../paimon/globalindex/btree/BTreeIndexWriter.java | 56 +++-
.../paimon/globalindex/btree/BTreePostingList.java | 307 +++++++++++++++++++
.../apache/paimon/utils/RoaringNavigableMap64.java | 25 +-
.../globalindex/btree/AbstractIndexReaderTest.java | 7 +-
.../globalindex/btree/BTreeIndexOptionsTest.java | 6 +
.../globalindex/btree/BTreeIndexReaderTest.java | 57 +++-
.../globalindex/btree/BTreePostingListTest.java | 232 ++++++++++++++
.../paimon/utils/RoaringNavigableMap64Test.java | 26 ++
.../apache/paimon/tests/SparkE2eOutputTest.java | 73 +++++
.../java/org/apache/paimon/tests/SparkE2eTest.java | 36 ++-
.../eslib/index/ESIndexGlobalIndexReader.java | 28 +-
.../paimon/eslib/index/ESIndexGlobalIndexer.java | 44 ++-
.../apache/paimon/eslib/index/ESIndexOptions.java | 21 +-
.../eslib/index/ESIndexVectorMetricTest.java | 116 +++++++
.../flink/sink/GlobalFullCompactionSinkWrite.java | 2 +-
.../sink/GlobalFullCompactionSinkWriteTest.java | 147 +++++++++
.../apache/paimon/format/avro/AvroFileFormat.java | 23 +-
.../paimon/format/avro/AvroFileFormatTest.java | 29 +-
.../pypaimon/multimodal/lerobot/dataset.py | 19 ++
paimon-python/pypaimon/multimodal/query.py | 29 +-
paimon-python/pypaimon/ray/batch_vector_search.py | 116 +++++++
paimon-python/pypaimon/ray/vector_search.py | 25 +-
.../pypaimon/read/reader/format_blob_reader.py | 11 +-
.../pypaimon/read/reader/format_pyarrow_reader.py | 7 +
paimon-python/pypaimon/schema/data_types.py | 5 +-
.../pypaimon/table/source/vector_search_read.py | 29 +-
paimon-python/pypaimon/tests/blob_test.py | 253 +++++++++++++---
.../pypaimon/tests/daft/daft_on_ray_e2e_test.py | 2 +
paimon-python/pypaimon/tests/data_types_test.py | 59 ++++
.../pypaimon/tests/multimodal_lerobot_test.py | 77 +++++
paimon-python/pypaimon/tests/native_plan_test.py | 34 +++
.../pypaimon/tests/parquet_metadata_cache_test.py | 130 ++++++--
.../pypaimon/tests/ray_batch_vector_search_test.py | 335 +++++++++++++++++++++
.../pypaimon/tests/write/write_buffer_test.py | 17 +-
.../pypaimon/write/writer/composite_data_writer.py | 186 ++++++++++++
.../pypaimon/write/writer/data_vector_writer.py | 152 +---------
paimon-python/pypaimon/write/writer/data_writer.py | 89 +++---
.../write/writer/dedicated_format_writer.py | 197 +-----------
paimon-spark/paimon-spark-ut-4.0/pom.xml | 2 +-
.../paimon/spark/sql/DeletionVectorTest.scala | 3 +-
46 files changed, 2573 insertions(+), 550 deletions(-)
create mode 100644
paimon-common/src/main/java/org/apache/paimon/globalindex/btree/BTreePostingList.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/btree/BTreePostingListTest.java
create mode 100644
paimon-e2e-tests/src/test/java/org/apache/paimon/tests/SparkE2eOutputTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/sink/GlobalFullCompactionSinkWriteTest.java
create mode 100644 paimon-python/pypaimon/ray/batch_vector_search.py
create mode 100644 paimon-python/pypaimon/tests/ray_batch_vector_search_test.py
create mode 100644 paimon-python/pypaimon/write/writer/composite_data_writer.py
