This is an automated email from the ASF dual-hosted git repository.
XiaoHongbo-Hope pushed a change to branch release-0.4
in repository https://gitbox.apache.org/repos/asf/paimon-rust.git
from 0139f7a0 Upgrade paimon-vindex-core to 0.5.0 (#879)
add d9718da6 [rust] Bound sparse BlobDescriptor range reads (#883)
add d05cb03a fix: Map precision-zero timestamps to Arrow seconds (#884)
add f33dee15 fix: make Python record batch reader cancellable (#885)
add 30c171d3 feat: expand Python native reads with nested projection and
row kinds (#887)
add afc8248f perf: optimize scalar index reads and support BTree V2 (#888)
add eefe3ece feat(python): deserialize cross-language splits (#889)
add b67e6e8d feat(python): plan chunk-shuffled reads with IndexedSplit
(#890)
add a48c01b1 perf(vindex): optimize IVF index build reads (#800)
add 157406bd feat: support append row-position selection in native scans
(#893)
add d6bc0024 Expose combined changelog scans to Python (#894)
add 07c0fb45 fix(datafusion): hand off the worker before blocking in
catalog callbacks (#891)
add db380ca3 refactor(vindex): trim granule planning overhead (#892)
add 51a8ce9e feat(read): bound data evolution BLOB reads by LIMIT (#896)
add ded9e59f fix: remove manifest minor compaction (#899)
add 2f077fc9 feat(read): preserve physical row IDs with data predicates
(#900)
add 0179439d feat(read): load BLOB views lazily under LIMIT (#902)
add 9122eb19 feat(read): avoid unselected managed BLOB payloads under
LIMIT (#901)
add 2fa48384 fix(table): reject duplicate ADD and DELETE file entries
(#904)
add 1aa6c493 Support PK partial-update and aggregation reads with deletion
vectors (#905)
add 4f5e3abd fix(read): stop BLOB predicate reads at LIMIT (#903)
add c040921b Support BLOB and row-id predicates in Python native reads
(#906)
add 7715f5c2 Align native read schema evolution, ROW decoding, and merge
semantics (#907)
add b2e53fa5 fix(global-index): stop reading index keys with a comparator
that can panic (#909)
add 5013b01e fix(lumina): reproduce Java's table, index, query option
order (#908)
add eb6a4f30 fix(lumina): reject non-finite vector elements when building
an index (#898)
add 444da2a1 fix(lumina): store the canonical native metric name in index
metadata (#895)
add 0601db4c fix(scan): keep the predicate when a partition set cannot
express it (#886)
add 2e1087a6 fix(vortex): reject columns the writer cannot convert instead
of panicking (#878)
add 14610088 feat(file_index): support range bitmap append writes (#877)
add 21f60b80 test(file_index): verify Java table read compatibility (#876)
add 5de0ba34 feat(datafusion): add REST management procedures (#837)
add aa4efcab perf(datafusion): push down partition-grouped COUNT(*) to
manifests (#858)
add ef540b1e fix(commit): align TableCommit validation and retries with
Java (#913)
add 3ce22437 feat: Add Java-compatible CommitMessage serialization and
Python commit API (#912)
add 5aeb1144 fix(commit): close remaining TableCommit parity gaps (#915)
add c270543f [python] Preserve external batch commit identity (#916)
add f73ebe4e feat: Support Java-compatible scan.timestamp time travel
(#919)
add 86f0cae8 feat: integrate reader memory reservations with DataFusion
(#914)
add 9978b9dc [rust] Account writer memory with ResourceContext (#928)
add f5ee1c01 [rust] Reduce I/O requests for MAP BLOB descriptor reads
(#920)
add af57e751 fix(spec): don't prune files that may hold NaN on NOT(=
float) (#926)
add 69b68a91 fix(datafusion): reject unknown arguments for tag and index
procedures (#917)
add 544ea703 fix(python): declare read-scan methods in the datafusion type
stub (#918)
add 07f11467 fix(variant): emit valid JSON for non-finite floats in
to_json (#927)
add 42392f50 fix(arrow): scale INTEGER/BIGINT epoch seconds evolved to
TIMESTAMP (#924)
add 944e3893 fix(c): guard predicate AND/OR/NOT against null inputs (#923)
add 07d33074 build: align ASF configuration with Paimon (#930)
add 089861ff fix(table): keep PK sequence numbers advancing across
TableWrite reuse (#925)
add f5b27f30 feat(c): expose reader memory reservations through
ResourceContext (#931)
add bb4f8bb3 feat(c): attach ResourceContext to table writers (#933)
add aa60a6ac Honor data-file prefix in Python native writes (#932)
add e16b01c5 Give postpone writers distinct Java-style write IDs (#935)
add e357a660 [python] Construct native tables from resolved REST responses
(#922)
add dd3bc614 feat: support more scalar types in incremental diff (#940)
add c8e18ccb fix: write value stats for primary-key parquet files (#938)
add dd6c95ae Support nested and vector values and deletion vectors in
batch Diff (#944)
add 0923cbc5 Support BSI indexes and indexed native write paths (#947)
add 8cbafd4d Support native Format Table append and overwrite writes (#948)
add b2f1eb3c Support Avro writes, defaults, and custom partition locations
for native Format Tables (#950)
add b8073f20 Support Mosaic writes for native Paimon tables (#952)
add 9746004b Support managed BLOBs in native primary-key Parquet tables
(#954)
add b116e665 feat(go): expose shared resource reservations (#949)
add 78ae3fe0 test(file_index): verify Java range-bitmap table reads (#953)
add 442dfd55 fix(variant): truncate decimals and floats in variant_get int
casts (#945)
add 7cc54602 docs(python-binding): auto incremental mode follows
changelog-producer (#943)
add 2148923f feat(datafusion): support the $aggregation_fields system
table (#937)
add 9b5245be Align merge engines and all field aggregators with Java (#955)
add 169db37c Align native table writes and merge ordering with Java (#958)
add 5a780c4e Support nested ROW data evolution in native table reads and
writes (#960)
add b85a380c Extend Format Table support to ORC, CSV, JSON, TEXT and
Mosaic (#959)
add 2f513757 Expose core table upsert and native updates to PyPaimon (#961)
add 50725c84 fix(core): complete native update partition and deletion
vector paths (#969)
add 5c87ec53 fix(core): close gaps for PyPaimon native Parquet writers
(#970)
add 952f6b81 [core] Complete nested updates and partition-aware temporal
upserts (#973)
add b8fbc1c3 [core] Support configured data directories throughout table
IO (#974)
add 52b4bb23 [auth] Support local-file DLF token loading (#936)
add 833edacc [rust] Cache decoded BLOB indexes across readers (#897)
add 603ecdec [rust] Cache parsed Parquet metadata across readers (#929)
add fc27163e [core] Honor external data paths across table writers (#981)
add 73aab79b [core] Honor scan.manifest.parallelism during planning (#988)
add 91a8b21d [core] Avoid excess manifest range reads with sidecars (#990)
add 0514f9a1 [core] Align dedicated Blob writes and URI resolution with
Java (#989)
add 6267e9c0 [rust] Reuse BTree data blocks across index lookups (#995)
add c9cf6c82 [python] Add batch numeric extraction for Variant fields
(#997)
add 2d0dda18 feat(storage): support Jindo for OSS scan planning (#696)
add c7735709 feat(catalog): support tag creation with retention (#1000)
add cd5d36c4 feat(datafusion): add SQL options to SQLContext (#996)
add 89ec844b [core] Unify the REST User-Agent format (#992)
add 73ab9c33 [core] Unify the object storage User-Agent format (#991)
add a2414a9d fix(variant): store shredded timestamps as microseconds (#983)
add d81476a0 feat(spec): support the Avro deflate codec (#982)
add 3be82f50 fix(arrow): read CSV boolean columns like Java's CsvParser
(#980)
add 519792a5 fix(arrow): convert dictionary and fixed-size-binary Arrow
types (#978)
add 297965a5 feat(datafusion): support the $buckets system table (#934)
add 69118de8 feat(auth): authorize query-auth reads and carry the grant on
the split (#758)
add b2642a59 [io] Reduce BLOB footer and index reads with tail prefetch
(#1010)
add d57e85a9 [jindo] Fix unsafe fallback after process fork (#999)
add be6d5094 [core] Align FileType classification and mutability with Java
(#1013)
add 0216923f [rust] Parallelize OR global-index planning across files
(#1011)
add b9bab593 perf(btree): load null bitmap on demand (#1008)
add 18fc5797 fix(datafusion): map column-less MERGE INSERT VALUES by
position (#984)
add ea07f5c0 fix(datafusion): guard ALTER TABLE DROP PARTITION against
time travel (#979)
add fcb8eada fix(rest): preserve column errors through the REST catalog
round trip (#977)
add 783a0e3f fix(spec): reject non-primitive primary-key and partition-key
types (#976)
add c41ce209 feat(btree): support composite global indexes (#1015)
add d2462936 [rust] Replace BLOB index entry limit with a memory budget
(#998)
add 0a4470e8 Align dynamic bucket writes and HASH index recovery with Java
(#1016)
add 66b14a0d Extend native cross-partition and postpone writes (#1017)
add bede6650 Align postpone retract validation and managed Blob writes
with Java (#1020)
add 5a81a6d9 Align MAP writes and Variant shredding factories with Java
(#1027)
add 6555d4c3 feat(io): add optional OSS C++ SDK read backend (#1028)
add ce13e8f9 feat(c,go): list databases and tables through the C and Go
bindings (#1024)
add 6342c2a6 feat(datafusion): add CALL sys.rename_tag (#1023)
add 5c22b080 feat(go): expose string and range predicates through the Go
binding (#1022)
add 0c6b5118 perf(table): parallelize compound AND index evaluation (#1012)
add 4bf77696 [python] Push batched Variant extraction into native reads
(#1009)
add 23656594 fix(spec): match Java whitespace for CHAR/VARCHAR partition
values (#985)
add 6b0d5c59 fix(arrow): read ragged CSV format-table rows permissively
(#975)
add 32331329 fix(spec): reject a non-positive or non-integer bucket number
(#956)
add 1707abc8 feat(go): support a read-builder limit hint for split pruning
(#946)
add 94872428 feat(datafusion): support the rename_branch procedure (#942)
add 8c3527be Align row-tracking commit metadata and conflicts with Java
(#1029)
add e2872baf Support Java-aligned Paimon branch writes (#1033)
add d53f64a0 Align inline Blob layouts and updates with Java (#1039)
add 283f1f93 Drive nested, MAP and VARIANT projections from structured
read types (#1041)
add f58be6f2 [core] Write BLOB collections and read packed video files
(#1044)
add 3ba159ca [core] Align commit failure cleanup with Java (#1045)
add 066f8078 [core] Support Java Data Evolution ROW sidecars (#1047)
add 2457d30c [core] Support row-tracking sequence projections and
predicates (#1051)
add 326af03c [python] Discover an optionally packaged OSS C++ bridge
(#1043)
add dd230dbc [parquet] Support per-column dictionary encoding options
(#1034)
add b262b5f8 [core] Apply primary-key BLOB filters and limits before
payload resolution (#1053)
add 174ea898 [core] Support column-subset append writes and upserts (#1054)
add 8e472c9f feat(table): build and drop full-text global indexes (#963)
add 303ac287 perf(table): skip redundant BTree not-null queries (#1038)
add 0452c6ee [core] Support precomputed bucket writes and snapshot
restoration (#1056)
add dd25596e [core] Support Java worker sharding in native scans (#1059)
add ca54d4a4 Preserve supplied fields across key upsert batches (#1061)
add e51971ef perf(arrow): evaluate integer IN with a bitmap or binary
search (#1058)
add 29600d99 perf(arrow): read Variant metadata as a dictionary for field
extraction (#1057)
add b560965d fix(arrow): map NULL to false in decimal residual leaf masks
(#1030)
add 820f0dc4 feat(predicate): read Java's string temporal and decimal
literals in REST predicate JSON (#1018)
add 29727bfe Support BlobConsumer for native append writes (#1063)
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 4 +-
.github/workflows/ci.yml | 80 +-
Cargo.lock | 17 +
benchmarks/tpcds/DEPENDENCIES.rust.tsv | 1 +
bindings/c/Cargo.toml | 1 +
bindings/c/DEPENDENCIES.rust.tsv | 880 +++---
bindings/c/src/catalog.rs | 81 +-
bindings/c/src/error.rs | 39 +
bindings/c/src/lib.rs | 1 +
bindings/c/src/resource.rs | 89 +
bindings/c/src/result.rs | 12 +
bindings/c/src/table.rs | 137 +-
bindings/c/src/tests.rs | 468 +++
bindings/c/src/types.rs | 68 +
bindings/c/src/write.rs | 67 +
bindings/go/DEPENDENCIES.rust.tsv | 880 +++---
bindings/go/catalog.go | 71 +
bindings/go/error.go | 19 +-
bindings/go/postpone_fixed_bucket_write.go | 20 +-
bindings/go/postpone_fixed_bucket_write_ffi.go | 12 +
bindings/go/predicate.go | 162 ++
bindings/go/read_builder.go | 69 +
bindings/go/resource.go | 124 +
bindings/go/tests/catalog_tag_test.go | 35 +
bindings/go/tests/paimon_test.go | 105 +
bindings/go/tests/resource_test.go | 209 ++
bindings/go/types.go | 77 +
bindings/go/write.go | 20 +-
bindings/go/write_ffi.go | 12 +
bindings/python/Cargo.toml | 1 +
bindings/python/README.md | 108 +-
.../{Makefile => python/pypaimon_rust/data.pyi} | 14 +-
.../python/python/pypaimon_rust/datafusion.pyi | 233 +-
bindings/python/src/context.rs | 144 +-
bindings/python/src/error.rs | 133 +-
bindings/python/src/lib.rs | 5 +
bindings/python/src/oss_cpp.rs | 153 +
bindings/python/src/predicate.rs | 39 +-
bindings/python/src/read.rs | 342 ++-
bindings/python/src/schema.rs | 4 +
bindings/python/src/table.rs | 63 +-
bindings/python/src/update_assignment.rs | 133 +
bindings/python/src/variant.rs | 55 +
bindings/python/src/write.rs | 1393 +++++++--
bindings/python/tests/test_blob_consumer.py | 124 +
bindings/python/tests/test_datafusion.py | 30 +
bindings/python/tests/test_read.py | 700 ++++-
bindings/python/tests/test_resolved_table.py | 122 +
bindings/python/tests/test_runtime_fork.py | 5 +
bindings/python/tests/test_table.py | 34 +
bindings/python/tests/test_table_commit.py | 457 +++
bindings/python/tests/test_update.py | 431 +++
bindings/python/tests/test_variant.py | 65 +
bindings/python/tests/test_write.py | 207 +-
crates/integration_tests/DEPENDENCIES.rust.tsv | 808 +++---
crates/integration_tests/tests/read_tables.rs | 2 +-
crates/integrations/datafusion/Cargo.toml | 2 +-
crates/integrations/datafusion/README.md | 14 +
crates/integrations/datafusion/src/error.rs | 7 +-
.../datafusion/src/format_partition_ddl.rs | 15 +-
.../datafusion/src/lateral_vector_search.rs | 4 +
crates/integrations/datafusion/src/lib.rs | 2 +
crates/integrations/datafusion/src/memory.rs | 66 +
crates/integrations/datafusion/src/merge_into.rs | 193 +-
.../datafusion/src/partition_count_pushdown.rs | 583 ++++
.../datafusion/src/physical_plan/audit_log.rs | 4 +-
.../datafusion/src/physical_plan/scan.rs | 161 +-
.../datafusion/src/physical_plan/sink.rs | 4 +-
crates/integrations/datafusion/src/procedures.rs | 673 ++++-
crates/integrations/datafusion/src/runtime.rs | 80 +-
crates/integrations/datafusion/src/sql_context.rs | 148 +-
.../src/system_tables/aggregation_fields.rs | 133 +
.../datafusion/src/system_tables/buckets.rs | 258 ++
.../datafusion/src/system_tables/mod.rs | 10 +-
.../src/system_tables/physical_files_size.rs | 12 +-
.../src/system_tables/referenced_files_size.rs | 7 +-
crates/integrations/datafusion/src/table/mod.rs | 7 +-
crates/integrations/datafusion/src/update.rs | 6 +-
.../datafusion/src/variant_functions.rs | 66 +-
.../datafusion/src/variant_pushdown.rs | 8 +-
crates/integrations/datafusion/tests/blob_tests.rs | 88 +-
.../tests/catalog_callbacks_on_runtime_workers.rs | 112 +
.../datafusion/tests/dynamic_bucket_tables.rs | 56 +-
.../tests/format_table_partition_filters.rs | 51 +
.../tests/global_index_schema_evolution.rs | 107 +
.../tests/partition_conjunct_pushdown.rs | 136 +
.../datafusion/tests/partition_count_pushdown.rs | 956 ++++++
crates/integrations/datafusion/tests/pk_tables.rs | 48 +-
crates/integrations/datafusion/tests/procedures.rs | 309 +-
.../datafusion/tests/rest_format_partition_sql.rs | 14 +-
.../datafusion/tests/rest_format_table_dml.rs | 205 +-
.../datafusion/tests/rest_format_table_scan.rs | 131 +-
.../datafusion/tests/rest_management_procedures.rs | 847 ++++++
.../datafusion/tests/sql_context_tests.rs | 74 +-
.../integrations/datafusion/tests/system_tables.rs | 214 ++
.../datafusion/tests/time_travel_schema_tests.rs | 156 +
.../datafusion/tests/variant_pushdown.rs | 42 +-
crates/paimon-rest-server/DEPENDENCIES.rust.tsv | 820 +++---
crates/paimon-rest-server/src/lib.rs | 24 +-
crates/paimon-rest-server/tests/e2e.rs | 77 +
crates/paimon/Cargo.toml | 23 +-
crates/paimon/DEPENDENCIES.rust.tsv | 1279 ++++----
.../tpcds/src/main.rs => crates/paimon/build.rs | 16 +-
crates/paimon/src/api/api_request.rs | 10 +
crates/paimon/src/api/api_response.rs | 5 +
crates/paimon/src/api/auth/dlf_provider.rs | 395 ++-
crates/paimon/src/api/auth/factory.rs | 14 +-
crates/paimon/src/api/auth/mod.rs | 4 +-
crates/paimon/src/api/mod.rs | 1 +
crates/paimon/src/api/rest_api.rs | 60 +-
crates/paimon/src/api/rest_client.rs | 165 +-
crates/paimon/src/api/user_agent.rs | 90 +
crates/paimon/src/arrow/filtering.rs | 6 +-
crates/paimon/src/arrow/format/avro.rs | 92 +-
crates/paimon/src/arrow/format/avro_write.rs | 1034 +++++++
crates/paimon/src/arrow/format/blob.rs | 1747 ++++++++---
crates/paimon/src/arrow/format/blob/writer.rs | 751 +++++
.../src/arrow/format/blob/writer_consumer_tests.rs | 286 ++
crates/paimon/src/arrow/format/delta_varint.rs | 97 +
crates/paimon/src/arrow/format/metadata_cache.rs | 440 +++
crates/paimon/src/arrow/format/mod.rs | 595 +++-
crates/paimon/src/arrow/format/mosaic.rs | 11 +-
crates/paimon/src/arrow/format/mosaic_write.rs | 1215 ++++++++
crates/paimon/src/arrow/format/orc.rs | 527 +++-
crates/paimon/src/arrow/format/parquet.rs | 3057 ++++++++++++++++++--
crates/paimon/src/arrow/format/row.rs | 456 ++-
crates/paimon/src/arrow/format/shredding.rs | 597 ++--
crates/paimon/src/arrow/format/text.rs | 1522 ++++++++++
.../paimon/src/arrow/format/variant_projection.rs | 297 ++
crates/paimon/src/arrow/format/video.rs | 514 ++++
crates/paimon/src/arrow/format/vortex.rs | 137 +
crates/paimon/src/arrow/mod.rs | 97 +-
crates/paimon/src/arrow/nested_evolution.rs | 560 +++-
crates/paimon/src/arrow/partition.rs | 19 +-
crates/paimon/src/arrow/read_budget.rs | 199 +-
crates/paimon/src/arrow/residual.rs | 428 ++-
crates/paimon/src/arrow/shredding/map.rs | 625 +++-
crates/paimon/src/arrow/shredding/mod.rs | 35 +-
crates/paimon/src/arrow/shredding/variant.rs | 813 +++++-
crates/paimon/src/arrow/variant.rs | 427 +++
crates/paimon/src/btree/block.rs | 50 +-
crates/paimon/src/btree/composite_key.rs | 268 ++
crates/paimon/src/btree/composite_predicate.rs | 457 +++
crates/paimon/src/btree/composite_tests.rs | 664 +++++
crates/paimon/src/btree/data_block_cache.rs | 104 +
crates/paimon/src/btree/footer.rs | 7 +
crates/paimon/src/btree/key_serde.rs | 289 +-
crates/paimon/src/btree/meta.rs | 104 +-
crates/paimon/src/btree/mod.rs | 17 +-
crates/paimon/src/btree/posting_list.rs | 271 ++
crates/paimon/src/btree/query.rs | 141 +-
crates/paimon/src/btree/reader.rs | 275 +-
crates/paimon/src/btree/tests.rs | 931 +++++-
crates/paimon/src/btree/var_len.rs | 56 +
crates/paimon/src/btree/writer.rs | 49 +-
crates/paimon/src/catalog/filesystem.rs | 282 +-
crates/paimon/src/catalog/mod.rs | 42 +
crates/paimon/src/catalog/rest/rest_catalog.rs | 116 +-
.../paimon/src/catalog/rest/rest_token_file_io.rs | 28 +-
crates/paimon/src/common/options.rs | 20 +-
crates/paimon/src/error.rs | 177 +-
crates/paimon/src/file_index/bsi.rs | 519 ++++
crates/paimon/src/file_index/bsi/tests.rs | 498 ++++
.../paimon/src/file_index/file_indexer_factory.rs | 47 +-
crates/paimon/src/file_index/mod.rs | 2 +
crates/paimon/src/file_index/range_bitmap.rs | 60 +-
.../paimon/src/file_index/range_bitmap/writer.rs | 255 ++
.../writer/fixtures/GenerateRangeBitmap.java | 66 +
.../range_bitmap/writer/fixtures/README.md | 39 +
.../src/file_index/range_bitmap/writer/tests.rs | 389 +++
crates/paimon/src/io/cache/file_type.rs | 160 +-
crates/paimon/src/io/cache/mod.rs | 2 +
crates/paimon/src/io/file_io.rs | 1200 +++++++-
crates/paimon/src/io/file_io/provider_tests.rs | 33 +
crates/paimon/src/io/jindo_ffi.cc | 93 +
crates/paimon/src/io/mod.rs | 26 +-
crates/paimon/src/io/storage.rs | 163 +-
crates/paimon/src/io/storage_azdls.rs | 22 +-
crates/paimon/src/io/storage_cos.rs | 11 +-
crates/paimon/src/io/storage_gcs.rs | 11 +-
crates/paimon/src/io/storage_hdfs.rs | 42 +-
crates/paimon/src/io/storage_jindo.rs | 1795 ++++++++++++
crates/paimon/src/io/storage_obs.rs | 11 +-
crates/paimon/src/io/storage_oss.rs | 64 +-
crates/paimon/src/io/storage_oss_cpp.rs | 780 +++++
crates/paimon/src/io/storage_oss_cpp_test.rs | 506 ++++
crates/paimon/src/io/storage_s3.rs | 15 +-
crates/paimon/src/io/uri_reader.rs | 406 +++
crates/paimon/src/io/user_agent.rs | 291 ++
crates/paimon/src/lib.rs | 4 +-
crates/paimon/src/lumina/mod.rs | 84 +-
crates/paimon/src/lumina/reader.rs | 173 +-
crates/paimon/src/predicate_stats.rs | 65 +-
crates/paimon/src/resource/memory.rs | 150 +
crates/paimon/src/resource/mod.rs | 115 +
crates/paimon/src/resource/tests.rs | 163 ++
crates/paimon/src/spec/aggregation.rs | 157 +-
crates/paimon/src/spec/avro/decode_helpers.rs | 2 +-
.../src/spec/avro/index_manifest_entry_decode.rs | 117 +-
.../paimon/src/spec/avro/manifest_entry_decode.rs | 354 ++-
crates/paimon/src/spec/avro/mod.rs | 61 +
crates/paimon/src/spec/avro/ocf.rs | 49 +
crates/paimon/src/spec/binary_row.rs | 44 +-
crates/paimon/src/spec/blob_consumer.rs | 41 +
crates/paimon/src/spec/core_options.rs | 795 ++++-
crates/paimon/src/spec/index_file_meta.rs | 217 ++
crates/paimon/src/spec/index_manifest.rs | 73 +-
crates/paimon/src/spec/manifest.rs | 3 +-
crates/paimon/src/spec/manifest_file_meta.rs | 15 +-
crates/paimon/src/spec/manifest_sidecar.rs | 149 +-
crates/paimon/src/spec/map_selected_keys.rs | 120 +
crates/paimon/src/spec/map_shredding.rs | 438 +++
crates/paimon/src/spec/mod.rs | 18 +-
crates/paimon/src/spec/murmur_hash.rs | 18 +
crates/paimon/src/spec/objects_file.rs | 77 +-
crates/paimon/src/spec/partial_update.rs | 206 +-
crates/paimon/src/spec/partition_utils.rs | 455 ++-
crates/paimon/src/spec/predicate.rs | 271 +-
crates/paimon/src/spec/read_type.rs | 238 ++
crates/paimon/src/spec/schema.rs | 1070 ++++++-
crates/paimon/src/spec/snapshot.rs | 8 +
crates/paimon/src/spec/types.rs | 41 +-
crates/paimon/src/spec/variant_metadata.rs | 115 +-
crates/paimon/src/spec/video_frame_descriptor.rs | 215 ++
crates/paimon/src/table/aggregator/bitmap.rs | 139 +
crates/paimon/src/table/aggregator/bitmap64.rs | 520 ++++
crates/paimon/src/table/aggregator/collect.rs | 440 +++
crates/paimon/src/table/aggregator/listagg.rs | 144 +-
crates/paimon/src/table/aggregator/merge_map.rs | 567 ++++
crates/paimon/src/table/aggregator/mod.rs | 160 +-
crates/paimon/src/table/aggregator/nested.rs | 755 +++++
crates/paimon/src/table/aggregator/numeric.rs | 472 ++-
crates/paimon/src/table/aggregator/sketch.rs | 598 ++++
crates/paimon/src/table/aggregator/value.rs | 96 +-
crates/paimon/src/table/audit_log_table/read.rs | 8 +-
.../paimon/src/table/bitmap_global_index_format.rs | 14 +-
.../paimon/src/table/bitmap_global_index_reader.rs | 109 +-
.../paimon/src/table/bitmap_global_index_writer.rs | 52 +-
crates/paimon/src/table/blob_consumer_tests.rs | 247 ++
crates/paimon/src/table/blob_resolver.rs | 251 +-
crates/paimon/src/table/branch_manager.rs | 42 +-
crates/paimon/src/table/branch_write_tests.rs | 728 +++++
crates/paimon/src/table/bucket_assigner_cross.rs | 269 +-
crates/paimon/src/table/bucket_assigner_dynamic.rs | 770 ++++-
crates/paimon/src/table/chunk_shuffle.rs | 1035 +++++++
crates/paimon/src/table/commit_message.rs | 320 +-
crates/paimon/src/table/cow_writer.rs | 4 +-
crates/paimon/src/table/data_evolution_fields.rs | 368 +++
crates/paimon/src/table/data_evolution_nested.rs | 786 +++++
crates/paimon/src/table/data_evolution_reader.rs | 1308 +++++++--
.../table/data_evolution_reader/blob_fallback.rs | 62 +-
crates/paimon/src/table/data_evolution_writer.rs | 1542 ++++++++--
crates/paimon/src/table/data_file_index_writer.rs | 30 +-
.../src/table/data_file_index_writer/tests.rs | 1007 ++++++-
crates/paimon/src/table/data_file_path_factory.rs | 101 +
crates/paimon/src/table/data_file_reader.rs | 1311 +++++++--
crates/paimon/src/table/data_file_writer.rs | 798 ++++-
.../src/table/dedicated_format_file_writer.rs | 431 ++-
crates/paimon/src/table/external_path.rs | 351 +++
crates/paimon/src/table/format_partition.rs | 129 +-
.../paimon/src/table/format_partition_location.rs | 544 ++++
crates/paimon/src/table/format_table_commit.rs | 633 ++++
crates/paimon/src/table/format_table_defaults.rs | 1404 +++++++++
crates/paimon/src/table/format_table_read.rs | 29 +-
crates/paimon/src/table/format_table_scan.rs | 67 +-
.../paimon/src/table/format_table_write_tests.rs | 1842 ++++++++++++
crates/paimon/src/table/format_table_writer.rs | 592 ++++
crates/paimon/src/table/format_write_builder.rs | 33 +-
.../src/table/full_text_index_build_builder.rs | 540 ++++
.../table/full_text_index_build_builder/tests.rs | 646 +++++
.../paimon/src/table/global_index_build_common.rs | 110 +-
.../src/table/global_index_build_common/vector.rs | 4 +-
.../paimon/src/table/global_index_drop_builder.rs | 109 +-
crates/paimon/src/table/global_index_scanner.rs | 70 +-
.../src/table/global_index_scanner/all_match.rs | 115 +
.../src/table/global_index_scanner/composite.rs | 227 ++
.../paimon/src/table/global_index_scanner/entry.rs | 9 +-
.../src/table/global_index_scanner/evaluator.rs | 275 +-
.../src/table/global_index_scanner/reader.rs | 88 +-
.../src/table/global_index_scanner/row_ranges.rs | 39 +-
.../paimon/src/table/global_index_scanner/tests.rs | 1783 +++++++++++-
crates/paimon/src/table/global_index_types.rs | 25 +-
.../paimon/src/table/goldens/GenerateHllDense.java | 51 +
.../src/table/goldens/GenerateRoaring64Dense.java | 22 +-
.../src/table/goldens/hll_java_compact_aux.bin | Bin 0 -> 32856 bytes
.../paimon/src/table/goldens/hll_java_dense6_a.bin | Bin 0 -> 3113 bytes
.../paimon/src/table/goldens/hll_java_dense6_b.bin | Bin 0 -> 3113 bytes
.../paimon/src/table/goldens/hll_java_dense8_a.bin | Bin 0 -> 4136 bytes
.../paimon/src/table/goldens/hll_java_dense8_b.bin | Bin 0 -> 4136 bytes
.../paimon/src/table/goldens/hll_java_dense_a.bin | Bin 0 -> 2088 bytes
.../paimon/src/table/goldens/hll_java_dense_b.bin | Bin 0 -> 2092 bytes
.../src/table/goldens/hll_java_updatable_aux.bin | Bin 0 -> 33832 bytes
.../src/table/goldens/roaring64_java_dense.bin | Bin 0 -> 542 bytes
crates/paimon/src/table/incremental_scan.rs | 59 +-
crates/paimon/src/table/index_file_path.rs | 250 +-
crates/paimon/src/table/inline_blob.rs | 68 +
crates/paimon/src/table/kv_file_reader.rs | 802 ++++-
crates/paimon/src/table/kv_file_writer.rs | 1655 +++++++++--
.../table/lumina_index_build_builder/extraction.rs | 18 +-
.../src/table/lumina_index_build_builder/tests.rs | 52 +
.../src/table/lumina_index_build_builder/writer.rs | 37 +-
crates/paimon/src/table/managed_blob_reader.rs | 962 ++++++
crates/paimon/src/table/managed_blob_reference.rs | 287 ++
.../paimon/src/table/managed_blob_table_tests.rs | 1549 ++++++++++
crates/paimon/src/table/managed_blob_writer.rs | 475 +++
.../paimon/src/table/map_shredding_write_tests.rs | 297 ++
crates/paimon/src/table/mod.rs | 288 +-
.../paimon/src/table/mosaic_table_write_tests.rs | 962 ++++++
crates/paimon/src/table/partition_filter.rs | 414 ++-
crates/paimon/src/table/partition_row_count.rs | 1573 ++++++++++
crates/paimon/src/table/pk_vector_scan.rs | 61 +-
crates/paimon/src/table/postpone_bucket_plan.rs | 52 +-
crates/paimon/src/table/postpone_file_writer.rs | 336 ++-
.../src/table/postpone_fixed_bucket_router.rs | 8 +
.../src/table/postpone_fixed_bucket_write.rs | 353 ++-
.../table/postpone_fixed_bucket_write_builder.rs | 47 +-
crates/paimon/src/table/postpone_retract.rs | 141 +
crates/paimon/src/table/query_auth.rs | 561 ++++
crates/paimon/src/table/read_builder.rs | 275 +-
.../paimon/src/table/read_limit.rs | 31 +-
crates/paimon/src/table/referenced_files.rs | 170 +-
crates/paimon/src/table/rest_env.rs | 244 +-
crates/paimon/src/table/row_position_selection.rs | 394 ++-
crates/paimon/src/table/row_sidecar.rs | 248 ++
crates/paimon/src/table/snapshot_commit.rs | 42 +-
crates/paimon/src/table/snapshot_manager.rs | 11 +-
crates/paimon/src/table/sort_merge.rs | 829 +++++-
.../src/table/sorted_global_index_build_builder.rs | 40 +-
.../extraction.rs | 120 +-
.../sorted_global_index_build_builder/planning.rs | 4 +-
.../sorted_global_index_build_builder/tests.rs | 1132 +++++++-
.../validation.rs | 1 +
.../sorted_global_index_build_builder/writer.rs | 44 +-
.../src/table/sorted_global_index_options.rs | 32 +
crates/paimon/src/table/source.rs | 102 +-
crates/paimon/src/table/stats_filter.rs | 9 +-
crates/paimon/src/table/table_commit.rs | 1799 ++++++++----
.../paimon/src/table/table_commit/abort_tests.rs | 121 +
.../paimon/src/table/table_commit/parity_tests.rs | 663 +++++
.../src/table/table_commit/recovery_tests.rs | 629 ++++
.../paimon/src/table/table_commit/row_tracking.rs | 141 +
.../src/table/table_commit/row_tracking_tests.rs | 565 ++++
crates/paimon/src/table/table_read.rs | 1151 +++++++-
crates/paimon/src/table/table_scan.rs | 1060 ++++++-
crates/paimon/src/table/table_update.rs | 225 +-
crates/paimon/src/table/table_update_by_row_id.rs | 125 +
crates/paimon/src/table/table_update_predicate.rs | 222 ++
crates/paimon/src/table/table_upsert.rs | 430 +++
crates/paimon/src/table/table_write.rs | 1398 ++++++---
crates/paimon/src/table/table_write_precomputed.rs | 102 +
crates/paimon/src/table/tag_manager.rs | 297 +-
crates/paimon/src/table/time_travel.rs | 125 +
crates/paimon/src/table/update_assignment.rs | 160 +
crates/paimon/src/table/update_input.rs | 2595 +++++++++++++++++
crates/paimon/src/table/upsert_key_matcher.rs | 309 ++
.../paimon/src/table/vindex_index_build_builder.rs | 3 +
.../table/vindex_index_build_builder/extraction.rs | 108 +-
.../table/vindex_index_build_builder/pipeline.rs | 1662 +++++++++++
.../src/table/vindex_index_build_builder/tests.rs | 379 ++-
.../src/table/vindex_index_build_builder/timing.rs | 8 +-
.../src/table/vindex_index_build_builder/writer.rs | 149 +-
crates/paimon/src/table/write_batch_normalize.rs | 508 ++++
crates/paimon/src/table/write_builder.rs | 161 +-
crates/paimon/src/variant.rs | 431 ++-
crates/paimon/src/variant/inference.rs | 632 ++++
crates/paimon/src/variant/numeric.rs | 367 +++
crates/paimon/src/vindex/mod.rs | 74 +
crates/paimon/testdata/btree/GenerateBTreeV2.java | 68 +
.../testdata/btree/GenerateCompositeBTree.java | 96 +
crates/paimon/testdata/btree/README.md | 79 +
.../testdata/btree/btree_composite_java_scalar.key | Bin 0 -> 52 bytes
.../testdata/btree/btree_composite_v1_java_lz4.bin | Bin 0 -> 873 bytes
.../btree/btree_composite_v1_java_lz4.meta | Bin 0 -> 23 bytes
.../btree/btree_composite_v1_java_none.bin | Bin 0 -> 1144 bytes
.../btree/btree_composite_v1_java_none.meta | Bin 0 -> 23 bytes
.../testdata/btree/btree_composite_v2_java_lz4.bin | Bin 0 -> 799 bytes
.../btree/btree_composite_v2_java_lz4.meta | Bin 0 -> 23 bytes
.../btree/btree_composite_v2_java_none.bin | Bin 0 -> 1249 bytes
.../btree/btree_composite_v2_java_none.meta | Bin 0 -> 23 bytes
crates/paimon/testdata/btree/btree_v2_java_lz4.bin | Bin 0 -> 419 bytes
.../paimon/testdata/btree/btree_v2_java_none.bin | Bin 0 -> 8562 bytes
crates/paimon/testdata/file_index/README.md | 41 +
...-c6f0f8ca-a74f-42ce-af35-fc4f4ef2d003-0.parquet | Bin 0 -> 564 bytes
...anifest-982c31e0-7caa-4462-a270-3b130005e913-0} | Bin 2187 -> 2273 bytes
...st-list-9d29ca76-96e0-424d-8ccf-28d75ee5b70e-0} | Bin 1110 -> 1158 bytes
...st-list-9d29ca76-96e0-424d-8ccf-28d75ee5b70e-1} | Bin 1110 -> 1263 bytes
.../default.db/bitmap_embedded/schema/schema-0 | 25 +
.../default.db/bitmap_embedded}/snapshot/EARLIEST | 0
.../default.db/bitmap_embedded}/snapshot/LATEST | 0
.../default.db/bitmap_embedded/snapshot/snapshot-1 | 18 +
...-d9e65532-1aa9-4878-bb30-24862120212c-0.parquet | Bin 0 -> 564 bytes
...532-1aa9-4878-bb30-24862120212c-0.parquet.index | Bin 0 -> 130 bytes
...anifest-392c52de-fdae-4262-81fb-4049e95c37de-0} | Bin 2187 -> 2211 bytes
...st-list-973ebadd-00a4-4672-a3b1-81e48cc9ec83-0} | Bin 1110 -> 1158 bytes
...st-list-973ebadd-00a4-4672-a3b1-81e48cc9ec83-1} | Bin 1110 -> 1263 bytes
.../default.db/bitmap_sidecar/schema/schema-0 | 25 +
.../default.db/bitmap_sidecar}/snapshot/EARLIEST | 0
.../default.db/bitmap_sidecar}/snapshot/LATEST | 0
.../default.db/bitmap_sidecar/snapshot/snapshot-1 | 18 +
...-9ce3b83d-6c45-4dc1-a34e-819b5fb2e331-0.parquet | Bin 0 -> 564 bytes
...anifest-8dd937f3-747e-4f87-a296-489d11ef050c-0} | Bin 2187 -> 2255 bytes
...st-list-ac933951-c2f2-4af6-b3d4-d20e80e79189-0} | Bin 1110 -> 1158 bytes
...st-list-ac933951-c2f2-4af6-b3d4-d20e80e79189-1} | Bin 1110 -> 1263 bytes
.../bloom_filter_embedded/schema/schema-0 | 27 +
.../bloom_filter_embedded}/snapshot/EARLIEST | 0
.../bloom_filter_embedded}/snapshot/LATEST | 0
.../bloom_filter_embedded/snapshot/snapshot-1 | 18 +
...-2e01926c-6cd0-4dd5-b576-80ddf6e55632-0.parquet | Bin 0 -> 564 bytes
...26c-6cd0-4dd5-b576-80ddf6e55632-0.parquet.index | Bin 0 -> 78 bytes
...anifest-214eaa6e-a8e8-4903-a41a-3fa04732e5e4-0} | Bin 2187 -> 2212 bytes
...st-list-baa54195-a358-4f0b-9ca9-51503661a534-0} | Bin 1110 -> 1158 bytes
...st-list-baa54195-a358-4f0b-9ca9-51503661a534-1} | Bin 1110 -> 1260 bytes
.../bloom_filter_sidecar/schema/schema-0 | 27 +
.../bloom_filter_sidecar}/snapshot/EARLIEST | 0
.../bloom_filter_sidecar}/snapshot/LATEST | 0
.../bloom_filter_sidecar/snapshot/snapshot-1 | 18 +
...-791a54a7-4d19-43c3-8752-31cd6073433a-0.parquet | Bin 0 -> 565 bytes
...anifest-155640a3-e50d-4be7-ba72-de0afe2a8fec-0} | Bin 2187 -> 2302 bytes
...st-list-ecab87c6-de6d-44d0-a6ae-3705321d971d-0} | Bin 1110 -> 1158 bytes
...st-list-ecab87c6-de6d-44d0-a6ae-3705321d971d-1} | Bin 1110 -> 1263 bytes
.../range_bitmap_embedded/schema/schema-0 | 25 +
.../range_bitmap_embedded}/snapshot/EARLIEST | 0
.../range_bitmap_embedded}/snapshot/LATEST | 0
.../range_bitmap_embedded/snapshot/snapshot-1 | 18 +
...-9db69c4d-d10c-4f94-aed9-c16cd7fbe1e1-0.parquet | Bin 0 -> 565 bytes
...c4d-d10c-4f94-aed9-c16cd7fbe1e1-0.parquet.index | Bin 0 -> 191 bytes
...anifest-e1c11d52-6e22-4d10-b62a-833b60216bf2-0} | Bin 2187 -> 2211 bytes
...st-list-107ca82a-e4a8-4a88-8439-b15d770f0474-0} | Bin 1110 -> 1158 bytes
...st-list-107ca82a-e4a8-4a88-8439-b15d770f0474-1} | Bin 1110 -> 1263 bytes
.../range_bitmap_sidecar/schema/schema-0 | 25 +
.../range_bitmap_sidecar}/snapshot/EARLIEST | 0
.../range_bitmap_sidecar}/snapshot/LATEST | 0
.../range_bitmap_sidecar/snapshot/snapshot-1 | 18 +
.../JavaFileIndexTableFixtureGeneratorTest.java | 114 +
crates/paimon/testdata/file_index/regenerate.sh | 64 +
.../testdata/video/video-frame-descriptor-v1.hex | 8 +-
.../paimon/testdata/video/video-v1.hex | 8 +-
crates/paimon/tests/audit_log_table_test.rs | 31 +
.../tests/binary_partition_native_write_test.rs | 128 +
crates/paimon/tests/blob_collection_write_test.rs | 270 ++
crates/paimon/tests/blob_http_error_test.rs | 133 +
crates/paimon/tests/cross_partition_write_test.rs | 323 +++
.../tests/data_evolution_row_sidecar_test.rs | 797 +++++
.../tests/data_evolution_system_columns_test.rs | 599 ++++
crates/paimon/tests/data_file_directory_test.rs | 470 +++
crates/paimon/tests/dedicated_blob_write_test.rs | 556 ++++
crates/paimon/tests/dynamic_bucket_write_test.rs | 287 ++
crates/paimon/tests/external_data_write_test.rs | 383 +++
.../paimon/tests/file_index_java_fixture_test.rs | 312 ++
crates/paimon/tests/incremental_batch_scan_test.rs | 218 ++
.../paimon/tests/incremental_diff_extended_test.rs | 804 +++++
crates/paimon/tests/mock_server.rs | 190 +-
.../tests/nested_native_write_compat_test.rs | 239 ++
.../paimon/tests/parquet_write_page_index_test.rs | 122 +
...pk_partial_update_sequence_group_parity_test.rs | 357 +++
.../paimon/tests/pk_user_sequence_parity_test.rs | 426 +++
crates/paimon/tests/pk_value_stats_test.rs | 123 +
crates/paimon/tests/pk_write_reuse_seq_test.rs | 82 +
crates/paimon/tests/postpone_native_write_test.rs | 297 ++
.../paimon/tests/precomputed_bucket_write_test.rs | 731 +++++
crates/paimon/tests/reader_resources_test.rs | 332 +++
crates/paimon/tests/rest_api_test.rs | 33 +
crates/paimon/tests/rest_catalog_test.rs | 738 ++++-
crates/paimon/tests/rowkind_field_test.rs | 796 +++++
crates/paimon/tests/scan_planning_parity_test.rs | 64 +-
crates/paimon/tests/table_partial_write_test.rs | 344 +++
crates/paimon/tests/table_update_nested_test.rs | 439 +++
crates/paimon/tests/table_update_paths_test.rs | 563 ++++
crates/paimon/tests/table_update_test.rs | 2141 ++++++++++++++
crates/paimon/tests/target_file_row_num_test.rs | 184 ++
crates/paimon/tests/writer_resources_test.rs | 359 +++
docs/src/c-binding.md | 76 +
docs/src/getting-started.md | 97 +-
docs/src/go-binding.md | 25 +
docs/src/python-binding.md | 63 +-
docs/src/releases.md | 5 +
docs/src/sql.md | 227 +-
integrations/oss-cpp/CMakeLists.txt | 25 +
integrations/oss-cpp/README.md | 85 +
integrations/oss-cpp/bridge.cc | 190 ++
integrations/oss-cpp/bridge.h | 65 +
scripts/validate_asf_yaml.py | 1 +
482 files changed, 116275 insertions(+), 9166 deletions(-)
create mode 100644 bindings/c/src/resource.rs
create mode 100644 bindings/go/resource.go
create mode 100644 bindings/go/tests/resource_test.go
copy bindings/python/{Makefile => python/pypaimon_rust/data.pyi} (75%)
create mode 100644 bindings/python/src/oss_cpp.rs
create mode 100644 bindings/python/src/update_assignment.rs
create mode 100644 bindings/python/src/variant.rs
create mode 100644 bindings/python/tests/test_blob_consumer.py
create mode 100644 bindings/python/tests/test_table_commit.py
create mode 100644 bindings/python/tests/test_update.py
create mode 100644 bindings/python/tests/test_variant.py
create mode 100644 crates/integrations/datafusion/src/memory.rs
create mode 100644
crates/integrations/datafusion/src/partition_count_pushdown.rs
create mode 100644
crates/integrations/datafusion/src/system_tables/aggregation_fields.rs
create mode 100644 crates/integrations/datafusion/src/system_tables/buckets.rs
create mode 100644
crates/integrations/datafusion/tests/catalog_callbacks_on_runtime_workers.rs
create mode 100644
crates/integrations/datafusion/tests/global_index_schema_evolution.rs
create mode 100644
crates/integrations/datafusion/tests/partition_conjunct_pushdown.rs
create mode 100644
crates/integrations/datafusion/tests/partition_count_pushdown.rs
create mode 100644
crates/integrations/datafusion/tests/rest_management_procedures.rs
copy benchmarks/tpcds/src/main.rs => crates/paimon/build.rs (70%)
create mode 100644 crates/paimon/src/api/user_agent.rs
create mode 100644 crates/paimon/src/arrow/format/avro_write.rs
create mode 100644 crates/paimon/src/arrow/format/blob/writer.rs
create mode 100644 crates/paimon/src/arrow/format/blob/writer_consumer_tests.rs
create mode 100644 crates/paimon/src/arrow/format/delta_varint.rs
create mode 100644 crates/paimon/src/arrow/format/metadata_cache.rs
create mode 100644 crates/paimon/src/arrow/format/mosaic_write.rs
create mode 100644 crates/paimon/src/arrow/format/text.rs
create mode 100644 crates/paimon/src/arrow/format/variant_projection.rs
create mode 100644 crates/paimon/src/arrow/format/video.rs
create mode 100644 crates/paimon/src/arrow/variant.rs
create mode 100644 crates/paimon/src/btree/composite_key.rs
create mode 100644 crates/paimon/src/btree/composite_predicate.rs
create mode 100644 crates/paimon/src/btree/composite_tests.rs
create mode 100644 crates/paimon/src/btree/data_block_cache.rs
create mode 100644 crates/paimon/src/btree/posting_list.rs
create mode 100644 crates/paimon/src/file_index/bsi.rs
create mode 100644 crates/paimon/src/file_index/bsi/tests.rs
create mode 100644 crates/paimon/src/file_index/range_bitmap/writer.rs
create mode 100644
crates/paimon/src/file_index/range_bitmap/writer/fixtures/GenerateRangeBitmap.java
create mode 100644
crates/paimon/src/file_index/range_bitmap/writer/fixtures/README.md
create mode 100644 crates/paimon/src/file_index/range_bitmap/writer/tests.rs
create mode 100644 crates/paimon/src/io/jindo_ffi.cc
create mode 100644 crates/paimon/src/io/storage_jindo.rs
create mode 100644 crates/paimon/src/io/storage_oss_cpp.rs
create mode 100644 crates/paimon/src/io/storage_oss_cpp_test.rs
create mode 100644 crates/paimon/src/io/uri_reader.rs
create mode 100644 crates/paimon/src/io/user_agent.rs
create mode 100644 crates/paimon/src/resource/memory.rs
create mode 100644 crates/paimon/src/resource/mod.rs
create mode 100644 crates/paimon/src/resource/tests.rs
create mode 100644 crates/paimon/src/spec/blob_consumer.rs
create mode 100644 crates/paimon/src/spec/map_selected_keys.rs
create mode 100644 crates/paimon/src/spec/map_shredding.rs
create mode 100644 crates/paimon/src/spec/read_type.rs
create mode 100644 crates/paimon/src/spec/video_frame_descriptor.rs
create mode 100644 crates/paimon/src/table/aggregator/bitmap.rs
create mode 100644 crates/paimon/src/table/aggregator/bitmap64.rs
create mode 100644 crates/paimon/src/table/aggregator/collect.rs
create mode 100644 crates/paimon/src/table/aggregator/merge_map.rs
create mode 100644 crates/paimon/src/table/aggregator/nested.rs
create mode 100644 crates/paimon/src/table/aggregator/sketch.rs
create mode 100644 crates/paimon/src/table/blob_consumer_tests.rs
create mode 100644 crates/paimon/src/table/branch_write_tests.rs
create mode 100644 crates/paimon/src/table/chunk_shuffle.rs
create mode 100644 crates/paimon/src/table/data_evolution_fields.rs
create mode 100644 crates/paimon/src/table/data_evolution_nested.rs
create mode 100644 crates/paimon/src/table/data_file_path_factory.rs
create mode 100644 crates/paimon/src/table/external_path.rs
create mode 100644 crates/paimon/src/table/format_partition_location.rs
create mode 100644 crates/paimon/src/table/format_table_commit.rs
create mode 100644 crates/paimon/src/table/format_table_defaults.rs
create mode 100644 crates/paimon/src/table/format_table_write_tests.rs
create mode 100644 crates/paimon/src/table/format_table_writer.rs
create mode 100644 crates/paimon/src/table/full_text_index_build_builder.rs
create mode 100644
crates/paimon/src/table/full_text_index_build_builder/tests.rs
create mode 100644 crates/paimon/src/table/global_index_scanner/all_match.rs
create mode 100644 crates/paimon/src/table/global_index_scanner/composite.rs
create mode 100644 crates/paimon/src/table/goldens/GenerateHllDense.java
copy copyright.txt =>
crates/paimon/src/table/goldens/GenerateRoaring64Dense.java (53%)
create mode 100644 crates/paimon/src/table/goldens/hll_java_compact_aux.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense6_a.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense6_b.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense8_a.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense8_b.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense_a.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_dense_b.bin
create mode 100644 crates/paimon/src/table/goldens/hll_java_updatable_aux.bin
create mode 100644 crates/paimon/src/table/goldens/roaring64_java_dense.bin
create mode 100644 crates/paimon/src/table/inline_blob.rs
create mode 100644 crates/paimon/src/table/managed_blob_reader.rs
create mode 100644 crates/paimon/src/table/managed_blob_reference.rs
create mode 100644 crates/paimon/src/table/managed_blob_table_tests.rs
create mode 100644 crates/paimon/src/table/managed_blob_writer.rs
create mode 100644 crates/paimon/src/table/map_shredding_write_tests.rs
create mode 100644 crates/paimon/src/table/mosaic_table_write_tests.rs
create mode 100644 crates/paimon/src/table/partition_row_count.rs
create mode 100644 crates/paimon/src/table/postpone_retract.rs
create mode 100644 crates/paimon/src/table/query_auth.rs
copy bindings/python/src/tag.rs => crates/paimon/src/table/read_limit.rs (62%)
create mode 100644 crates/paimon/src/table/row_sidecar.rs
create mode 100644 crates/paimon/src/table/table_commit/abort_tests.rs
create mode 100644 crates/paimon/src/table/table_commit/parity_tests.rs
create mode 100644 crates/paimon/src/table/table_commit/recovery_tests.rs
create mode 100644 crates/paimon/src/table/table_commit/row_tracking.rs
create mode 100644 crates/paimon/src/table/table_commit/row_tracking_tests.rs
create mode 100644 crates/paimon/src/table/table_update_by_row_id.rs
create mode 100644 crates/paimon/src/table/table_update_predicate.rs
create mode 100644 crates/paimon/src/table/table_upsert.rs
create mode 100644 crates/paimon/src/table/table_write_precomputed.rs
create mode 100644 crates/paimon/src/table/update_assignment.rs
create mode 100644 crates/paimon/src/table/update_input.rs
create mode 100644 crates/paimon/src/table/upsert_key_matcher.rs
create mode 100644
crates/paimon/src/table/vindex_index_build_builder/pipeline.rs
create mode 100644 crates/paimon/src/table/write_batch_normalize.rs
create mode 100644 crates/paimon/src/variant/inference.rs
create mode 100644 crates/paimon/src/variant/numeric.rs
create mode 100644 crates/paimon/testdata/btree/GenerateBTreeV2.java
create mode 100644 crates/paimon/testdata/btree/GenerateCompositeBTree.java
create mode 100644 crates/paimon/testdata/btree/README.md
create mode 100644 crates/paimon/testdata/btree/btree_composite_java_scalar.key
create mode 100644 crates/paimon/testdata/btree/btree_composite_v1_java_lz4.bin
create mode 100644
crates/paimon/testdata/btree/btree_composite_v1_java_lz4.meta
create mode 100644
crates/paimon/testdata/btree/btree_composite_v1_java_none.bin
create mode 100644
crates/paimon/testdata/btree/btree_composite_v1_java_none.meta
create mode 100644 crates/paimon/testdata/btree/btree_composite_v2_java_lz4.bin
create mode 100644
crates/paimon/testdata/btree/btree_composite_v2_java_lz4.meta
create mode 100644
crates/paimon/testdata/btree/btree_composite_v2_java_none.bin
create mode 100644
crates/paimon/testdata/btree/btree_composite_v2_java_none.meta
create mode 100644 crates/paimon/testdata/btree/btree_v2_java_lz4.bin
create mode 100644 crates/paimon/testdata/btree/btree_v2_java_none.bin
create mode 100644 crates/paimon/testdata/file_index/README.md
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_embedded/bucket-0/data-c6f0f8ca-a74f-42ce-af35-fc4f4ef2d003-0.parquet
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/bitmap_embedded/manifest/manifest-982c31e0-7caa-4462-a270-3b130005e913-0}
(87%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bitmap_embedded/manifest/manifest-list-9d29ca76-96e0-424d-8ccf-28d75ee5b70e-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bitmap_embedded/manifest/manifest-list-9d29ca76-96e0-424d-8ccf-28d75ee5b70e-1}
(70%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_embedded/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/bitmap_embedded}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/bitmap_embedded}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_embedded/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_sidecar/bucket-0/data-d9e65532-1aa9-4878-bb30-24862120212c-0.parquet
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_sidecar/bucket-0/data-d9e65532-1aa9-4878-bb30-24862120212c-0.parquet.index
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/bitmap_sidecar/manifest/manifest-392c52de-fdae-4262-81fb-4049e95c37de-0}
(89%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bitmap_sidecar/manifest/manifest-list-973ebadd-00a4-4672-a3b1-81e48cc9ec83-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bitmap_sidecar/manifest/manifest-list-973ebadd-00a4-4672-a3b1-81e48cc9ec83-1}
(70%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_sidecar/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/bitmap_sidecar}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/bitmap_sidecar}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bitmap_sidecar/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_embedded/bucket-0/data-9ce3b83d-6c45-4dc1-a34e-819b5fb2e331-0.parquet
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/bloom_filter_embedded/manifest/manifest-8dd937f3-747e-4f87-a296-489d11ef050c-0}
(87%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bloom_filter_embedded/manifest/manifest-list-ac933951-c2f2-4af6-b3d4-d20e80e79189-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bloom_filter_embedded/manifest/manifest-list-ac933951-c2f2-4af6-b3d4-d20e80e79189-1}
(70%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_embedded/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/bloom_filter_embedded}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/bloom_filter_embedded}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_embedded/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_sidecar/bucket-0/data-2e01926c-6cd0-4dd5-b576-80ddf6e55632-0.parquet
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_sidecar/bucket-0/data-2e01926c-6cd0-4dd5-b576-80ddf6e55632-0.parquet.index
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/bloom_filter_sidecar/manifest/manifest-214eaa6e-a8e8-4903-a41a-3fa04732e5e4-0}
(89%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bloom_filter_sidecar/manifest/manifest-list-baa54195-a358-4f0b-9ca9-51503661a534-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/bloom_filter_sidecar/manifest/manifest-list-baa54195-a358-4f0b-9ca9-51503661a534-1}
(71%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_sidecar/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/bloom_filter_sidecar}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/bloom_filter_sidecar}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/bloom_filter_sidecar/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_embedded/bucket-0/data-791a54a7-4d19-43c3-8752-31cd6073433a-0.parquet
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/range_bitmap_embedded/manifest/manifest-155640a3-e50d-4be7-ba72-de0afe2a8fec-0}
(86%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/range_bitmap_embedded/manifest/manifest-list-ecab87c6-de6d-44d0-a6ae-3705321d971d-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/range_bitmap_embedded/manifest/manifest-list-ecab87c6-de6d-44d0-a6ae-3705321d971d-1}
(70%)
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_embedded/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/range_bitmap_embedded}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/range_bitmap_embedded}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_embedded/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_sidecar/bucket-0/data-9db69c4d-d10c-4f94-aed9-c16cd7fbe1e1-0.parquet
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_sidecar/bucket-0/data-9db69c4d-d10c-4f94-aed9-c16cd7fbe1e1-0.parquet.index
copy
crates/paimon/testdata/{pkvector_split/table/manifest/manifest-003ae88e-3c2b-43af-8704-4f5a5e532a0b-0
=>
file_index/default.db/range_bitmap_sidecar/manifest/manifest-e1c11d52-6e22-4d10-b62a-833b60216bf2-0}
(89%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/range_bitmap_sidecar/manifest/manifest-list-107ca82a-e4a8-4a88-8439-b15d770f0474-0}
(77%)
copy
crates/paimon/testdata/{pkvector/pk_vector_ivf_flat/manifest/manifest-list-f5034e53-2d6f-40ae-9665-8778726154b4-2
=>
file_index/default.db/range_bitmap_sidecar/manifest/manifest-list-107ca82a-e4a8-4a88-8439-b15d770f0474-1}
(70%)
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_sidecar/schema/schema-0
copy crates/paimon/testdata/{pkvector/pk_vector_ivf_flat =>
file_index/default.db/range_bitmap_sidecar}/snapshot/EARLIEST (100%)
copy {bindings/go/tests/testdata/map_blob_table =>
crates/paimon/testdata/file_index/default.db/range_bitmap_sidecar}/snapshot/LATEST
(100%)
create mode 100644
crates/paimon/testdata/file_index/default.db/range_bitmap_sidecar/snapshot/snapshot-1
create mode 100644
crates/paimon/testdata/file_index/java/JavaFileIndexTableFixtureGeneratorTest.java
create mode 100755 crates/paimon/testdata/file_index/regenerate.sh
copy .gitignore => crates/paimon/testdata/video/video-frame-descriptor-v1.hex
(83%)
copy .gitignore => crates/paimon/testdata/video/video-v1.hex (78%)
create mode 100644 crates/paimon/tests/binary_partition_native_write_test.rs
create mode 100644 crates/paimon/tests/blob_collection_write_test.rs
create mode 100644 crates/paimon/tests/blob_http_error_test.rs
create mode 100644 crates/paimon/tests/cross_partition_write_test.rs
create mode 100644 crates/paimon/tests/data_evolution_row_sidecar_test.rs
create mode 100644 crates/paimon/tests/data_evolution_system_columns_test.rs
create mode 100644 crates/paimon/tests/data_file_directory_test.rs
create mode 100644 crates/paimon/tests/dedicated_blob_write_test.rs
create mode 100644 crates/paimon/tests/dynamic_bucket_write_test.rs
create mode 100644 crates/paimon/tests/external_data_write_test.rs
create mode 100644 crates/paimon/tests/file_index_java_fixture_test.rs
create mode 100644 crates/paimon/tests/incremental_diff_extended_test.rs
create mode 100644 crates/paimon/tests/nested_native_write_compat_test.rs
create mode 100644 crates/paimon/tests/parquet_write_page_index_test.rs
create mode 100644
crates/paimon/tests/pk_partial_update_sequence_group_parity_test.rs
create mode 100644 crates/paimon/tests/pk_user_sequence_parity_test.rs
create mode 100644 crates/paimon/tests/pk_value_stats_test.rs
create mode 100644 crates/paimon/tests/pk_write_reuse_seq_test.rs
create mode 100644 crates/paimon/tests/postpone_native_write_test.rs
create mode 100644 crates/paimon/tests/precomputed_bucket_write_test.rs
create mode 100644 crates/paimon/tests/reader_resources_test.rs
create mode 100644 crates/paimon/tests/table_partial_write_test.rs
create mode 100644 crates/paimon/tests/table_update_nested_test.rs
create mode 100644 crates/paimon/tests/table_update_paths_test.rs
create mode 100644 crates/paimon/tests/table_update_test.rs
create mode 100644 crates/paimon/tests/target_file_row_num_test.rs
create mode 100644 crates/paimon/tests/writer_resources_test.rs
create mode 100644 integrations/oss-cpp/CMakeLists.txt
create mode 100644 integrations/oss-cpp/README.md
create mode 100644 integrations/oss-cpp/bridge.cc
create mode 100644 integrations/oss-cpp/bridge.h
