This is an automated email from the ASF dual-hosted git repository.
XiaoHongbo-Hope pushed a change to branch release-2.1
in repository https://gitbox.apache.org/repos/asf/paimon.git
from 8ce380d29b [release] Align Spark 4.0 UT parent version with 2.1
new 45c42b6b9c [core] Avoid data-file conflict scans for index-only
compaction (#9963)
new a2e01c74b5 [format] Follow Parquet list backward-compatibility rules
in schema clipping (#9550)
new 5419cd07a4 [common] Keep the selected variant schema when its evidence
is gone (#9532)
new 53020ea338 [common] Release the BTree index file on every path that
abandons the writer (#9474)
new 5868c23105 [flink] Support negated predicates in PredicateConverter
(#9427)
new 128960d179 [python] Enable Parquet page indexes when supported (#9976)
new 8543cac0af [common] Reject a serialized row shorter than its
fixed-length part (#9991)
new d3df8b8dea [python] Stream vector index candidates during global
merging (#9989)
new d4bc20758d [python] Distribute batch vector refinement across Ray
workers (#9988)
new ff4737ff5c [core] Validate record counts read from serialized bytes
(#9985)
new 1625bfa230 [spark][core] Preserve SQL semantics for CAST predicate
pushdown (#9980)
new 7f6717f533 [core] Preserve merge correctness when pushing filters into
chain scans (#9979)
new c43ca187c0 [core] Allow primary-key-only deletes for deduplicate
tables (#9978)
new 510cf9e06b Fix: partition index bucket overflow (#9977)
new b1c2893521 [core] Refine candidate-only scalar index answers before
vector top-k (#9953)
new 4e859e83fe [flink] Preserve query authorization when narrowing splits
in the Flink source (#9949)
new d29b11899a [python] Unify RoboMIND ingestion and ACT table groups
(#9936)
new 85f23efce3 [core] Avoid redundant row-range sorting for global-index
results (#9957)
new 429d9a3808 [python] Stream native reads with split parallelism (#9996)
new 693d7a0132 [python] Reduce memory usage for overlapping temporal
windows (#10000)
new df72823821 [python] Verify candidate-only scalar filters before vector
top-k (#9999)
new ba03ea75b3 [python] Support Ray options in vector result conversions
(#9998)
new c03270067b [python] Close full-text readers when search raises
synchronously (#9997)
new 52108ead76 [core] Preserve reader errors for compacted changelog files
(#9995)
new 5bfcdac376 [python] Improve parallel native read scheduling (#10003)
new 54515f438f [core] Optimize bitmap range conversion and selected-row
reads (#10007)
new 3efdcfa30a [python] Expand native read coverage for nested and
streaming reads (#10009)
new 928698d359 [java][python] Read Python-planned splits with the native
reader (#10017)
new 03c4dc489d [python] Plan chunk-shuffled native reads with IndexedSplit
(#10023)
new d00a805168 [python][rest] Use HTTP status when error response omits
code (#10029)
new 9f27be3bf2 [core][flink] Support long limits in the read path (#10028)
new 4132be08ad [python] Reject vector and full-text search on a query-auth
table (#10027)
new ec34755dc1 [python] Apply query authorization to the Daft blob-table
native read (#10026)
new d8d74e47bb [python] Improve catalog API type annotations (#10025)
new c9a6bb4296 [core] Sort provably all-null splits last in TopN pushdown
with NULLS LAST (#10020)
new 4f2e1f3201 [python] Refuse a read whose query auth rules cannot be
bound to columns (#10018)
new 2459ff5c1a [python] Avoid full-window conversion for first and last
aggregates (#10016)
new 496adb298f [format] Decline ORC pushdown of empty IN/NOT IN instead of
crashing the reader (#10015)
new 43059bea65 [spark] Support LEGACY store assignment for format tables
(#10013)
new 90fe665dc3 [python][torch] Support shuffle for Arrow and tensor batch
streaming (#10012)
new 28d841b0ba [spark] Support timestamp_ntz columns in zorder and hilbert
clustering (#10011)
new 0c1eafcf5f [Python] Keep append distribution native end to end (#10032)
new f4b363ff0d [format] Honor JSON null map key options (#10031)
new f725268b83 [format] Cross-check the row-file block index against its
footer (#10001)
new 7f81e703b6 [flink][core] Introduce file-size option in Paimon source
(#9368)
new 4dd3d49687 [core] Fix snapshot expiration for chain table delta
commits (#9611)
new 0e3e513c08 [python] Fix Arrow large_string compatibility in PyPaimon
(#10002)
new 97b68f7730 [release] Fix Maven staging upload from reactor root
(#10004)
new 83d9dab971 [python] Add search result scores and explicit relevance
ordering (#10005)
new cf55c730be [python] Support data prefilters for full-text and hybrid
search (#10006)
new 397d341400 [core][format][spark] Support nested field predicate
pushdown (#9423)
new f70227e6be [python] Read Parquet row windows with OffsetIndex (#9850)
new 0dc7808e6b [python] Route descriptor-backed BLOB reads through the
table FileIO. (#9608)
new d9b543c9fc [Python] Use native planning and reads for streaming
changelog (#10036)
new 3ddd007e4e [Python] Align overwrite changelog with Java streaming
defaults (#10046)
new d1f7a7620a [python] Read BlobStore listings in bounded batches (#10044)
new abb9f2af3e [python] Support ARRAY BLOB in multimodal payload reads
(#10043)
new bb48ddc506 [python] Preserve retained snapshots in Ray row ID reads
(#10042)
new dfb03c3a2b [core] Do not prune TopN splits with unknown sort-column
statistics (#10039)
new dce26d474d [spark] Use the self-merge shortcut for conditional data
evolution UPDATE (#10037)
new 46e212b4c8 [python] Widen the read with the columns query auth rules
need (#10033)
new 176074d76f [python] Use native plan and read for BLOB LIMIT scans
(#10047)
new 036d318743 [python] Simplify native plan and read capability gates
(#10056)
new d54343f711 Expand Python native plan and read coverage for BLOB and PK
tables (#10065)
new 6dd3433c98 [spark] Identify the self-merge source by table location
and branch, not by name (#10071)
new 197cd4afeb [core] Validate deletion vector checksums (#10070)
new f3fd03e57f [format][python] State every block index invariant in the
row format spec (#10069)
new a7f8016ed1 [core] Keep shared IOManager open after clustering sort
(#10068)
new 13ac94a011 [spark] Resolve V1 UPDATE expressions against the relation
on tables with CHAR columns (#10061)
new c61ac7d850 [python] Support ARRAY and MAP BLOBs in Ray batch
processing (#10060)
new fb3baa1249 [python] Close BLOB stream iterators on errors (#10059)
new 78a2528c2e [python] Preserve schema fields in Python multimodal inputs
(#10058)
new 92eeea4617 [python] Keep BlobViewStruct bytes when blob-as-descriptor
is true. (#10057)
new 75168a5e36 [spark] Ignore source row ids outside the table in the data
evolution MERGE row-id shortcut (#10055)
new 8a224b015b [python] Fix MAP key resolution and ROW projection
nullability (#10054)
new 997b84b2d4 [spark] Read split statistics through the query
authorization wrapper (#10051)
new ce8a384896 [core] Improve BUCKET_APPEND_ORDERED test coverage (#10049)
new 0768c2314e [vortex] Free intermediate expressions in the predicate
converter (#10073)
new a777d4ae5e [java][python] Support OSS blob presigned URLs (#10040)
new e8a8c6d989 [flink] Support remove_orphan_blobs action and procedure.
(#10014)
new d527dfd030 [python] Enable native read in Python Rust Plan CI (#10067)
new 26a631a309 [python] Read VARIANT pages with Parquet OffsetIndex (#9962)
new abd2b9fdca [docs] Fix stale type support, procedure and CDC action
references (#9973)
new 27d9c5ebaf [format] Reject csv values containing a row separator
(#9938)
new afec9cf247 [vortex] Free the intermediate native expressions built for
a scan (#9841)
new 6b0499e392 [lumina] Reject non-positive encoding.pq.m at index build
time (#9823)
new f47a8a7623 [core] Use Locale.ROOT for every case conversion in main
sources (#9771)
new 901a54cabe [python] Remove dynamic bucket base snapshot commit guard
(#10075)
new d961d8085d [spark] Make bucket-preserving scans opt-in and retain
parallel read units (#10076)
new 9f948fcac6 [common] Limit REST token lifetime check to presigned URLs
new f64a3d1929 [core] Update row-count defaults and rename sorted index
file option (#10035)
new 4b225e7656 [core] Derive row-id conflict checks from commit messages
(#10079)
new 79bb901e37 [python] Remove manifest merging from commits (#10082)
new 06ab8965c9 [python] Add Java-compatible CommitMessage serialization
and native commit (#10081)
new 6e27f7aaae [common] Initialize JNI libraries independently by name
(#10100)
new bf075ad806 [python] Forward blob-as-descriptor to native readers
(#10112)
new 5b9a82eea3 [python] Avoid repeated catalog setup in parallel native
reads (#10118)
new 40f3caa849 [python] Support native batch overwrite commits (#10097)
new 1d48c35932 [python] Remove legacy native read option forwarding
(#10120)
new bf7eb2738f [core] Persist row ID reassignment plans and mark snapshots
(#10008)
new 9401c4a875 [vector] Upgrade paimon-vector-index to 0.5.0 and support
new build options (#10080)
new 40b185257a [format][python] Coalesce map blob metadata reads (#10121)
new 4386fc3241 [python][ray] Reduce Blob request QPS with URI affinity
(#9130)
new 68dcaea6b1 [avro] Avoid redundant sync when opening reader (#10140)
new 5d28406cec [minor] Clean useless class
new 2346437b22 [python] Run native commit tests only in the Rust Plan CI
job (#10139)
new f2927fd921 [cosn] Support alternative COSN authentication options
(#10130)
new 3085ee1072 [python] Preserve high-precision DECIMAL values in the row
file format (#10124)
new adacfadd51 [python] Speed up full shared-shredding MAP reconstruction
(#10148)
new 40e031c845 [python] Avoid redundant file metadata requests in Parquet
reads (#10152)
new 47ea308fb0 [core] Avoid eager manifest reads for append-only LIMIT
scans (#10153)
new b5786c473e [python] Speed up selected-key projection for ordinary MAP
columns (#10147)
new 71a364383a [spark] Expose Paimon metrics through JMX (#10131)
new 16e5f5c12a [benchmark] Close directory stream when loading sink
configurations (#10138)
new 69c3b272f6 [python] Keep reading an Avro file after a fully-filtered
batch (#10128)
new 1dd499c5a7 [core] Preserve sequence-group deletes in projected reads
(#10135)
new 9c82586106 [python] Vectorize selected-key reconstruction for
shared-shredding MAPs (#10143)
new 27be2c95bb [core] Validate chain partition drops against the
post-commit state (#10127)
new da9d8fd6ea [python] Add opt-in native writes to PyPaimon (#10154)
new 07cb0f6a9e [core] Rebuild file index over the correct column after
schema evolution (#10122)
new 1f8075c287 [spark] Fix a shifted timestamp written from before a zone
left local mean time (#10117)
new 4aa4368cf5 [core] Reject map key type changes in schema merging
(#10116)
new dbf56c2b2e [python] Fix RecursionError when reading a shard or slice
of a large file (#10113)
new 5165bbd3b8 [flink] Fix batch database compaction for Data Evolution
tables (#10110)
new f29077474b [common] Reject out-of-range bloom filter items and fpp
options (#10108)
new 925bd43078 [core] Report reachable dynamic-bucket max-buckets
misconfiguration clearly (#10104)
new 57b1961a47 [python] Decode palette image colors for training tensors
(#10096)
new 7e336f58fc [python] Resolve ARRAY BLOB payloads in training windows
(#10095)
new b19a193f28 [python] Support nested payloads in local MERGE INTO
(#10094)
new 490ee2e291 [python] Preserve fields from later Python MERGE source
rows (#10093)
new 740ec29fd6 [docs] Fix broken Parquet Variant encoding link in PyPaimon
data types (#10090)
new 397420f860 [core] Fix expire_snapshots accepting retain_min <= 0 and
deleting files of retained snapshots (#10089)
new 46e21f0cc6 [api] Use Locale.ROOT in DLFOpenApiV4Signer (#10087)
new 01a4357943 [core] Never prune postpone buckets in BucketSelector
(#10085)
new a0f7681e84 [core][spark] Support remove_unexisting_manifests procedure
(#10083)
new 155e7839a7 [core] Refactor sort compact to produce COMPACT commits
(#8631)
new 8a5ac6cc4c [core] Derive partition layout from active files (#10052)
new cefde382ca [python] Fix $files reporting empty min/max value stats
(#10111)
new cdaea14cf2 [python] Cast WHERE literals for DECIMAL, DATE and TIME
columns (#10119)
new e65c115075 [core] Push down deletion vectors for merged data evolution
groups (#9987)
new 921a99048f [python] Support vector index builds on deletion-vector
tables (#10157)
new c9603893ec [flink] Push down predicates on fields nested inside a row
(#10144)
new a340164c51 [python] Enable native write and REST commit in Native CI
(#10155)
new 7ede6a3aee [python] Fix CI for unpartitioned split generation (#10169)
new 4971003926 [flink] Support distributed global-index queries (#10146)
new f4f0511855 [flink] Refine distributed global-index query abstractions
(#10172)
new 2d96dc702c [python] Support encoding images as video (#10171)
new dbeee63ae8 [python] Support date/time extraction transforms in
query-auth rules (#10180)
new 0e1c407868 [python] Add a table rollback CLI command (#10177)
new feb9afb817 [python] Add the $consumers system table (#10174)
new 1a97db768a [test] Replace the removed MinIO test image with the
pgsty/silo drop-in (#10167)
new ba4554942a [python] Apply filters when reading system tables (#10164)
new fae6dae4e1 [spark] Fix a shifted date written from before the
Gregorian cutover (#10160)
new 26566f18b3 [python] Honor data-file.path-directory when resolving data
files (#10158)
new 840ba612a7 [python] Read row-format TIMESTAMP values in the
precision's time unit (#10125)
new 9bfa185350 [python] Reuse resolved REST metadata for native readers
(#10123)
new 3d794db40b Bump image-size from 2.0.2 to 2.0.4 in /docs (#10168)
new dd6034c748 [python] Honor sequence.field in primary-key writes (#10166)
new 4ade182070 [core] Add regression test for rewrite_file_index over an
arity-shrinking column drop (#10092)
new a601bde0a5 [python] Expose row-id updater through TableUpdate (#10188)
new 79c6ed7eb8 Use Rust core for native Arrow upserts and table updates
(#10185)
new df766b72b3 [python] Enable supported native Parquet writer
capabilities (#10202)
new 6bbf6c6272 [python] Enable native nested updates, temporal upserts and
descending sequences (#10242)
new a4783e3973 [python] Align data directories with Java and enable Native
IO (#10249)
new 328ea53f15 [fs] Optimize OSS retry strategy for QpsLimitExceeded and
5xx errors (#10299)
new dbf1e8637e [python] Remove ExpirationAt from DLF token JSON (#10186)
new f05a6de64f [python] Enable native writes with external data paths
(#10301)
new ed8ae2bd6a [python] Enable native scalar Blob Arrow writes (#10305)
new eb0d09d266 [python] Support efficient field selection from wide
VARIANT objects (#10320)
new 260937ec69 [core] Treat a snapshot rename that throws but succeeded as
committed (#10324)
new 0bc69d7b9f [jindo] Unify the Jindo User-Agent format (#10321)
new 017549756e [python] Fix DECIMAL Arrow predicates across scales (#10317)
new 78f0b99e0b [spark] Honor delete.force-produce-changelog on a
full-table DELETE (#10311)
new 0a77e9e9e2 [python] Unify the REST User-Agent format (#10309)
new 24cc15731b [rest] Unify the REST User-Agent format (#10308)
new 330b0cc967 [python] Unify the OSS User-Agent format (#10307)
new 4f76ae25e1 [oss] Unify the OSS User-Agent format (#10306)
new 8d42cb8552 [core] Preserve partial-update sequence-group deletion
state (#10300)
new d6067c433f [lance] Close the native and Arrow writers even when the
final flush fails (#10298)
new 74be45796c [core] Reject a merge fan-in below two up front (#10293)
new 4cd63e0aa7 [core] Push down bitmap indexes for data evolution merged
reads (#10289)
new d4b1e22f32 [core] Continue FileStoreWrite cleanup after writer close
failures (#10288)
new 6f6ee6b235 [core] Skip DV reads for file-index-rejected single files
(#10287)
new d2abc0ea06 [core] Tolerate retracting an absent secondary index key
(#10280)
new cb5cd053cf [core] Rethrow foreign typed 404s from alterTable (#10268)
new c12cb9144b [python] Replace blob index entry limit with a memory
budget (#10151)
new f98b8dfb95 [core] Add composite BTree key storage and point lookups
(#10338)
new 04ff162588 Bump fast-uri from 3.1.7 to 3.1.8 in /docs (#10331)
new 1515e76f67 [core] Support composite BTree equality queries for data
evolution tables (#10339)
new 1c4b925dbe [flink][spark] Support composite BTree global index
creation (#10345)
new 0fe8364609 [core] Support composite BTree prefix and range queries
(#10346)
new b61cda00d1 [python] Align FileType mutability with Java (#10344)
new 0e7081883b [core] Treat temp and Iceberg hint files as mutable in
FileType (#10342)
new dfd3b5efc6 [core] Skip surrogate code points in truncated string max
(#10335)
new 122064a681 Bump brace-expansion from 1.1.18 to 1.1.21 in /docs (#10330)
new 121f3c0b16 [python] Add REST permission management (#10322)
new c8433add0c [python] Centralize PyPaimon packaging configuration
(#10241)
new f3f95fb762 [python] Reduce chunk-shuffle planning memory (#10232)
new ca6e4fb3e1 [spark] Reject recursive Paimon view definitions (#10231)
new 21a133056b [oss] Fix blob presigned URL validation when
fs.oss.sld.enabled is set (#10230)
new 104b3f39dc [filesystem][s3] Mirror s3.session-token to
fs.s3a.session.token (#10228)
new 5b2b387754 [flink] Preserve rollback protection tags on uncertain
commit outcomes (#10204)
new a1618e852a [core] Prune FM index partitions by query row ranges
(#10201)
new 92cc745225 [spark] Add rollback_to_as_latest procedure (#10199)
new 34154080c4 [python] Support the remaining date-extraction transforms
in query-auth rules (#10197)
new 24e9f5ce2f [spark] Support decimal columns in zorder clustering
(#10196)
new 652fe151ad [core] Restore interrupt status during snapshot expiration
(#10195)
new 4246095ed9 [format] Treat an unrecognized boolean string as malformed
in CSV and JSON (#10194)
new 09e46ca790 [python] Write TIMESTAMP(0) columns to Avro as
timestamp-millis (#10193)
new fb0280fc29 [python] Implement write_row in HdfsNativeFileIO (#10192)
new 2bda94fd54 [python] Treat auto-create=false as false in create_table
(#10191)
new 4448503c59 [python] Refuse a blank query auth rule instead of skipping
it (#10189)
new a7aa1e19cb [iceberg] Keep tables loadable when the mirror cannot
publish their schema (#10187)
new 85e416f7d0 [filesystem][s3] Cache FileSystem by scheme and authority
in HadoopCompliantFileIO (#10182)
new 2ebe99109a [python] Support LENGTH, BIT_LENGTH, OVERLAY, PAD and
TRANSLATE in query-auth rules (#10179)
new bdd017a26b [python] Fall back to table metadata when REST partition
listing is unimplemented (#10176)
new 23211604c8 [python] Reject unsupported aggregation options before
writing (#10165)
new 54e27b9efa [spark] Support expire_changelogs procedure (#10062)
new 279f015643 [core] Support composite BTree IN and NULL queries (#10347)
new 519a0df790 [test] Fix index field list in partition pruning restore
test (#10348)
new 1929cfb1d0 [core] Compare composite BTree keys directly from slices
(#10349)
new 49cf457ef0 [python] Enable native dynamic bucket writes and align HASH
commits (#10353)
new 837408a73d [python] Expand native cross-partition and postpone write
support (#10354)
new e4766d48a0 [python] Preserve postpone events and enable native
postpone merge engines (#10368)
new 93723440b9 [python] Enable Native MAP shared-shredding writes and
updates (#10382)
new 7da3243f78 [python] Enable native REST commits for Data Evolution
tables (#10383)
new 7e0e0bbb96 [python] Enable native Paimon branch writes (#10385)
new 4b24b59cdc [Python] Enable native Blob and VARIANT updates with Java
layouts (#10390)
new c90876aa5c [python] Push down typed Variant projections (#10336)
new 2e2a000a57 [core] Push down DV-aware bitmap selection for merged
groups (#10387)
new dfad52a039 [python] Fix fragmented Parquet row groups in MAP
shared-shredding writes (#10386)
new 21a059d800 [core] Decide full-text row filters from data when no
scalar index can evaluate them (#10381)
new ef89d18f00 [format] Drop nested filters on unprojected Parquet fields
before conversion (#10379)
new d1e1111fd3 [format] Support lineSep as fallback key of
json.line-delimiter (#10377)
new 1ca4c2db7e [python] Parse SQL NOT NULL case-insensitively in pypaimon
(#10373)
new b976727d46 [cdc] Return an empty string for canal/aliyun enum index 0
(#10371)
new d83c70d03d [cdc] Pair canal UPDATE rows by position instead of by
content (#10370)
new 6436395a1f [flink] Reject a non-positive
scan.split-enumerator.batch-size (#10369)
new 7424afc9e6 [core] Validate the timestamp field type for
merge_map_with_keytime (#10367)
new d7a7df48a7 [python] Normalize LTZ global index keys to UTC (#10365)
new 43d771933a [python] Preserve full precision in decimal global index
keys (#10363)
new f06a590080 [python] Handle unsupported composite BTree indexes safely
(#10352)
new 38394aa6af [python] Avoid redundant manifest size lookups (#10310)
new 83e9ef9785 [core] Skip manual-only tag groups in preview time travel
(#10284)
new 20d5ad7ad7 [core] Keep all sibling tags of a snapshot as Iceberg refs
(#10248)
new bcd0306c71 [core] Rebuild the default value row when the write type
changes (#10208)
new 3fa88b962b [flink] Fix flaky AppendTableITCase streaming compaction
tests (#10184)
new 8150015eee [core] Return empty schemas/snapshots table result for
out-of-range id filters (#10206)
new 9adece5609 [core] Do not count inherited index entries in the
strict-mode partition check (#10218)
new 355b5db22c [core] Replay index file changes when restoring
MergeTreeWriter (#10260)
new 11d3520f6a [format][python] Persist keyframe indexes for efficient
random video frame reads (#9831)
new 6c410588ea [iceberg] Refuse types the mirror cannot publish before the
commit (#9477)
new 788a50678b [python][ray] Fix merge hang with shuffled sources on small
clusters (#9360)
new 7da6c4d12f [python] Use structured read types for nested, MAP and
VARIANT projections (#10392)
new fd1d5db825 [python] Enable native BLOB collection writes and packed
video reads (#10399)
new e979303290 [Python] Enable Native Data Evolution ROW sidecars (#10404)
new 27779e233a [python] Propagate native fork-safety errors (#10328)
new 99ab60ac66 [Python] Enable native row-tracking sequence reads and
predicates (#10408)
new 59fce1931f [Python] Enable native primary-key BLOB limits and floating
sequence fields (#10410)
new 7a8bce984d [Python] Enable native column-subset writes and upserts
(#10411)
new 998af97f55 [Python] Keep grouped bucket writes on Native (#10412)
new 60cf0bce50 [Python] Delegate primary-key scan sharding to Native
(#10413)
new 869908e366 [python] Avoid duplicate LeRobot video batch copies (#10405)
new 0841d4ecd1 [python] Use native upsert for rows with different supplied
fields (#10414)
new 3df4d486e6 [python] Keep BlobConsumer writes native (#10416)
new 5253feb8a6 [python] Keep custom Blob URI readers on native writes
(#10417)
new 3958024fba [python] Run data evolution MERGE through native core
(#10418)
new 73edf931fa [python] Skip redundant null masks in nested projections
(#10421)
new fb6fbe5160 [python] Run Paimon table-source MERGE in native core
(#10419)
new 2b1f438f29 [python] Run timestamp-window MERGE sources in native core
(#10428)
new cfd0d9b420 [python] Support deletion vectors in Ray row-id reads and
updates (#10427)
new 681f91d914 [python] Execute full-text and hybrid search with Ray
(#10430)
new 71c2f2a737 [spark][flink] Add import_files procedure with external
paths (#10423)
The 273 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci-python.yml | 174 +-
.github/workflows/release-python-publish.yml | 2 +-
.github/workflows/release-python.yml | 16 +-
.github/workflows/release.yml | 2 +-
.gitignore | 1 +
docs/docs/cdc-ingestion/mysql-cdc.mdx | 3 +-
docs/docs/concepts/spec/fileformat.md | 50 +-
docs/docs/concepts/spec/rowformat.md | 25 +-
docs/docs/flink/procedures.md | 4 +-
docs/docs/flink/procedures/compaction.md | 5 +-
docs/docs/flink/procedures/indexes.md | 2 +-
docs/docs/flink/procedures/repair.md | 79 +-
docs/docs/flink/procedures/snapshots.md | 7 +-
docs/docs/flink/procedures/table-operations.md | 41 +
docs/docs/flink/sql-query.mdx | 2 +-
docs/docs/maintenance/filesystems.mdx | 9 +
docs/docs/multimodal-table/blob-references.mdx | 22 +-
.../data-evolution-maintenance.mdx | 10 +
docs/docs/multimodal-table/global-index/bitmap.mdx | 6 +-
docs/docs/multimodal-table/global-index/btree.mdx | 81 +-
.../multimodal-table/global-index/full-text.mdx | 9 +-
.../global-index/manage-indexes.mdx | 8 +-
.../multimodal-table/global-index/multivalue.mdx | 6 +-
docs/docs/multimodal-table/global-index/vector.mdx | 66 +-
docs/docs/multimodal-table/video.mdx | 37 +-
docs/docs/primary-key-table/blob-storage.md | 5 +-
docs/docs/primary-key-table/data-distribution.md | 2 +-
.../merge-engine/partial-update.md | 5 +
docs/docs/program-api/file-cache.mdx | 8 +-
docs/docs/project/creating-a-release.md | 9 +-
docs/docs/project/releases.md | 2 +-
docs/docs/project/verifying-a-release-candidate.md | 14 +-
docs/docs/pypaimon/blob-store.md | 19 +-
docs/docs/pypaimon/blob.md | 6 +
docs/docs/pypaimon/data-types.md | 33 +-
docs/docs/pypaimon/installation.md | 18 +-
docs/docs/pypaimon/lerobot.md | 21 +-
docs/docs/pypaimon/multimodal-reading.md | 30 +-
docs/docs/pypaimon/multimodal-search.md | 130 +-
docs/docs/pypaimon/pyjindosdk-support.md | 3 +
docs/docs/pypaimon/pytorch.md | 52 +-
docs/docs/pypaimon/ray-data.md | 7 +-
docs/docs/pypaimon/ray-row-ids.md | 21 +-
docs/docs/pypaimon/robomind-act-benchmark.md | 47 +-
docs/docs/pypaimon/robomind-agilex.md | 231 ++-
docs/docs/pypaimon/video.md | 30 +
docs/docs/pypaimon/writing.md | 25 +-
docs/docs/spark/configuration.mdx | 3 +
docs/docs/spark/procedures.md | 6 +-
docs/docs/spark/procedures/indexes.md | 2 +-
docs/docs/spark/procedures/maintenance.md | 46 +
docs/docs/spark/procedures/migration.md | 38 +
docs/docs/spark/procedures/versions.md | 23 +
docs/docs/spark/sql-query.md | 38 +
docs/docs/spark/sql-write.md | 3 +
docs/generated/catalog_configuration.html | 20 +-
docs/generated/core_configuration.html | 16 +-
docs/generated/flink_connector_configuration.html | 6 +
docs/generated/kafka_sync_database.html | 4 -
docs/generated/mysql_sync_database.html | 4 +
docs/generated/pulsar_sync_database.html | 4 -
docs/generated/spark_connector_configuration.html | 6 +
docs/generated/temporal_functions.html | 2 +-
docs/yarn.lock | 18 +-
paimon-api/pom.xml | 13 +
.../org/apache/paimon/utils/BuildVersions.java | 13 +-
.../main/java/org/apache/paimon/CoreOptions.java | 43 +-
.../org/apache/paimon/options/CatalogOptions.java | 28 +-
.../java/org/apache/paimon/options/Options.java | 49 +-
.../org/apache/paimon/options/OptionsUtils.java | 2 +-
.../java/org/apache/paimon/rest/HttpClient.java | 5 +-
.../org/apache/paimon/rest/HttpClientUtils.java | 35 +
.../main/java/org/apache/paimon/rest/RESTApi.java | 19 +-
.../paimon/rest/auth/DLFAuthProviderFactory.java | 3 +-
.../apache/paimon/rest/auth/DLFDefaultSigner.java | 15 +-
.../apache/paimon/rest/auth/DLFOpenApiSigner.java | 2 +-
.../paimon/rest/auth/DLFOpenApiV4Signer.java | 3 +-
.../apache/paimon/types/DataTypeJsonParser.java | 5 +-
.../main/java/org/apache/paimon/types/RowKind.java | 4 +-
.../java/org/apache/paimon/utils/StringUtils.java | 9 +-
.../apache/paimon/TurkishLocaleParsingTest.java | 73 +
.../apache/paimon/rest/HttpClientUtilsTest.java | 40 +
.../paimon/rest/auth/DLFOpenApiV4SignerTest.java | 22 +
.../java/org/apache/paimon/benchmark/Sink.java | 27 +-
.../benchmark/metric/cpu/OperatingSystem.java | 4 +-
paimon-benchmark/paimon-micro-benchmarks/pom.xml | 6 +
.../benchmark/AvroSmallFileReadBenchmark.java | 152 ++
.../org/apache/paimon/casting/DefaultValueRow.java | 33 +-
.../compression/BlockCompressionFactory.java | 4 +-
.../apache/paimon/data/VideoFrameDescriptor.java | 54 +-
.../data/serializer/BinaryRowSerializer.java | 16 +
.../data/variant/InferVariantShreddingSchema.java | 49 +-
.../bloomfilter/BloomFilterFileIndex.java | 7 +
.../java/org/apache/paimon/format/FileFormat.java | 12 +-
.../src/main/java/org/apache/paimon/fs/FileIO.java | 5 +-
.../paimon/globalindex/CompositeKeySerializer.java | 91 ++
.../globalindex/ContainsRefinementEvaluator.java | 9 +-
.../paimon/globalindex/GlobalIndexEvaluator.java | 43 +-
.../globalindex/GlobalIndexKeyExtractor.java | 2 +-
.../paimon/globalindex/GlobalIndexReader.java | 6 +
.../apache/paimon/globalindex/GlobalIndexer.java | 18 +-
.../paimon/globalindex/GlobalIndexerFactory.java | 23 +-
.../globalindex/GlobalIndexerFactoryUtils.java | 14 +
.../paimon/globalindex/HybridSearchRanker.java | 3 +-
.../apache/paimon/globalindex/KeySerializer.java | 6 +
.../globalindex/SortedFileGlobalIndexReader.java | 105 +-
.../paimon/globalindex/SortedFileIndexPlanner.java | 178 +++
.../paimon/globalindex/SortedFileMetaSelector.java | 17 +
.../paimon/globalindex/SortedGlobalIndexer.java | 5 +-
.../globalindex/bitmap/BitmapGlobalIndexer.java | 2 +
.../bitmap/BitmapGlobalIndexerFactory.java | 20 +-
.../bitmap/MultiValueGlobalIndexer.java | 2 +
.../bitmap/MultiValueGlobalIndexerFactory.java | 9 +-
.../globalindex/btree/BTreeGlobalIndexer.java | 31 +-
.../btree/BTreeGlobalIndexerFactory.java | 23 +-
.../paimon/globalindex/btree/BTreeIndexReader.java | 65 +-
.../paimon/globalindex/btree/BTreeIndexWriter.java | 34 +-
.../paimon/globalindex/btree/BTreePostingList.java | 34 +-
.../globalindex/btree/CompositeBTreePredicate.java | 437 +++++
.../globalindex/btree/LazyFilteredBTreeReader.java | 47 +-
.../globalindex/fmindex/FMGlobalIndexer.java | 18 +
.../fmindex/FMGlobalIndexerFactory.java | 9 +-
.../globalindex/sorted/SortedIndexOptions.java | 9 +-
.../apache/paimon/lookup/sort/db/LocalKvDb.java | 4 +-
.../org/apache/paimon/predicate/CastTransform.java | 3 +-
.../paimon/predicate/NestedFieldTransform.java | 223 +++
.../apache/paimon/predicate/PredicateBuilder.java | 12 +-
.../org/apache/paimon/predicate/Transform.java | 1 +
.../apache/paimon/reader/FileRecordIterator.java | 25 +-
.../apache/paimon/reader/LimitRecordReader.java | 2 +-
.../org/apache/paimon/rest/RESTTokenFileIO.java | 91 +-
.../java/org/apache/paimon/sst/BlockIterator.java | 8 +-
.../java/org/apache/paimon/sst/SstFileReader.java | 9 +-
.../paimon/statistics/SimpleColStatsCollector.java | 3 +-
.../TruncateSimpleColStatsCollector.java | 4 +
.../java/org/apache/paimon/utils/BinPacking.java | 4 +-
.../apache/paimon/utils/BlobDescriptorUtils.java | 3 +-
.../org/apache/paimon/utils/BloomFilter64.java | 12 +-
.../java/org/apache/paimon/utils/FileType.java | 30 +-
.../java/org/apache/paimon/utils/JNIUtils.java | 11 +-
.../org/apache/paimon/utils/OperatingSystem.java | 4 +-
.../org/apache/paimon/utils/ProcedureUtils.java | 29 +
.../apache/paimon/utils/RoaringNavigableMap64.java | 16 +-
.../org/apache/paimon/utils/RowRangeIndex.java | 10 +
.../apache/paimon/data/DefaultValueRowTest.java | 23 +
.../paimon/data/VideoFrameDescriptorTest.java | 48 +-
.../serializer/BinaryRowSerializerLengthTest.java | 96 ++
.../variant/InferVariantShreddingSchemaTest.java | 143 ++
.../data/variant/VariantMetadataUtilsTest.java | 61 +
.../bloomfilter/BloomFilterFileIndexTest.java | 41 +
.../paimon/format/FileFormatPrefixOptionsTest.java | 101 ++
.../globalindex/CompositeKeySerializerTest.java | 198 +++
.../globalindex/SortedFileMetaSelectorTest.java | 71 +
.../globalindex/SortedIndexFileMetaTest.java | 8 +-
.../bitmap/LazyFilteredBitmapIndexReaderTest.java | 26 +-
.../bitmap/MultiValueBitmapIndexReaderTest.java | 3 +
.../globalindex/btree/AbstractIndexReaderTest.java | 5 +-
.../globalindex/btree/BTreeBloomFilterTest.java | 48 +-
.../btree/BTreeIndexReaderCloseTest.java | 13 +-
.../globalindex/btree/BTreeIndexReaderTest.java | 87 +
.../btree/BTreeIndexWriterCloseTest.java | 204 +++
.../globalindex/btree/BTreePostingListTest.java | 29 +-
.../globalindex/btree/BTreeThreadSafetyTest.java | 13 +-
.../globalindex/btree/CompositeBTreeIndexTest.java | 494 ++++++
.../btree/CompositeBTreePredicateTest.java | 297 ++++
.../globalindex/btree/CompositeBTreeRangeTest.java | 319 ++++
.../btree/LazyFilteredBTreeIndexReaderTest.java | 45 +-
.../globalindex/fmindex/FMGlobalIndexTest.java | 172 +-
.../globalindex/sorted/SortedIndexOptionsTest.java | 28 +-
.../testfulltext/TestFullTextGlobalIndexer.java | 2 +
.../TestFullTextGlobalIndexerFactory.java | 23 +-
.../TestMultiFieldVectorGlobalIndexer.java | 12 +-
.../testvector/TestVectorGlobalIndexer.java | 2 +
.../testvector/TestVectorGlobalIndexerFactory.java | 15 +-
.../apache/paimon/predicate/CastTransformTest.java | 90 ++
.../paimon/predicate/NestedFieldTransformTest.java | 372 +++++
.../paimon/predicate/PredicateBuilderTest.java | 31 +
.../paimon/reader/FileRecordIteratorTest.java | 151 +-
.../paimon/reader/LimitRecordReaderTest.java | 20 +-
.../apache/paimon/rest/RESTTokenFileIOTest.java | 129 ++
.../org/apache/paimon/sst/BlockIteratorTest.java | 21 +
.../statistics/SimpleColStatsCollectorTest.java | 19 +
.../org/apache/paimon/utils/BinPackingTest.java | 18 +-
.../java/org/apache/paimon/utils/FileTypeTest.java | 81 +
.../paimon/utils/RoaringNavigableMap64Test.java | 45 +
.../org/apache/paimon/utils/RowRangeIndexTest.java | 21 +
.../org/apache/paimon/utils/StringUtilsTest.java | 16 +
.../paimon/data/video-frame-descriptor-v1.hex | 2 +-
.../append/SortCompactCommitMessageRewriter.java | 718 +++++++++
.../paimon/append/SortCompactPlanMetadata.java | 165 ++
.../org/apache/paimon/append/cluster/Sorter.java | 9 +-
.../DataEvolutionCompactTaskSerializer.java | 10 +-
.../DataEvolutionNormalCompactTask.java | 15 +-
.../DataEvolutionRowIdReassigner.java | 12 +-
.../append/dataevolution/RowRangeMappingIndex.java | 26 +
.../dataevolution/SerializationAssignment.java | 205 +++
.../org/apache/paimon/catalog/CatalogUtils.java | 3 +-
.../paimon/catalog/RenamingSnapshotCommit.java | 12 +-
.../paimon/deletionvectors/DeletionVector.java | 24 +-
.../deletionvectors/DeletionVectorChecksum.java | 52 +
.../java/org/apache/paimon/disk/FileIOChannel.java | 4 +-
.../paimon/globalindex/DataEvolutionBatchScan.java | 153 +-
.../DataEvolutionGlobalIndexCoverage.java | 30 +-
.../DataEvolutionGlobalIndexRefreshPlanner.java | 4 +-
.../DataEvolutionGlobalIndexScanner.java | 201 ++-
.../globalindex/GlobalIndexBuilderUtils.java | 16 +-
.../paimon/globalindex/GlobalIndexQuery.java | 812 ++++++++++
.../apache/paimon/globalindex/IndexQuerySplit.java | 184 +++
.../sorted/SortedGlobalIndexScanner.java | 35 +-
.../sorted/SortedGlobalIndexWriter.java | 29 +-
.../paimon/iceberg/IcebergCommitCallback.java | 21 +-
.../apache/paimon/iceberg/IcebergPathFactory.java | 4 +-
.../apache/paimon/index/FileIndexProcessor.java | 7 +-
.../apache/paimon/index/HashBucketAssigner.java | 1 +
.../paimon/index/IndexFileMetaV1Deserializer.java | 6 +-
.../paimon/index/IndexFileMetaV2Deserializer.java | 6 +-
.../paimon/index/IndexFileMetaV3Deserializer.java | 6 +-
.../paimon/index/IndexFileMetaV4Deserializer.java | 6 +-
.../org/apache/paimon/index/PartitionIndex.java | 44 +-
.../paimon/index/SimpleHashBucketAssigner.java | 22 +-
.../index/pkfulltext/PkFullTextIndexFile.java | 8 +-
.../index/pksorted/PkSortedIndexBuilder.java | 4 +-
.../paimon/index/pksorted/PkSortedIndexFile.java | 5 +-
.../index/pkvector/PkVectorAnnSegmentFile.java | 4 +-
.../index/pkvector/PkVectorAnnSegmentSearcher.java | 8 +-
.../java/org/apache/paimon/io/DataFileMeta.java | 10 +-
.../apache/paimon/io/DataFileMeta08Serializer.java | 6 +-
.../apache/paimon/io/DataFileMeta09Serializer.java | 6 +-
.../paimon/io/DataFileMeta10LegacySerializer.java | 6 +-
.../paimon/io/DataFileMeta12LegacySerializer.java | 6 +-
.../apache/paimon/io/DataFileMetaSerializer.java | 6 +-
.../org/apache/paimon/io/DataFileRecordReader.java | 10 +-
.../org/apache/paimon/io/FileIndexEvaluator.java | 36 +
.../org/apache/paimon/io/PojoDataFileMeta.java | 67 +-
.../apache/paimon/io/ProjectedDataFileMeta.java | 11 +-
.../paimon/jdbc/DistributedLockDialectFactory.java | 4 +-
.../paimon/lookup/local/LocalKvStateFactory.java | 3 +-
.../paimon/lookup/memory/InMemorySetState.java | 7 +-
.../paimon/manifest/IndexManifestFileHandler.java | 16 +-
.../manifest/ManifestCommittableSerializer.java | 5 +-
.../apache/paimon/mergetree/DataFileReader.java | 68 -
.../apache/paimon/mergetree/MergeTreeWriter.java | 28 +-
.../compact/PartialUpdateMergeFunction.java | 15 +-
.../FieldMergeMapWithKeyTimeAggFactory.java | 9 +
.../metastore/ChainTableCommitPreCallback.java | 63 +-
.../ChainTableOverwriteCommitCallback.java | 15 +-
.../paimon/metastore/ChainTableOverwriteScope.java | 75 +
.../paimon/migrate/ExternalFileImporter.java | 148 ++
.../paimon/operation/AbstractFileStoreScan.java | 38 +-
.../paimon/operation/AppendOnlyFileStoreScan.java | 7 +-
.../paimon/operation/BaseAppendFileStoreWrite.java | 8 +-
.../apache/paimon/operation/BucketSelector.java | 7 +
.../paimon/operation/DataEvolutionSplitRead.java | 273 +++-
.../apache/paimon/operation/FileDeletionBase.java | 12 +
.../apache/paimon/operation/FileStoreCommit.java | 8 +-
.../paimon/operation/FileStoreCommitImpl.java | 83 +-
.../paimon/operation/KeyValueFileStoreWrite.java | 4 +-
.../paimon/operation/MemoryFileStoreWrite.java | 8 +-
.../apache/paimon/operation/OrphanFilesClean.java | 6 +
.../apache/paimon/operation/RawFileSplitRead.java | 11 +-
.../operation/RemoveUnexistingManifests.java | 78 +-
.../org/apache/paimon/operation/SplitRead.java | 6 +-
.../paimon/operation/commit/ConflictDetection.java | 8 +
.../commit/DataEvolutionConflictDetection.java | 107 ++
.../paimon/operation/commit/StrictModeChecker.java | 38 +-
.../java/org/apache/paimon/rest/RESTCatalog.java | 4 +
.../apache/paimon/schema/SchemaMergingUtils.java | 16 +
.../org/apache/paimon/schema/SchemaValidation.java | 53 +-
.../paimon/sort/AbstractBinaryExternalMerger.java | 9 +
.../paimon/table/AbstractFileStoreTable.java | 12 +-
.../paimon/table/AppendOnlyFileStoreTable.java | 3 +-
.../apache/paimon/table/ChainGroupReadTable.java | 30 +-
.../paimon/table/ChainTableFileStoreTable.java | 7 +-
.../apache/paimon/table/ExpireChangelogImpl.java | 63 +-
.../apache/paimon/table/ExpireSnapshotsImpl.java | 7 +-
.../paimon/table/FallbackReadFileStoreTable.java | 2 +-
.../java/org/apache/paimon/table/FormatTable.java | 3 +-
.../paimon/table/PrimaryKeyFileStoreTable.java | 56 +-
.../paimon/table/format/FormatReadBuilder.java | 19 +-
.../paimon/table/format/FormatTableRead.java | 4 +-
.../paimon/table/format/FormatTableScan.java | 6 +-
.../table/sink/AppendCompactTaskSerializer.java | 6 +-
.../paimon/table/sink/BatchWriteBuilderImpl.java | 11 +-
.../apache/paimon/table/sink/CommitMessage.java | 8 +
.../paimon/table/sink/CommitMessageImpl.java | 40 +-
.../sink/CommitMessageLegacyV2Serializer.java | 6 +-
.../paimon/table/sink/CommitMessageSerializer.java | 29 +-
.../apache/paimon/table/sink/InnerTableCommit.java | 5 +-
.../sink/MultiTableCompactionTaskSerializer.java | 6 +-
.../apache/paimon/table/sink/TableCommitImpl.java | 11 +-
.../apache/paimon/table/sink/TableWriteImpl.java | 67 +-
.../table/source/AbstractBatchTableScan.java | 18 +-
.../source/AbstractDataEvolutionVectorRead.java | 49 +-
.../paimon/table/source/AbstractDataTableRead.java | 10 +-
.../paimon/table/source/AppendTableRead.java | 4 +-
.../table/source/BlobViewTableReadSupport.java | 2 +-
.../org/apache/paimon/table/source/ChainSplit.java | 4 +-
.../table/source/DataEvolutionFullTextRead.java | 72 +-
.../table/source/DataEvolutionFullTextScan.java | 2 +-
.../table/source/DataEvolutionTableRead.java | 40 +-
.../table/source/DataEvolutionVectorScan.java | 3 +-
.../org/apache/paimon/table/source/DataSplit.java | 5 +-
.../apache/paimon/table/source/DeletionFile.java | 5 +-
.../paimon/table/source/FilteredRowIdReader.java | 58 +
.../paimon/table/source/IncrementalSplit.java | 10 +-
.../apache/paimon/table/source/InnerTableRead.java | 2 +-
.../apache/paimon/table/source/InnerTableScan.java | 2 +-
.../paimon/table/source/KeyValueTableRead.java | 12 +-
.../org/apache/paimon/table/source/PlanImpl.java | 12 +-
.../paimon/table/source/PrimaryKeyBatchScan.java | 3 +-
.../table/source/PrimaryKeyFullTextRead.java | 5 +-
.../table/source/PrimaryKeySortedIndexScan.java | 4 +-
.../apache/paimon/table/source/QueryAuthSplit.java | 10 -
.../paimon/table/source/RawFullTextReadImpl.java | 3 +-
.../apache/paimon/table/source/ReadBuilder.java | 13 +
.../paimon/table/source/ReadBuilderImpl.java | 24 +-
.../paimon/table/source/SplitSerializer.java | 7 +
.../org/apache/paimon/table/source/Splits.java | 83 +
.../table/source/TopNDataSplitEvaluator.java | 131 +-
.../snapshot/IncrementalDeltaStartingScanner.java | 2 +-
.../table/source/snapshot/SnapshotReader.java | 6 +-
.../table/source/snapshot/SnapshotReaderImpl.java | 9 +-
.../apache/paimon/table/system/AuditLogTable.java | 4 +-
.../apache/paimon/table/system/SchemasTable.java | 12 +-
.../paimon/table/system/SystemTableLoader.java | 3 +-
.../java/org/apache/paimon/tag/TagPreview.java | 17 +-
.../apache/paimon/utils/DataEvolutionUtils.java | 6 +-
.../org/apache/paimon/utils/ObjectSerializer.java | 7 +-
.../apache/paimon/utils/PartitionPathUtils.java | 3 +-
.../apache/paimon/utils/SerializationUtils.java | 25 +
.../org/apache/paimon/utils/SnapshotManager.java | 12 +-
.../test/java/org/apache/paimon/JavaPyE2ETest.java | 70 +-
.../org/apache/paimon/append/BlobTableTest.java | 71 +-
.../SortCompactCommitMessageRewriterTest.java | 1370 ++++++++++++++++
.../apache/paimon/append/VectorStoreTableTest.java | 12 +-
.../apache/paimon/append/cluster/SorterTest.java | 55 +
.../DataEvolutionNormalCompactTaskTest.java | 29 +-
.../DataEvolutionRowIdReassignerTest.java | 427 ++++-
.../dataevolution/RowRangeMappingIndexTest.java | 25 +
.../paimon/catalog/RenamingSnapshotCommitTest.java | 70 +
.../paimon/catalog/TableQueryAuthResultTest.java | 57 +
.../paimon/deletionvectors/DeletionVectorTest.java | 77 +
.../DeletionVectorsIndexFileTest.java | 38 +
...DataEvolutionGlobalIndexRefreshPlannerTest.java | 24 +-
.../paimon/globalindex/GlobalIndexQueryTest.java | 816 ++++++++++
.../sorted/SortedGlobalIndexScannerTest.java | 31 +-
.../sorted/SortedGlobalIndexTestUtils.java | 83 +-
.../sorted/SortedGlobalIndexWriterTest.java | 2 +-
.../paimon/iceberg/IcebergCompatibilityTest.java | 205 +++
.../paimon/index/FileIndexProcessorTest.java | 143 ++
.../paimon/index/HashBucketAssignerTest.java | 35 +
.../apache/paimon/index/PartitionIndexTest.java | 70 +
.../paimon/index/SimpleHashBucketAssignerTest.java | 32 +
.../index/pkfulltext/PkFullTextIndexFileTest.java | 2 +
.../index/pksorted/PkSortedIndexBuilderTest.java | 6 +-
.../pksorted/PrimaryKeySortedIndexOptionsTest.java | 2 +
.../paimon/io/DataFileMetaSerializerTest.java | 40 +-
.../paimon/io/ProjectedDataFileMetaTest.java | 8 +-
.../paimon/lookup/memory/InMemorySetStateTest.java | 49 +
.../manifest/IndexManifestFileHandlerTest.java | 97 ++
...festCommittableSerializerCompatibilityTest.java | 32 +-
.../manifest/ManifestEntrySerializerTest.java | 9 +-
.../apache/paimon/manifest/ManifestFileTest.java | 6 +-
.../LookupChangelogMergeFunctionWrapperTest.java | 54 +
.../compact/PartialUpdateMergeFunctionTest.java | 314 ++++
.../compact/aggregate/FieldAggregatorTest.java | 35 +
.../metastore/VisibilityWaitCallbackTest.java | 8 +-
.../paimon/migrate/ExternalFileImporterTest.java | 386 +++++
.../operation/AbstractFileStoreWriteCloseTest.java | 108 +-
.../paimon/operation/BucketSelectorTest.java | 34 +
.../operation/ChainTablePartitionExpireTest.java | 78 +
.../paimon/operation/ExpireSnapshotsTest.java | 44 +
.../FileStoreScanPartitionBucketEntryTest.java | 32 +
.../operation/RemoveUnexistingManifestsTest.java | 213 +++
.../operation/commit/ConflictDetectionTest.java | 275 ++++
.../apache/paimon/rest/MockRESTCatalogTest.java | 82 +
.../apache/paimon/schema/SchemaManagerTest.java | 125 ++
.../paimon/schema/SchemaMergingUtilsTest.java | 103 ++
.../paimon/sort/BinaryExternalSortBufferTest.java | 9 +
.../table/AppendOnlyLimitManifestReadTest.java | 225 +++
.../paimon/table/AppendOnlySimpleTableTest.java | 111 ++
.../paimon/table/BitmapGlobalIndexTableTest.java | 5 +-
.../paimon/table/BtreeGlobalIndexTableTest.java | 6 +-
.../paimon/table/ChainTableFileStoreTableTest.java | 429 +++++
.../paimon/table/ChainTableSnapshotExpireTest.java | 281 ++++
.../apache/paimon/table/ChangelogExpireTest.java | 86 +-
.../paimon/table/CompositeBTreeTableTest.java | 1076 +++++++++++++
.../table/DataEvolutionDeletionVectorTest.java | 122 +-
.../paimon/table/DataEvolutionFileIndexTest.java | 204 ++-
.../paimon/table/DataEvolutionTableTest.java | 79 +-
.../table/ExpireSnapshotsInterruptStatusTest.java | 99 ++
.../apache/paimon/table/IndexQuerySplitTest.java | 1139 +++++++++++++
.../table/MultiValueGlobalIndexTableTest.java | 3 +-
.../paimon/table/NestedFieldQueryAuthTest.java | 157 ++
.../NestedSubfieldDataEvolutionTableTest.java | 48 +
.../paimon/table/PartitionLayoutScanTest.java | 124 ++
.../paimon/table/format/FormatReadBuilderTest.java | 7 +
.../table/sink/CommitMessageSerializerTest.java | 49 +-
.../apache/paimon/table/sink/TableCommitTest.java | 164 ++
.../apache/paimon/table/sink/TableWriteTest.java | 210 +++
.../table/source/AbstractDataTableReadTest.java | 16 +-
.../table/source/DataEvolutionTableReadTest.java | 4 +-
.../table/source/DataSplitCompatibleTest.java | 4 +-
.../table/source/FullTextSearchBuilderTest.java | 181 ++-
.../source/PrimaryKeySortedIndexBatchScanTest.java | 5 +-
.../source/PrimaryKeySortedIndexReadTest.java | 108 ++
.../paimon/table/source/ReadBuilderImplTest.java | 83 +
.../org/apache/paimon/table/source/SplitsTest.java | 167 ++
.../apache/paimon/table/source/TableScanTest.java | 200 +++
.../table/source/VectorSearchBuilderTest.java | 48 +-
.../source/VectorSearchRowFilterExactnessTest.java | 601 +++++++
.../paimon/table/system/SchemasTableTest.java | 69 +-
.../paimon/table/system/SnapshotsTableTest.java | 48 +
.../java/org/apache/paimon/tag/TagPreviewTest.java | 44 +
.../paimon/utils/DataEvolutionUtilsTest.java | 12 +-
.../paimon/utils/ObjectSerializerTestBase.java | 39 +
.../compatibility/manifest-committable-v14-v5 | Bin 0 -> 3155 bytes
.../configuration/ConfigOptionsDocGenerator.java | 3 +-
.../paimon/eslib/index/ESIndexGlobalIndexer.java | 2 +
.../eslib/index/ESIndexGlobalIndexerFactory.java | 18 +-
.../index/ESIndexGlobalIndexerExecutorTest.java | 6 +-
.../eslib/index/ESIndexVectorMetricTest.java | 15 +-
.../java/org/apache/paimon/cosn/COSNFileIO.java | 11 +-
.../java/org/apache/paimon/cosn/COSNLoader.java | 4 +-
.../org/apache/paimon/cosn/COSNLoaderTest.java | 69 +
paimon-filesystems/paimon-jindo/pom.xml | 13 +
.../apache/paimon/jindo/JindoBuildVersions.java | 11 +-
.../java/org/apache/paimon/jindo/JindoFileIO.java | 27 +-
.../org/apache/paimon/jindo/JindoUserAgent.java | 101 ++
.../org/apache/paimon/jindo/JindoFileIOTest.java | 2 +
.../apache/paimon/jindo/TestJindoUserAgent.java | 113 ++
.../main/java/org/apache/paimon/obs/OBSFileIO.java | 11 +-
paimon-filesystems/paimon-oss-impl/pom.xml | 13 +
.../org/apache/paimon/oss/OSSBuildVersions.java | 14 +-
.../org/apache/paimon/oss/OSSBlobPresigner.java | 31 +-
.../main/java/org/apache/paimon/oss/OSSFileIO.java | 32 +-
.../org/apache/paimon/oss/OSSRetryStrategy.java | 75 +
.../java/org/apache/paimon/oss/OSSUserAgent.java | 96 ++
.../java/org/apache/paimon/oss/OSSFileIOTest.java | 116 +-
.../apache/paimon/oss/OSSRetryStrategyTest.java | 263 +++
.../org/apache/paimon/oss/OSSUserAgentTest.java | 168 ++
.../apache/paimon/s3/HadoopCompliantFileIO.java | 18 +-
.../main/java/org/apache/paimon/s3/S3FileIO.java | 1 +
.../org/apache/paimon/s3/S3FileIOConfigTest.java | 81 +
.../apache/paimon/s3/S3FileIOSchemeCacheTest.java | 92 ++
.../procedure/RemoveOrphanBlobsProcedure.java | 132 ++
.../org/apache/paimon/flink/AppendTableITCase.java | 2 +-
.../flink/RemoveOrphanBlobsActionITCase.java | 14 +-
.../org/apache/paimon/flink/AppendTableITCase.java | 2 +-
.../flink/RemoveOrphanBlobsActionITCase.java | 11 +-
.../flink/action/cdc/CdcActionCommonUtils.java | 3 +-
.../apache/paimon/flink/action/cdc/Expression.java | 7 +-
.../flink/action/cdc/SyncDatabaseActionBase.java | 11 +-
.../flink/action/cdc/TableNameConverter.java | 15 +-
.../paimon/flink/action/cdc/TypeMapping.java | 5 +-
.../flink/action/cdc/format/DataFormatFactory.java | 4 +-
.../cdc/format/aliyun/AliyunFieldParser.java | 5 +
.../action/cdc/format/canal/CanalFieldParser.java | 5 +
.../action/cdc/format/canal/CanalRecordParser.java | 18 +-
.../flink/action/cdc/kafka/KafkaActionUtils.java | 2 +-
.../action/cdc/mongodb/MongoDBActionUtils.java | 3 +-
.../action/cdc/mongodb/MongoDBSchemaUtils.java | 4 +-
.../cdc/mongodb/strategy/MongoVersionStrategy.java | 4 +-
.../flink/action/cdc/mysql/MySqlTypeUtils.java | 11 +-
.../action/cdc/postgres/PostgresTypeUtils.java | 4 +-
.../apache/paimon/flink/sink/cdc/CdcRecord.java | 6 +-
.../action/cdc/TurkishLocaleCaseFoldingTest.java | 142 ++
.../cdc/format/aliyun/AliyunFieldParserTest.java | 24 +-
.../cdc/format/canal/CanalFieldParserTest.java | 24 +-
.../cdc/format/canal/CanalRecordParserTest.java | 84 +
.../cdc/mysql/TurkishLocaleTypeNameTest.java | 60 +
.../paimon/flink/sink/cdc/CdcRecordTest.java | 53 +
paimon-flink/paimon-flink-common/pom.xml | 7 +
.../apache/paimon/flink/FlinkConnectorOptions.java | 38 +
.../apache/paimon/flink/FormatCatalogTable.java | 3 +-
.../apache/paimon/flink/NestedFieldReferences.java | 85 +
.../apache/paimon/flink/PredicateConverter.java | 475 +++++-
.../apache/paimon/flink/action/ActionFactory.java | 3 +-
.../apache/paimon/flink/action/CloneAction.java | 3 +-
.../paimon/flink/action/CompactDatabaseAction.java | 34 +-
.../flink/action/DataEvolutionMergeIntoAction.java | 7 +-
.../paimon/flink/action/MultiTablesSinkMode.java | 5 +-
.../flink/action/RemoveOrphanBlobsAction.java | 88 +
.../action/RemoveOrphanBlobsActionFactory.java | 98 ++
.../action/RemoveUnexistingManifestsAction.java | 74 +-
.../paimon/flink/clone/CloneFileFormatUtils.java | 4 +-
.../dataevolution/DataEvolutionDeleteOperator.java | 21 +-
.../dataevolution/DataEvolutionDeleteSink.java | 3 +-
.../DataEvolutionPartialWriteOperator.java | 17 +-
.../dataevolution/MergeIntoUpdateChecker.java | 17 +-
.../flink/globalindex/GenericIndexTopoBuilder.java | 4 +-
.../flink/globalindex/SortedIndexTopoBuilder.java | 108 +-
.../paimon/flink/lookup/PartitionLoader.java | 3 +-
.../orphan/FlinkManagedBlobOrphanFilesClean.java | 965 +++++++++++
.../paimon/flink/orphan/FlinkOrphanFilesClean.java | 4 +
.../procedure/CreateGlobalIndexProcedure.java | 14 +-
.../flink/procedure/DropGlobalIndexProcedure.java | 3 +-
.../flink/procedure/ImportFilesProcedure.java | 50 +
.../procedure/RemoveOrphanBlobsProcedure.java | 120 ++
.../procedure/RollbackToAsLatestProcedure.java | 40 +-
.../DataEvolutionCommitPreparationOperator.java | 29 +-
...DataEvolutionDeletionVectorMaterializeSink.java | 5 +-
.../source/ContinuousFileSplitEnumerator.java | 14 +-
.../source/DynamicPartitionFilteringInfo.java | 29 +
.../flink/source/FileStoreSourceSplitReader.java | 6 +-
.../paimon/flink/source/FlinkSourceBuilder.java | 4 +-
.../paimon/flink/source/FlinkTableSource.java | 6 +-
.../paimon/flink/source/SplitWeightUtils.java | 82 +
.../source/StaticFileStoreSplitEnumerator.java | 4 +-
.../paimon/flink/source/SystemTableSource.java | 39 +
.../flink/source/aggregate/LocalAggregator.java | 3 +-
.../AlignedContinuousFileSplitEnumerator.java | 3 +-
.../source/assigners/AlignedSplitAssigner.java | 18 +-
.../assigners/DynamicPartitionPruningAssigner.java | 20 +-
.../source/assigners/PreAssignSplitAssigner.java | 74 +-
.../flink/source/operator/MonitorSource.java | 11 +-
.../paimon/flink/source/operator/ReadOperator.java | 16 +-
.../apache/paimon/flink/utils/TableScanUtils.java | 8 +-
.../services/org.apache.paimon.factories.Factory | 3 +
.../org/apache/paimon/flink/AppendTableITCase.java | 4 +-
.../apache/paimon/flink/CatalogTableITCase.java | 15 +-
.../org/apache/paimon/flink/FileStoreITCase.java | 73 +
.../paimon/flink/NestedFieldReferencesTest.java | 109 ++
.../paimon/flink/NestedPredicateConverterTest.java | 499 ++++++
.../paimon/flink/PredicateConverterTest.java | 454 +++++-
.../apache/paimon/flink/ReadWriteTableITCase.java | 92 ++
.../paimon/flink/SortedGlobalIndexITCase.java | 267 ++++
.../paimon/flink/action/ActionJobCoverageTest.java | 5 +-
.../flink/action/CompactDatabaseActionITCase.java | 214 +++
.../flink/action/MultiTablesSinkModeTest.java | 63 +
.../NestedSubfieldMergeIntoActionITCase.java | 22 +
.../action/RemoveOrphanBlobsActionITCase.java | 10 +-
.../action/RemoveOrphanBlobsActionITCaseBase.java | 935 +++++++++++
.../action/RemoveOrphanFilesActionITCaseBase.java | 24 +
.../ChangelogCompactTaskSerializerTest.java | 2 +-
.../CompactedChangelogFormatReaderFactoryTest.java | 117 ++
.../globalindex/SortedIndexTopoBuilderTest.java | 6 +-
.../flink/iceberg/FlinkIcebergITCaseBase.java | 13 +
.../procedure/CreateGlobalIndexProcedureTest.java | 4 +-
.../procedure/ImportFilesProcedureITCase.java | 102 ++
.../RollbackToAsLatestProcedureITCase.java | 120 +-
.../procedure/VectorSearchProcedureITCase.java | 15 +-
.../flink/sink/CommittableSerializerTest.java | 4 +-
.../source/ContinuousFileSplitEnumeratorTest.java | 53 +
.../paimon/flink/source/DataTableSourceTest.java | 97 ++
.../source/FileStoreSourceSplitReaderTest.java | 48 +-
.../flink/source/FlinkSourceBuilderTest.java | 357 +++++
.../paimon/flink/source/FlinkTableSourceTest.java | 235 +++
.../paimon/flink/source/IndexQuerySourceTest.java | 267 ++++
.../source/NestedFieldFilterPushDownITCase.java | 392 +++++
.../PendingSplitsCheckpointSerializerTest.java | 18 +-
...ContinuousFileSplitEnumeratorQueryAuthTest.java | 145 ++
.../source/assigners/AlignedSplitAssignerTest.java | 245 +++
.../DynamicPartitionPruningAssignerTest.java | 262 +++
.../assigners/PreAssignSplitAssignerTest.java | 233 +++
.../operator/ReadOperatorSplitShapeTest.java | 230 +++
.../paimon/flink/utils/TableScanUtilsTest.java | 116 ++
.../FlinkDataEvolutionVectorReadTest.java | 1 +
.../src/main/java/org/apache/orc/OrcFile.java | 5 +-
.../apache/paimon/format/avro/AvroBulkFormat.java | 2 +-
.../format/blob/AbstractBlobElementWriter.java | 34 +-
.../format/blob/MapBlobElementSerializer.java | 22 +-
.../apache/paimon/format/blob/VideoFileMeta.java | 60 +-
.../paimon/format/blob/VideoFormatReader.java | 6 +-
.../paimon/format/blob/VideoFormatWriter.java | 77 +-
.../paimon/format/blob/VideoKeyframeIndex.java | 138 ++
.../apache/paimon/format/csv/CsvFormatWriter.java | 51 +-
.../org/apache/paimon/format/csv/CsvOptions.java | 2 +
.../org/apache/paimon/format/csv/CsvParser.java | 13 +-
.../apache/paimon/format/json/JsonFileReader.java | 31 +-
.../paimon/format/json/JsonFormatWriter.java | 41 +-
.../paimon/format/json/JsonMapKeyConverter.java | 70 +
.../org/apache/paimon/format/json/JsonOptions.java | 1 +
.../orc/filter/OrcPredicateFunctionVisitor.java | 7 +
.../format/parquet/ParquetListLayoutResolver.java | 226 +++
.../format/parquet/ParquetReaderFactory.java | 109 +-
.../format/parquet/ParquetSchemaConverter.java | 346 ++--
.../parquet/VariantShreddingReadPlanFactory.java | 6 +-
.../format/parquet/VariantShreddingTypePruner.java | 51 +-
.../format/parquet/reader/ParquetReaderUtil.java | 112 +-
.../apache/paimon/format/row/RowBlockIndex.java | 87 +
.../apache/paimon/format/row/RowFileFooter.java | 35 +-
.../paimon/format/row/RowFormatReaderFactory.java | 9 +
.../parquet/filter2/predicate/ParquetFilters.java | 203 ++-
.../paimon/format/avro/AvroFileFormatTest.java | 20 +-
.../paimon/format/blob/BlobFileFormatTest.java | 60 +
.../paimon/format/blob/VideoFileFormatTest.java | 419 ++++-
.../paimon/format/csv/CsvFileFormatTest.java | 68 +
.../paimon/format/json/JsonFileFormatTest.java | 285 +++-
.../paimon/format/orc/OrcFormatReadWriteTest.java | 47 +
.../format/orc/filter/OrcFilterConverterTest.java | 43 +
.../paimon/format/parquet/ParquetFiltersTest.java | 434 +++++
.../format/parquet/ParquetFormatReadWriteTest.java | 334 ++++
.../format/parquet/ParquetLegacyListReadTest.java | 595 +++++++
.../parquet/ParquetListLayoutResolverTest.java | 191 +++
.../format/parquet/ParquetReadWriteTest.java | 63 +
.../format/parquet/ParquetSchemaConverterTest.java | 244 +++
.../format/row/RowFileIndexConsistencyTest.java | 217 +++
.../org/apache/paimon/format/blob/video-v1.hex | 2 +-
.../index/NativeFullTextGlobalIndexer.java | 2 +
.../index/NativeFullTextGlobalIndexerFactory.java | 8 +-
.../index/JavaPyNativeFullTextE2ETest.java | 2 +-
.../index/NativeFullTextGlobalIndexTest.java | 3 +-
.../NativeFullTextGlobalIndexerFactoryTest.java | 4 +-
.../index/NativeFullTextRowFilterTest.java | 4 +-
.../index/NativePrimaryKeyFullTextIndexTest.java | 7 +-
.../java/org/apache/paimon/hive/HiveCatalog.java | 7 +-
.../org/apache/paimon/hive/HiveTableUtils.java | 5 +-
.../apache/paimon/hive/clone/HiveCloneUtils.java | 3 +-
.../paimon/hive/clone/HiveTableCloneExtractor.java | 16 +-
.../apache/paimon/hive/pool/CachedClientPool.java | 2 +-
.../hive/clone/HiveTableCloneExtractorTest.java | 69 +
.../paimon/hive/pool/TestCachedClientPool.java | 18 +
.../java/org/apache/paimon/hive/HiveSchema.java | 5 +-
.../org/apache/paimon/hive/PaimonMetaHook.java | 11 +-
.../hive/SearchArgumentToPredicateConverter.java | 9 +-
.../paimon/hive/mapred/PaimonRecordReader.java | 3 +-
.../paimon/hive/FlinkGenericCatalogITCase.java | 15 +-
.../core/IcebergRowLineageCompatibilityTest.java | 16 +-
.../paimon/format/lance/LanceRecordsWriter.java | 55 +-
.../format/lance/LanceRecordsWriterTest.java | 77 +
.../index/LuminaVectorGlobalIndexWriter.java | 2 +-
.../lumina/index/LuminaVectorGlobalIndexer.java | 2 +
.../index/LuminaVectorGlobalIndexerFactory.java | 9 +-
.../lumina/index/LuminaVectorIndexOptions.java | 33 +-
.../paimon/lumina/index/LuminaVectorMetric.java | 4 +-
.../paimon/lumina/index/JavaPyLuminaE2ETest.java | 9 +-
.../lumina/index/LuminaVectorMetricTest.java | 56 +
.../lumina/index/LuminaVectorOptionsTest.java | 29 +
paimon-python/MANIFEST.in | 2 +-
paimon-python/README.md | 344 +++-
paimon-python/conftest.py | 247 ++-
paimon-python/dev/check-licensing.sh | 2 +-
paimon-python/dev/install.py | 71 +
paimon-python/dev/lint-python.sh | 11 +-
paimon-python/dev/requirements-dev.txt | 41 -
paimon-python/dev/requirements.txt | 44 -
paimon-python/dev/run_mixed_tests.sh | 29 +-
paimon-python/dev/test_packaging.py | 212 +++
.../{MANIFEST.in => pypaimon/_version.py} | 14 +-
paimon-python/pypaimon/api/api_request.py | 78 +-
paimon-python/pypaimon/api/api_response.py | 37 +
paimon-python/pypaimon/api/auth/dlf_signer.py | 6 +-
paimon-python/pypaimon/api/client.py | 17 +-
paimon-python/pypaimon/api/resource_paths.py | 15 +
paimon-python/pypaimon/api/rest_api.py | 55 +-
.../pypaimon/api/rest_permission_management.py | 38 +
paimon-python/pypaimon/api/rest_util.py | 10 +-
paimon-python/pypaimon/api/token_loader.py | 6 +-
paimon-python/pypaimon/benchmark/act/README.md | 104 ++
paimon-python/pypaimon/benchmark/act/__main__.py | 7 +
paimon-python/pypaimon/benchmark/act/hdf5.py | 1 +
paimon-python/pypaimon/benchmark/act/paimon.py | 83 +-
.../{sample => benchmark/act}/robomind_agilex.py | 322 +++-
paimon-python/pypaimon/benchmark/act/runner.py | 81 +-
.../pypaimon/benchmark/video_random_read.py | 432 +++++
paimon-python/pypaimon/branch/branch_manager.py | 5 +-
.../pypaimon/branch/filesystem_branch_manager.py | 4 +-
paimon-python/pypaimon/build_info.py | 56 +-
paimon-python/pypaimon/catalog/catalog.py | 44 +-
paimon-python/pypaimon/catalog/catalog_context.py | 39 +-
.../pypaimon/catalog/catalog_environment.py | 17 +-
paimon-python/pypaimon/catalog/catalog_factory.py | 2 +-
paimon-python/pypaimon/catalog/catalog_utils.py | 121 ++
.../pypaimon/catalog/filesystem_catalog.py | 130 +-
.../pypaimon/catalog/filesystem_catalog_loader.py | 2 +-
paimon-python/pypaimon/catalog/jdbc_catalog.py | 28 +-
.../pypaimon/catalog/rest/rest_catalog.py | 80 +-
.../pypaimon/catalog/rest/rest_token_file_io.py | 71 +-
.../pypaimon/catalog/rest/table_metadata.py | 4 +-
paimon-python/pypaimon/catalog/table_query_auth.py | 289 +++-
paimon-python/pypaimon/cli/cli_table.py | 78 +-
paimon-python/pypaimon/common/blob_index_cache.py | 93 ++
.../pypaimon/common/decimal_arrow_predicate.py | 142 ++
.../pypaimon/common/external_path_provider.py | 10 +-
paimon-python/pypaimon/common/file_io.py | 68 +-
paimon-python/pypaimon/common/json_util.py | 15 +
paimon-python/pypaimon/common/options/config.py | 19 +-
.../pypaimon/common/options/core_options.py | 180 ++-
paimon-python/pypaimon/common/predicate.py | 53 +-
.../pypaimon/common/predicate_json_parser.py | 187 ++-
paimon-python/pypaimon/common/uri_reader.py | 119 +-
paimon-python/pypaimon/common/user_agent.py | 64 +
paimon-python/pypaimon/common/where_parser.py | 83 +-
paimon-python/pypaimon/daft/daft_catalog.py | 24 +-
paimon-python/pypaimon/daft/daft_datasink.py | 53 +-
paimon-python/pypaimon/daft/daft_datasource.py | 18 +-
paimon-python/pypaimon/daft/daft_paimon.py | 13 -
paimon-python/pypaimon/data/__init__.py | 3 +
.../pypaimon/data/map_shared_shredding.py | 295 +++-
paimon-python/pypaimon/data/variant_path.py | 256 ++-
paimon-python/pypaimon/data/variant_shredding.py | 25 +-
.../pypaimon/filesystem/caching_file_io.py | 18 +-
.../pypaimon/filesystem/hdfs_native_file_io.py | 16 +
.../filesystem/jindo_file_system_handler.py | 52 +-
paimon-python/pypaimon/filesystem/local_file_io.py | 25 +-
.../pypaimon/filesystem/oss_blob_presigner.py | 250 +++
paimon-python/pypaimon/filesystem/oss_file_io.py | 58 +-
.../pypaimon/filesystem/oss_user_agent.py | 77 +
paimon-python/pypaimon/filesystem/pvfs.py | 3 +-
.../pypaimon/filesystem/pyarrow_file_io.py | 3 +
.../pypaimon/filesystem/resolving_file_io.py | 4 +
.../globalindex/btree/btree_index_reader.py | 8 +-
paimon-python/pypaimon/globalindex/build_plan.py | 3 +
.../pypaimon/globalindex/create_global_index.py | 28 +-
.../data_evolution_global_index_scanner.py | 8 +-
.../pypaimon/globalindex/global_index_evaluator.py | 11 +-
.../pypaimon/globalindex/global_index_result.py | 22 +-
.../pypaimon/globalindex/indexed_split.py | 18 +-
.../pypaimon/globalindex/key_serializer.py | 38 +-
.../vindex/vindex_vector_global_index_reader.py | 2 +-
.../vindex/vindex_vector_index_writer.py | 120 +-
paimon-python/pypaimon/index/dynamic_bucket.py | 83 +-
.../pypaimon/{api => management}/__init__.py | 0
.../pypaimon/management/java_string.py | 11 +-
.../management/list_permissions_request.py | 98 ++
.../pypaimon/management/permission_access.py | 84 +
.../pypaimon/management/permission_assignment.py | 170 ++
.../pypaimon/management/permission_columns.py | 76 +
.../permission_management.py} | 29 +-
.../pypaimon/management/permission_resource.py | 120 ++
.../resource_type.py} | 40 +-
.../pypaimon/manifest/index_manifest_file.py | 50 +-
.../pypaimon/manifest/manifest_file_manager.py | 31 +-
.../pypaimon/manifest/manifest_file_merger.py | 100 --
.../pypaimon/manifest/manifest_list_manager.py | 27 +-
.../pypaimon/manifest/manifest_sidecar.py | 5 +-
.../pypaimon/manifest/schema/data_file_meta.py | 41 +-
paimon-python/pypaimon/multimodal/blob_read.py | 23 +-
paimon-python/pypaimon/multimodal/blob_store.py | 57 +-
.../pypaimon/multimodal/lerobot/dataset.py | 509 +++++-
.../pypaimon/multimodal/lerobot/loader.py | 2 +
.../pypaimon/multimodal/lerobot/metadata.py | 5 +-
.../pypaimon/multimodal/lerobot/writer.py | 4 +-
paimon-python/pypaimon/multimodal/query.py | 231 ++-
paimon-python/pypaimon/multimodal/table.py | 95 +-
paimon-python/pypaimon/multimodal/temporal.py | 86 +-
paimon-python/pypaimon/multimodal/video.py | 148 +-
.../pypaimon/multimodal/window_dataset.py | 13 +-
.../pypaimon/multimodal/window_transforms.py | 10 +-
paimon-python/pypaimon/ray/batch_vector_search.py | 37 +-
.../pypaimon/ray/data_evolution_merge_into.py | 40 +
.../pypaimon/ray/data_evolution_merge_join.py | 112 +-
paimon-python/pypaimon/ray/full_text_search.py | 105 ++
paimon-python/pypaimon/ray/ray_paimon.py | 300 +++-
paimon-python/pypaimon/ray/read_by_row_id.py | 26 +-
paimon-python/pypaimon/ray/shuffle.py | 79 +-
paimon-python/pypaimon/ray/update_by_row_id.py | 11 +-
paimon-python/pypaimon/ray/vector_search.py | 28 +-
.../pypaimon/read/datasource/ray_datasource.py | 16 +-
.../pypaimon/read/datasource/split_provider.py | 31 +-
.../pypaimon/read/datasource/torch_dataset.py | 200 ++-
.../pypaimon/read/merge_engine_support.py | 25 +-
paimon-python/pypaimon/read/native_plan.py | 305 ++--
paimon-python/pypaimon/read/read_builder.py | 394 +++--
paimon-python/pypaimon/read/read_type.py | 214 +++
.../pypaimon/read/reader/auth_masking_reader.py | 128 +-
.../read/reader/blob_descriptor_convert_reader.py | 135 +-
.../pypaimon/read/reader/blob_view_read_support.py | 68 +
.../pypaimon/read/reader/concat_batch_reader.py | 20 +-
.../pypaimon/read/reader/data_file_batch_reader.py | 55 +-
.../read/reader/deferred_blob_resolve_reader.py | 5 +
.../pypaimon/read/reader/field_indices.py | 25 +
.../read/reader/filter_record_batch_reader.py | 5 +-
.../pypaimon/read/reader/filter_record_reader.py | 2 +
.../pypaimon/read/reader/format_avro_reader.py | 41 +-
.../pypaimon/read/reader/format_blob_reader.py | 24 +-
.../pypaimon/read/reader/format_pyarrow_reader.py | 123 +-
.../pypaimon/read/reader/format_row_reader.py | 112 +-
.../read/reader/iface/record_batch_reader.py | 67 +-
.../pypaimon/read/reader/iface/record_reader.py | 15 +
.../pypaimon/read/reader/limited_record_reader.py | 3 +
.../read/reader/nested_leaf_batch_reader.py | 24 +-
.../read/reader/outer_projection_record_reader.py | 35 +-
.../read/reader/parquet_page_index_reader.py | 731 +++++++++
.../read/reader/row_range_filter_record_reader.py | 1 +
.../pypaimon/read/reader/shard_batch_reader.py | 13 +-
.../pypaimon/read/reader/sort_merge_reader.py | 24 +-
.../pypaimon/read/reader/video_format_reader.py | 71 +-
paimon-python/pypaimon/read/scan_distribution.py | 16 +-
.../read/scanner/chunk_shuffle_split_generator.py | 211 +--
.../read/scanner/data_evolution_split_generator.py | 17 +-
.../pypaimon/read/scanner/file_scanner.py | 16 +-
.../scanner/primary_key_table_split_generator.py | 27 +-
.../pypaimon/read/scanner/split_generator.py | 30 +-
paimon-python/pypaimon/read/sliced_split.py | 15 +-
paimon-python/pypaimon/read/split.py | 12 +-
paimon-python/pypaimon/read/split_read.py | 374 ++++-
paimon-python/pypaimon/read/split_serializer.py | 450 +++++-
paimon-python/pypaimon/read/stream_read_builder.py | 48 +-
.../pypaimon/read/streaming_table_scan.py | 87 +-
paimon-python/pypaimon/read/table_read.py | 792 +++++++--
paimon-python/pypaimon/read/table_scan.py | 454 +++---
paimon-python/pypaimon/read/variant_read_type.py | 92 ++
paimon-python/pypaimon/schema/arrow_schema.py | 92 ++
paimon-python/pypaimon/schema/data_types.py | 13 +-
.../pypaimon/table/data_evolution_merge_into.py | 79 +-
paimon-python/pypaimon/table/file_store_table.py | 38 +-
.../pypaimon/table/format/format_table_write.py | 12 +-
paimon-python/pypaimon/table/row/blob.py | 64 +-
paimon-python/pypaimon/table/row/offset_row.py | 56 +-
.../pypaimon/table/row/video_keyframe_index.py | 348 ++++
.../table/source/batch_vector_search_builder.py | 2 +
.../pypaimon/table/source/full_text_read.py | 49 +-
.../pypaimon/table/source/full_text_scan.py | 23 +-
.../table/source/full_text_search_builder.py | 26 +
.../table/source/global_index_row_filter.py | 42 +
.../pypaimon/table/source/hybrid_search_builder.py | 11 +-
.../table/source/primary_key_scored_result.py | 9 +-
.../pypaimon/table/source/vector_search_builder.py | 3 +
.../pypaimon/table/source/vector_search_read.py | 89 +-
.../pypaimon/table/system/consumers_table.py | 63 +
.../pypaimon/table/system/file_key_ranges_table.py | 7 +-
paimon-python/pypaimon/table/system/files_table.py | 45 +-
.../pypaimon/table/system/system_table_loader.py | 5 +-
.../pypaimon/table/system/system_table_read.py | 70 +-
paimon-python/pypaimon/tests/act_runner_test.py | 117 +-
.../pypaimon/tests/api/test_branch_dto_serde.py | 4 +-
.../pypaimon/tests/api/test_partition_dto_serde.py | 2 +-
.../pypaimon/tests/api/test_tag_dto_serde.py | 4 +-
paimon-python/pypaimon/tests/arrow_schema_test.py | 173 ++
.../pypaimon/tests/batch_vector_lookup_test.py | 3 +
.../pypaimon/tests/batch_vector_raw_scan_test.py | 4 +
paimon-python/pypaimon/tests/binary_row_test.py | 22 +-
paimon-python/pypaimon/tests/blob_table_test.py | 521 +++++-
paimon-python/pypaimon/tests/blob_test.py | 1676 +++++++++++++++++++-
.../pypaimon/tests/branch_manager_test.py | 4 +-
paimon-python/pypaimon/tests/build_info_test.py | 7 +-
.../pypaimon/tests/catalog_environment_test.py | 35 +
paimon-python/pypaimon/tests/cli_table_test.py | 51 +
.../tests/commit_message_serializer_test.py | 113 ++
.../tests/composite_btree_fallback_test.py | 182 +++
.../pypaimon/tests/concat_batch_reader_test.py | 65 +
.../tests/contiguous_window_dataset_test.py | 54 +
.../pypaimon/tests/daft/daft_blob_auth_test.py | 175 ++
.../pypaimon/tests/daft/daft_integration_test.py | 3 +-
.../tests/daft/daft_pk_distributed_write_test.py | 57 +
.../pypaimon/tests/daft/daft_sink_test.py | 40 +
.../tests/data_evolution_deletion_vector_test.py | 78 +-
.../pypaimon/tests/data_evolution_formats_test.py | 33 +-
.../tests/data_evolution_group_stats_test.py | 9 +
.../tests/data_evolution_row_rolling_test.py | 32 +-
.../tests/data_evolution_split_generator_test.py | 13 +-
.../tests/data_file_path_directory_test.py | 254 +++
paimon-python/pypaimon/tests/data_types_test.py | 122 +-
.../pypaimon/tests/decimal_key_serializer_test.py | 120 ++
.../pypaimon/tests/decimal_predicate_test.py | 216 +++
.../pypaimon/tests/deferred_blob_resolve_test.py | 29 +-
.../pypaimon/tests/deletion_vector_path_test.py | 134 +-
.../pypaimon/tests/e2e/java_py_read_write_test.py | 49 +
.../pypaimon/tests/external_paths_test.py | 10 +-
.../pypaimon/tests/file_store_commit_test.py | 450 ++----
paimon-python/pypaimon/tests/file_type_test.py | 77 +
.../pypaimon/tests/filesystem_catalog_test.py | 18 +
.../format_pyarrow_shared_shredding_map_test.py | 195 ++-
.../tests/full_text_reader_cleanup_test.py | 64 +
.../pypaimon/tests/full_text_scalar_filter_test.py | 179 +++
.../pypaimon/tests/global_index_build_test.py | 86 +-
.../pypaimon/tests/hdfs_native_write_row_test.py | 74 +
.../pypaimon/tests/interval_partition_test.py | 11 +-
.../pypaimon/tests/jindo_file_system_test.py | 118 +-
.../manifest/manifest_entry_identifier_test.py | 47 -
.../tests/manifest/manifest_manager_test.py | 109 +-
.../tests/manifest/manifest_sidecar_test.py | 15 +-
.../tests/map_selected_key_projection_test.py | 85 +
.../tests/map_shared_shredding_write_test.py | 91 +-
.../pypaimon/tests/mosaic_writer_options_test.py | 15 +-
.../pypaimon/tests/multimodal_lerobot_test.py | 521 +++++-
.../pypaimon/tests/multimodal_table_test.py | 729 ++++++++-
.../pypaimon/tests/multimodal_temporal_test.py | 190 ++-
.../pypaimon/tests/multimodal_video_test.py | 99 +-
.../tests/native_blob_collection_write_test.py | 354 +++++
.../pypaimon/tests/native_blob_consumer_test.py | 287 ++++
.../pypaimon/tests/native_blob_update_test.py | 330 ++++
.../pypaimon/tests/native_blob_uri_reader_test.py | 350 ++++
.../pypaimon/tests/native_blob_write_test.py | 476 ++++++
.../pypaimon/tests/native_branch_write_test.py | 481 ++++++
paimon-python/pypaimon/tests/native_commit_test.py | 605 +++++++
.../tests/native_cross_partition_write_test.py | 193 +++
.../pypaimon/tests/native_data_directory_test.py | 226 +++
.../tests/native_data_evolution_commit_test.py | 394 +++++
.../pypaimon/tests/native_dynamic_write_test.py | 280 ++++
.../pypaimon/tests/native_external_write_test.py | 225 +++
.../tests/native_managed_blob_write_test.py | 165 ++
.../tests/native_map_shredding_write_test.py | 327 ++++
.../tests/native_merge_incremental_test.py | 192 +++
.../pypaimon/tests/native_merge_into_test.py | 532 +++++++
.../pypaimon/tests/native_merge_read_test.py | 147 ++
.../pypaimon/tests/native_nested_update_test.py | 301 ++++
.../pypaimon/tests/native_pk_blob_limit_test.py | 283 ++++
.../tests/native_plan_capabilities_test.py | 18 +-
.../tests/native_plan_chunk_shuffle_test.py | 92 +-
.../tests/native_plan_distribution_test.py | 143 +-
.../tests/native_plan_dynamic_bucket_test.py | 47 +-
.../pypaimon/tests/native_plan_expanded_test.py | 16 +-
.../pypaimon/tests/native_plan_incremental_test.py | 285 +++-
.../pypaimon/tests/native_plan_integration_test.py | 773 ++++++++-
.../tests/native_plan_materialized_pk_test.py | 120 +-
.../tests/native_plan_resolved_schema_test.py | 21 +-
.../pypaimon/tests/native_plan_rest_test.py | 156 +-
paimon-python/pypaimon/tests/native_plan_test.py | 874 ++++++----
.../tests/native_postpone_fixed_write_test.py | 606 +++++++
.../pypaimon/tests/native_postpone_write_test.py | 228 +++
.../tests/native_precomputed_bucket_test.py | 421 +++++
paimon-python/pypaimon/tests/native_read_test.py | 931 ++++++++++-
.../pypaimon/tests/native_read_type_test.py | 495 ++++++
.../pypaimon/tests/native_row_sidecar_test.py | 421 +++++
.../pypaimon/tests/native_sequence_test.py | 122 ++
.../pypaimon/tests/native_system_columns_test.py | 197 +++
paimon-python/pypaimon/tests/native_update_test.py | 1031 ++++++++++++
.../tests/native_upsert_capabilities_test.py | 221 +++
.../tests/native_upsert_row_shapes_test.py | 184 +++
.../pypaimon/tests/native_variant_update_test.py | 195 +++
.../pypaimon/tests/native_video_read_test.py | 194 +++
.../tests/native_write_capabilities_test.py | 285 ++++
paimon-python/pypaimon/tests/native_write_test.py | 563 +++++++
.../tests/normal_map_selected_keys_test.py | 111 ++
.../pypaimon/tests/oss_blob_presigner_test.py | 530 +++++++
.../pypaimon/tests/oss_user_agent_test.py | 150 ++
.../pypaimon/tests/parquet_metadata_cache_test.py | 53 +
.../pypaimon/tests/parquet_page_index_test.py | 682 ++++++++
.../tests/parquet_write_page_index_test.py | 253 +++
.../tests/permission_management_json_test.py | 303 ++++
.../pypaimon/tests/polars_map_null_test.py | 73 +
.../pypaimon/tests/predicate_json_parser_test.py | 157 ++
.../tests/primary_key_index_definitions_test.py | 1 +
.../pypaimon/tests/py36/ao_simple_test.py | 6 +-
.../tests/py36/named_variant_projection_test.py | 54 +
.../pypaimon/tests/py36/where_literal_cast_test.py | 63 +
.../pypaimon/tests/query_auth_validation_test.py | 968 +++++++++++
.../pypaimon/tests/query_auth_widening_test.py | 191 +++
.../tests/ray_batch_vector_refinement_test.py | 145 ++
.../pypaimon/tests/ray_batch_vector_search_test.py | 13 +-
.../pypaimon/tests/ray_blob_affinity_test.py | 250 +++
.../tests/ray_data_evolution_merge_into_test.py | 168 +-
.../pypaimon/tests/ray_full_text_search_test.py | 95 ++
.../pypaimon/tests/ray_integration_test.py | 72 +
.../tests/ray_process_row_id_ranges_test.py | 8 +-
.../pypaimon/tests/ray_range_join_test.py | 2 +
.../pypaimon/tests/ray_read_by_row_id_test.py | 107 +-
.../pypaimon/tests/ray_repartition_test.py | 36 +-
.../tests/ray_row_id_conflict_rewriter_test.py | 6 +-
.../tests/ray_search_result_metadata_test.py | 57 +
paimon-python/pypaimon/tests/ray_sink_test.py | 34 +-
.../pypaimon/tests/ray_update_by_row_id_test.py | 56 +-
.../tests/ray_vector_filter_exactness_test.py | 60 +
.../tests/ray_vector_result_conversion_test.py | 87 +
.../pypaimon/tests/ray_vector_search_test.py | 80 +-
.../pypaimon/tests/read_builder_explain_test.py | 3 +-
.../pypaimon/tests/reader_append_only_test.py | 38 +-
paimon-python/pypaimon/tests/reader_base_test.py | 18 +-
.../pypaimon/tests/reader_parallel_test.py | 4 +
.../pypaimon/tests/resolving_file_io_test.py | 34 +
paimon-python/pypaimon/tests/rest/api_test.py | 20 +
paimon-python/pypaimon/tests/rest/client_test.py | 49 +
.../pypaimon/tests/rest/dlf_signer_test.py | 13 +
.../pypaimon/tests/rest/rest_base_test.py | 25 +-
.../pypaimon/tests/rest/rest_branch_test.py | 4 +
.../rest/rest_catalog_commit_snapshot_test.py | 2 +
.../rest/rest_list_partitions_fallback_test.py | 104 ++
.../tests/rest/rest_permission_management_test.py | 220 +++
.../pypaimon/tests/rest/rest_read_write_test.py | 45 +-
paimon-python/pypaimon/tests/rest/rest_server.py | 91 +-
.../pypaimon/tests/rest/rest_simple_test.py | 5 +
.../pypaimon/tests/rest/rest_token_file_io_test.py | 121 ++
.../pypaimon/tests/rest/token_loader_test.py | 19 +
.../pypaimon/tests/rest_user_agent_test.py | 152 ++
.../tests/robomind_agilex_pipeline_test.py | 190 ++-
.../pypaimon/tests/scan_file_shard_test.py | 64 +
.../scanner/chunk_shuffle_split_generator_test.py | 195 ++-
.../tests/schema_evolution_nested_read_test.py | 36 +
.../pypaimon/tests/schema_evolution_read_test.py | 43 +-
.../tests/search_query_auth_reject_test.py | 96 ++
.../pypaimon/tests/search_result_metadata_test.py | 206 +++
.../pypaimon/tests/shard_batch_reader_test.py | 94 ++
.../tests/shared_shredding_full_restore_test.py | 98 ++
.../pypaimon/tests/sorted_index_options_test.py | 51 +
.../pypaimon/tests/split_provider_test.py | 16 +-
.../pypaimon/tests/split_serializer_test.py | 89 +-
.../pypaimon/tests/stream_read_builder_test.py | 23 +-
.../pypaimon/tests/streaming_table_scan_test.py | 162 ++
.../pypaimon/tests/system/consumers_table_test.py | 99 ++
.../pypaimon/tests/system/files_table_test.py | 106 +-
.../tests/system/system_table_loader_test.py | 2 +-
.../pypaimon/tests/system/system_table_test.py | 135 +-
.../table/time_travel_schema_evolution_test.py | 28 +-
paimon-python/pypaimon/tests/table_commit_test.py | 83 +-
.../pypaimon/tests/table_merge_into_test.py | 102 ++
.../pypaimon/tests/table_query_auth_test.py | 70 +-
.../tests/table_read_chunked_overflow_test.py | 10 +
.../tests/table_update_by_row_id_chunked_test.py | 4 +-
paimon-python/pypaimon/tests/table_update_test.py | 154 +-
.../pypaimon/tests/table_upsert_by_key_test.py | 80 +-
.../pypaimon/tests/test_aggregation_e2e.py | 76 +-
.../pypaimon/tests/test_early_row_range_filter.py | 29 +-
.../tests/test_format_row_reader_writer.py | 253 ++-
.../pypaimon/tests/test_nested_projection_e2e.py | 33 +
.../tests/test_outer_projection_record_reader.py | 29 +-
.../pypaimon/tests/test_partial_update_e2e.py | 33 +-
.../pypaimon/tests/test_projection_utility.py | 20 +
.../pypaimon/tests/test_ray_shuffle_helper.py | 30 -
.../tests/test_read_builder_nested_projection.py | 296 +++-
.../pypaimon/tests/test_sequence_field_read.py | 32 +-
.../pypaimon/tests/test_sequence_field_write.py | 233 +++
.../pypaimon/tests/test_write_merge_buffer.py | 3 +
.../tests/timestamp_key_serializer_test.py | 97 ++
.../tests/torch_distributed_sharding_worker.py | 4 +
paimon-python/pypaimon/tests/torch_read_test.py | 240 ++-
.../pypaimon/tests/uri_reader_factory_test.py | 91 ++
paimon-python/pypaimon/tests/variant_path_test.py | 343 ++++
paimon-python/pypaimon/tests/variant_test.py | 267 ++++
.../pypaimon/tests/vector_filter_exactness_test.py | 223 +++
.../pypaimon/tests/vector_index_dv_build_test.py | 184 +++
.../tests/vector_index_parallel_search_test.py | 75 +-
.../pypaimon/tests/vector_search_filter_test.py | 110 +-
paimon-python/pypaimon/tests/vector_table_test.py | 27 +-
paimon-python/pypaimon/tests/video_format_test.py | 595 ++++++-
.../pypaimon/tests/video_keyframe_index_test.py | 231 +++
paimon-python/pypaimon/tests/where_parser_test.py | 133 +-
.../pypaimon/tests/window_transforms_test.py | 41 +
.../tests/write/changelog_producer_test.py | 81 +-
.../pypaimon/tests/write/commit_callback_test.py | 6 +-
.../pypaimon/tests/write/dynamic_bucket_test.py | 309 ++--
.../tests/write/postpone_event_write_test.py | 152 ++
.../write/simple_hash_bucket_assigner_test.py | 15 +
.../pypaimon/tests/write/table_write_test.py | 141 +-
.../pypaimon/tests/write/write_buffer_test.py | 50 +-
paimon-python/pypaimon/utils/arrow_utils.py | 53 +
paimon-python/pypaimon/utils/blob_view_lookup.py | 58 +-
.../pypaimon/utils/file_store_path_factory.py | 86 +-
paimon-python/pypaimon/utils/file_type.py | 16 +-
paimon-python/pypaimon/utils/path.py | 117 ++
paimon-python/pypaimon/utils/projection.py | 22 +-
.../pypaimon/write/commit/commit_scanner.py | 6 +-
.../pypaimon/write/commit/conflict_detection.py | 35 +-
.../write/commit/fixed_bucket_commit_check.py | 72 +
.../write/commit/overwrite_changes_provider.py | 4 +-
.../write/commit/row_id_conflict_rewriter.py | 30 +-
paimon-python/pypaimon/write/commit_message.py | 14 +-
.../pypaimon/write/commit_message_serializer.py | 265 ++++
paimon-python/pypaimon/write/file_store_commit.py | 368 ++---
paimon-python/pypaimon/write/file_store_write.py | 119 +-
.../pypaimon/write/map_shared_shredding_writer.py | 102 +-
paimon-python/pypaimon/write/native_commit.py | 179 +++
paimon-python/pypaimon/write/native_merge_into.py | 125 ++
paimon-python/pypaimon/write/native_update.py | 309 ++++
paimon-python/pypaimon/write/native_write.py | 295 ++++
.../pypaimon/write/postpone_batch_table_write.py | 23 +-
paimon-python/pypaimon/write/postpone_bucket.py | 76 +-
paimon-python/pypaimon/write/ray_datasink.py | 106 +-
paimon-python/pypaimon/write/row_id_file_index.py | 121 ++
paimon-python/pypaimon/write/row_key_extractor.py | 96 +-
paimon-python/pypaimon/write/row_kind.py | 66 +
paimon-python/pypaimon/write/row_utils.py | 17 +-
paimon-python/pypaimon/write/table_commit.py | 59 +-
paimon-python/pypaimon/write/table_update.py | 121 +-
.../pypaimon/write/table_update_by_row_id.py | 158 +-
.../pypaimon/write/table_upsert_by_key.py | 46 +-
paimon-python/pypaimon/write/table_write.py | 177 +--
.../pypaimon/write/video_format_writer.py | 98 +-
paimon-python/pypaimon/write/write_builder.py | 78 +-
.../pypaimon/write/writer/composite_data_writer.py | 11 +
paimon-python/pypaimon/write/writer/data_writer.py | 55 +-
.../write/writer/dedicated_format_writer.py | 38 +-
.../pypaimon/write/writer/format_row_writer.py | 17 +-
.../pypaimon/write/writer/key_value_data_writer.py | 43 +-
.../writer/parquet_writer_options.py} | 31 +-
.../pypaimon/write/writer/postpone_data_writer.py | 109 ++
.../pypaimon/write/writer/single_file_writer.py | 4 +-
.../pypaimon/write/writer/vector_writer.py | 8 +-
paimon-python/pyproject.toml | 211 +++
paimon-python/setup.py | 335 +---
.../scala/org/apache/paimon/spark/PaimonScan.scala | 2 +-
.../paimon/spark/catalyst/Compatibility.scala | 23 +-
.../scala/org/apache/paimon/spark/PaimonScan.scala | 28 +-
.../paimon/spark/catalyst/Compatibility.scala | 12 +-
.../paimon/spark/catalyst/Compatibility.scala | 12 +-
.../MergeIntoPaimonDataEvolutionTable.scala | 76 +-
.../spark/sql/BucketedScanCompatibilityTest.scala | 9 +-
.../spark/sql/BucketedScanPlanningTest.scala | 9 +-
.../java/org/apache/paimon/spark/SparkCatalog.java | 2 +-
.../apache/paimon/spark/SparkConnectorOptions.java | 13 +
.../org/apache/paimon/spark/SparkProcedures.java | 9 +
.../java/org/apache/paimon/spark/SparkRow.java | 15 +-
.../paimon/spark/catalog/FormatTableCatalog.java | 13 +-
.../globalindex/DefaultGlobalIndexBuilder.java | 2 +-
.../globalindex/sorted/SortedIndexTopoBuilder.java | 59 +-
.../procedure/CreateGlobalIndexProcedure.java | 6 +-
.../procedure/DataEvolutionRewriteExecutor.java | 3 +
.../spark/procedure/ExpireChangelogsProcedure.java | 157 ++
.../spark/procedure/ImportFilesProcedure.java | 101 ++
.../MaterializeDeletionVectorsProcedure.java | 19 +-
.../RemoveUnexistingManifestsProcedure.java | 102 ++
.../procedure/RollbackToAsLatestProcedure.java | 242 +++
.../spark/read/SparkDataEvolutionVectorRead.java | 5 +-
.../apache/paimon/spark/sort/SparkHilbertUDF.java | 17 +
.../apache/paimon/spark/sort/SparkZOrderUDF.java | 67 +
.../scala/org/apache/paimon/spark/PaimonScan.scala | 28 +-
.../apache/paimon/spark/PaimonScanBuilder.scala | 10 +-
.../org/apache/paimon/spark/SparkSource.scala | 3 +-
.../paimon/spark/catalyst/Compatibility.scala | 18 +-
.../spark/catalyst/analysis/PaimonAnalysis.scala | 3 +
.../analysis/PaimonFormatOutputResolver.scala | 59 +
.../catalyst/analysis/PaimonFunctionResolver.scala | 4 +-
.../catalyst/analysis/PaimonUpdateTable.scala | 53 +-
.../analysis/PaimonViewDependencyChecker.scala | 146 ++
.../catalyst/analysis/PaimonViewResolver.scala | 41 +-
.../catalyst/analysis/ReplacePaimonFunctions.scala | 4 +-
...OptimizeMetadataOnlyDeleteFromPaimonTable.scala | 9 +-
.../spark/catalyst/plans/logical/CopyOptions.scala | 8 +-
.../DataEvolutionRowIdConflictRewriter.scala | 7 +-
.../MergeIntoPaimonDataEvolutionTable.scala | 75 +-
.../paimon/spark/commands/PaimonSparkWriter.scala | 8 +-
.../UpdatePaimonDataEvolutionTableCommand.scala | 58 +-
.../paimon/spark/execution/PaimonStrategy.scala | 1 +
.../paimon/spark/execution/PaimonViewExec.scala | 7 +-
.../DisableUnnecessaryPaimonBucketedScan.scala | 178 ---
.../extensions/PaimonSparkSessionExtensions.scala | 3 +-
.../paimon/spark/format/PaimonFormatTable.scala | 21 +-
.../paimon/spark/metric/SparkMetricGroup.scala | 71 +
.../paimon/spark/metric/SparkMetricRegistry.scala | 5 +-
.../paimon/spark/read/BinPackingSplits.scala | 34 +-
.../spark/util/SparkExpressionConverter.scala | 81 +-
.../org/apache/paimon/spark/util/SplitUtils.scala | 11 +-
.../apache/paimon/spark/write/WriteHelper.scala | 6 +-
.../spark/metrics/source/PaimonMetricsSource.scala | 64 +
.../extensions/PaimonSqlExtensionsAstBuilder.scala | 4 +-
.../spark/catalog/FormatTableCatalogTest.java | 69 +
.../paimon/spark/copy/CopyFilesUtilTest.java | 6 +-
.../procedure/CreateGlobalIndexProcedureTest.java | 4 +-
.../read/SparkDataEvolutionVectorReadTest.java | 1 +
.../paimon/spark/sort/SparkHilbertUDFTest.java | 57 +
.../paimon/spark/sort/SparkZOrderUDFTest.java | 188 +++
.../spark/util/SplitUtilsQueryAuthTest.scala | 83 +
.../apache/paimon/spark/BinPackingSplitsTest.scala | 45 +-
.../spark/procedure/CompactProcedureTestBase.scala | 20 +-
.../CompositeBTreeIndexProcedureTest.scala | 234 +++
.../procedure/ExpireChangelogsProcedureTest.scala | 265 ++++
.../spark/procedure/ImportFilesProcedureTest.scala | 111 ++
.../RemoveUnexistingManifestsProcedureTest.scala | 159 ++
.../RollbackToAsLatestProcedureTest.scala | 186 +++
.../spark/sql/BucketedScanPlanningTestBase.scala | 338 ++++
...canSuite.scala => BucketedScanPolicyTest.scala} | 127 +-
.../paimon/spark/sql/BucketedTableQueryTest.scala | 1 +
.../spark/sql/CastPredicatePushDownTest.scala | 315 ++++
.../org/apache/paimon/spark/sql/DDLTestBase.scala | 63 +
.../apache/paimon/spark/sql/DefaultValueTest.scala | 12 +
.../paimon/spark/sql/DeleteFromTableTestBase.scala | 75 +
.../paimon/spark/sql/FormatTableTestBase.scala | 272 +++-
.../paimon/spark/sql/FullTextSearchTest.scala | 45 +
.../apache/paimon/spark/sql/PaimonMetricTest.scala | 70 +
.../paimon/spark/sql/PaimonPushDownTestBase.scala | 3 +-
.../paimon/spark/sql/PaimonSystemTableTest.scala | 12 +
.../paimon/spark/sql/PaimonViewTestBase.scala | 225 ++-
.../paimon/spark/sql/RowTrackingTestBase.scala | 405 ++++-
.../spark/sql/SparkV2FilterConverterTestBase.scala | 59 +-
.../paimon/spark/sql/UpdateTableTestBase.scala | 52 +
.../testutils/junit/DockerImageVersions.java | 5 +-
paimon-vector/pom.xml | 2 +-
.../vector/index/NativeVectorGlobalIndexer.java | 2 +
.../index/NativeVectorGlobalIndexerFactory.java | 145 +-
.../test/java/org/apache/paimon/JavaPyE2ETest.java | 4 +-
.../vector/index/NativeVectorGlobalIndexTest.java | 2 +-
.../NativeVectorGlobalIndexerFactoryTest.java | 133 +-
.../paimon/vfs/hadoop/PaimonVirtualFileSystem.java | 12 +-
.../vfs/hadoop/MockRestVirtualFileSystemTest.java | 39 +-
.../format/vortex/VortexPredicateConverter.java | 101 +-
.../paimon/format/vortex/VortexReaderFactory.java | 20 +-
.../paimon/format/vortex/VortexRecordsReader.java | 25 +-
.../src/main/java/dev/vortex/api/Expression.java | 10 +-
.../dev/vortex/api/ExpressionLifetimeTest.java | 59 +
pom.xml | 6 +
.../tools/ci/licensecheck/JarFileChecker.java | 2 +-
tools/ci/test_deploy_maven_repository.py | 89 ++
tools/releasing/deploy_maven_repository.sh | 2 +-
1175 files changed, 92752 insertions(+), 9236 deletions(-)
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-api/src/main/java-templates/org/apache/paimon/utils/BuildVersions.java
(71%)
create mode 100644
paimon-api/src/test/java/org/apache/paimon/TurkishLocaleParsingTest.java
create mode 100644
paimon-benchmark/paimon-micro-benchmarks/src/test/java/org/apache/paimon/benchmark/AvroSmallFileReadBenchmark.java
create mode 100644
paimon-common/src/main/java/org/apache/paimon/globalindex/CompositeKeySerializer.java
create mode 100644
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedFileIndexPlanner.java
create mode 100644
paimon-common/src/main/java/org/apache/paimon/globalindex/btree/CompositeBTreePredicate.java
create mode 100644
paimon-common/src/main/java/org/apache/paimon/predicate/NestedFieldTransform.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/data/serializer/BinaryRowSerializerLengthTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/format/FileFormatPrefixOptionsTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/CompositeKeySerializerTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/btree/BTreeIndexWriterCloseTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/btree/CompositeBTreeIndexTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/btree/CompositeBTreePredicateTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/globalindex/btree/CompositeBTreeRangeTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/predicate/CastTransformTest.java
create mode 100644
paimon-common/src/test/java/org/apache/paimon/predicate/NestedFieldTransformTest.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/append/SortCompactCommitMessageRewriter.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/append/SortCompactPlanMetadata.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/SerializationAssignment.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/deletionvectors/DeletionVectorChecksum.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/globalindex/IndexQuerySplit.java
delete mode 100644
paimon-core/src/main/java/org/apache/paimon/mergetree/DataFileReader.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/metastore/ChainTableOverwriteScope.java
create mode 100644
paimon-core/src/main/java/org/apache/paimon/migrate/ExternalFileImporter.java
copy
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/RemoveUnexistingManifestsAction.java
=>
paimon-core/src/main/java/org/apache/paimon/operation/RemoveUnexistingManifests.java
(61%)
create mode 100644
paimon-core/src/main/java/org/apache/paimon/table/source/Splits.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/append/SortCompactCommitMessageRewriterTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/globalindex/GlobalIndexQueryTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/index/PartitionIndexTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/lookup/memory/InMemorySetStateTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/migrate/ExternalFileImporterTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/operation/RemoveUnexistingManifestsTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/AppendOnlyLimitManifestReadTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/ChainTableSnapshotExpireTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/CompositeBTreeTableTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/ExpireSnapshotsInterruptStatusTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/IndexQuerySplitTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/NestedFieldQueryAuthTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/PartitionLayoutScanTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/source/ReadBuilderImplTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/source/SplitsTest.java
create mode 100644
paimon-core/src/test/java/org/apache/paimon/table/source/VectorSearchRowFilterExactnessTest.java
create mode 100644
paimon-core/src/test/resources/compatibility/manifest-committable-v14-v5
create mode 100644
paimon-filesystems/paimon-cosn/src/test/java/org/apache/paimon/cosn/COSNLoaderTest.java
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-filesystems/paimon-jindo/src/main/java-templates/org/apache/paimon/jindo/JindoBuildVersions.java
(71%)
create mode 100644
paimon-filesystems/paimon-jindo/src/main/java/org/apache/paimon/jindo/JindoUserAgent.java
create mode 100644
paimon-filesystems/paimon-jindo/src/test/java/org/apache/paimon/jindo/TestJindoUserAgent.java
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-filesystems/paimon-oss-impl/src/main/java-templates/org/apache/paimon/oss/OSSBuildVersions.java
(69%)
create mode 100644
paimon-filesystems/paimon-oss-impl/src/main/java/org/apache/paimon/oss/OSSRetryStrategy.java
create mode 100644
paimon-filesystems/paimon-oss-impl/src/main/java/org/apache/paimon/oss/OSSUserAgent.java
create mode 100644
paimon-filesystems/paimon-oss-impl/src/test/java/org/apache/paimon/oss/OSSRetryStrategyTest.java
create mode 100644
paimon-filesystems/paimon-oss-impl/src/test/java/org/apache/paimon/oss/OSSUserAgentTest.java
create mode 100644
paimon-filesystems/paimon-s3-impl/src/test/java/org/apache/paimon/s3/S3FileIOConfigTest.java
create mode 100644
paimon-filesystems/paimon-s3-impl/src/test/java/org/apache/paimon/s3/S3FileIOSchemeCacheTest.java
create mode 100644
paimon-flink/paimon-flink-1.18/src/main/java/org/apache/paimon/flink/procedure/RemoveOrphanBlobsProcedure.java
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-flink/paimon-flink-1.18/src/test/java/org/apache/paimon/flink/RemoveOrphanBlobsActionITCase.java
(67%)
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-flink/paimon-flink-1.19/src/test/java/org/apache/paimon/flink/RemoveOrphanBlobsActionITCase.java
(71%)
create mode 100644
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/action/cdc/TurkishLocaleCaseFoldingTest.java
copy
paimon-common/src/test/java/org/apache/paimon/utils/RowRangeIndexTest.java =>
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/action/cdc/format/aliyun/AliyunFieldParserTest.java
(55%)
copy
paimon-common/src/test/java/org/apache/paimon/utils/RowRangeIndexTest.java =>
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/action/cdc/format/canal/CanalFieldParserTest.java
(55%)
create mode 100644
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/action/cdc/format/canal/CanalRecordParserTest.java
create mode 100644
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/action/cdc/mysql/TurkishLocaleTypeNameTest.java
create mode 100644
paimon-flink/paimon-flink-cdc/src/test/java/org/apache/paimon/flink/sink/cdc/CdcRecordTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/NestedFieldReferences.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/RemoveOrphanBlobsAction.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/RemoveOrphanBlobsActionFactory.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkManagedBlobOrphanFilesClean.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/ImportFilesProcedure.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/RemoveOrphanBlobsProcedure.java
create mode 100644
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/SplitWeightUtils.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/NestedFieldReferencesTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/NestedPredicateConverterTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/action/MultiTablesSinkModeTest.java
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/action/RemoveOrphanBlobsActionITCase.java
(71%)
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/action/RemoveOrphanBlobsActionITCaseBase.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/compact/changelog/format/CompactedChangelogFormatReaderFactoryTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/ImportFilesProcedureITCase.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/IndexQuerySourceTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/NestedFieldFilterPushDownITCase.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/align/AlignedContinuousFileSplitEnumeratorQueryAuthTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/assigners/AlignedSplitAssignerTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/assigners/DynamicPartitionPruningAssignerTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/assigners/PreAssignSplitAssignerTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/operator/ReadOperatorSplitShapeTest.java
create mode 100644
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/utils/TableScanUtilsTest.java
create mode 100644
paimon-format/src/main/java/org/apache/paimon/format/blob/VideoKeyframeIndex.java
create mode 100644
paimon-format/src/main/java/org/apache/paimon/format/json/JsonMapKeyConverter.java
create mode 100644
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetListLayoutResolver.java
create mode 100644
paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetLegacyListReadTest.java
create mode 100644
paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetListLayoutResolverTest.java
create mode 100644
paimon-format/src/test/java/org/apache/paimon/format/row/RowFileIndexConsistencyTest.java
create mode 100644
paimon-hive/paimon-hive-catalog/src/test/java/org/apache/paimon/hive/clone/HiveTableCloneExtractorTest.java
create mode 100644
paimon-lumina/src/test/java/org/apache/paimon/lumina/index/LuminaVectorMetricTest.java
create mode 100644 paimon-python/dev/install.py
delete mode 100644 paimon-python/dev/requirements-dev.txt
delete mode 100644 paimon-python/dev/requirements.txt
create mode 100644 paimon-python/dev/test_packaging.py
copy paimon-python/{MANIFEST.in => pypaimon/_version.py} (77%)
create mode 100644 paimon-python/pypaimon/api/rest_permission_management.py
create mode 100644 paimon-python/pypaimon/benchmark/act/README.md
rename paimon-python/pypaimon/{sample => benchmark/act}/robomind_agilex.py
(70%)
create mode 100644 paimon-python/pypaimon/benchmark/video_random_read.py
create mode 100644 paimon-python/pypaimon/catalog/catalog_utils.py
create mode 100644 paimon-python/pypaimon/common/blob_index_cache.py
create mode 100644 paimon-python/pypaimon/common/decimal_arrow_predicate.py
create mode 100644 paimon-python/pypaimon/common/user_agent.py
create mode 100644 paimon-python/pypaimon/filesystem/oss_blob_presigner.py
create mode 100644 paimon-python/pypaimon/filesystem/oss_user_agent.py
copy paimon-python/pypaimon/{api => management}/__init__.py (100%)
copy
paimon-common/src/test/resources/org/apache/paimon/data/video-frame-descriptor-v1.hex
=> paimon-python/pypaimon/management/java_string.py (66%)
create mode 100644
paimon-python/pypaimon/management/list_permissions_request.py
create mode 100644 paimon-python/pypaimon/management/permission_access.py
create mode 100644 paimon-python/pypaimon/management/permission_assignment.py
create mode 100644 paimon-python/pypaimon/management/permission_columns.py
copy paimon-python/pypaimon/{read/reader/iface/record_reader.py =>
management/permission_management.py} (57%)
create mode 100644 paimon-python/pypaimon/management/permission_resource.py
copy paimon-python/pypaimon/{catalog/rest/table_metadata.py =>
management/resource_type.py} (58%)
delete mode 100644 paimon-python/pypaimon/manifest/manifest_file_merger.py
create mode 100644 paimon-python/pypaimon/ray/full_text_search.py
create mode 100644 paimon-python/pypaimon/read/read_type.py
create mode 100644 paimon-python/pypaimon/read/reader/blob_view_read_support.py
create mode 100644
paimon-python/pypaimon/read/reader/parquet_page_index_reader.py
create mode 100644 paimon-python/pypaimon/read/variant_read_type.py
create mode 100644 paimon-python/pypaimon/schema/arrow_schema.py
create mode 100644 paimon-python/pypaimon/table/row/video_keyframe_index.py
create mode 100644
paimon-python/pypaimon/table/source/global_index_row_filter.py
create mode 100644 paimon-python/pypaimon/table/system/consumers_table.py
create mode 100644 paimon-python/pypaimon/tests/arrow_schema_test.py
create mode 100644
paimon-python/pypaimon/tests/commit_message_serializer_test.py
create mode 100644
paimon-python/pypaimon/tests/composite_btree_fallback_test.py
create mode 100644 paimon-python/pypaimon/tests/daft/daft_blob_auth_test.py
create mode 100644
paimon-python/pypaimon/tests/data_file_path_directory_test.py
create mode 100644 paimon-python/pypaimon/tests/decimal_key_serializer_test.py
create mode 100644 paimon-python/pypaimon/tests/decimal_predicate_test.py
create mode 100644
paimon-python/pypaimon/tests/full_text_reader_cleanup_test.py
create mode 100644 paimon-python/pypaimon/tests/full_text_scalar_filter_test.py
create mode 100644 paimon-python/pypaimon/tests/hdfs_native_write_row_test.py
create mode 100644
paimon-python/pypaimon/tests/native_blob_collection_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_blob_consumer_test.py
create mode 100644 paimon-python/pypaimon/tests/native_blob_update_test.py
create mode 100644 paimon-python/pypaimon/tests/native_blob_uri_reader_test.py
create mode 100644 paimon-python/pypaimon/tests/native_blob_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_branch_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_commit_test.py
create mode 100644
paimon-python/pypaimon/tests/native_cross_partition_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_data_directory_test.py
create mode 100644
paimon-python/pypaimon/tests/native_data_evolution_commit_test.py
create mode 100644 paimon-python/pypaimon/tests/native_dynamic_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_external_write_test.py
create mode 100644
paimon-python/pypaimon/tests/native_managed_blob_write_test.py
create mode 100644
paimon-python/pypaimon/tests/native_map_shredding_write_test.py
create mode 100644
paimon-python/pypaimon/tests/native_merge_incremental_test.py
create mode 100644 paimon-python/pypaimon/tests/native_merge_into_test.py
create mode 100644 paimon-python/pypaimon/tests/native_merge_read_test.py
create mode 100644 paimon-python/pypaimon/tests/native_nested_update_test.py
create mode 100644 paimon-python/pypaimon/tests/native_pk_blob_limit_test.py
create mode 100644
paimon-python/pypaimon/tests/native_postpone_fixed_write_test.py
create mode 100644 paimon-python/pypaimon/tests/native_postpone_write_test.py
create mode 100644
paimon-python/pypaimon/tests/native_precomputed_bucket_test.py
create mode 100644 paimon-python/pypaimon/tests/native_read_type_test.py
create mode 100644 paimon-python/pypaimon/tests/native_row_sidecar_test.py
create mode 100644 paimon-python/pypaimon/tests/native_sequence_test.py
create mode 100644 paimon-python/pypaimon/tests/native_system_columns_test.py
create mode 100644 paimon-python/pypaimon/tests/native_update_test.py
create mode 100644
paimon-python/pypaimon/tests/native_upsert_capabilities_test.py
create mode 100644
paimon-python/pypaimon/tests/native_upsert_row_shapes_test.py
create mode 100644 paimon-python/pypaimon/tests/native_variant_update_test.py
create mode 100644 paimon-python/pypaimon/tests/native_video_read_test.py
create mode 100644
paimon-python/pypaimon/tests/native_write_capabilities_test.py
create mode 100644 paimon-python/pypaimon/tests/native_write_test.py
create mode 100644
paimon-python/pypaimon/tests/normal_map_selected_keys_test.py
create mode 100644 paimon-python/pypaimon/tests/oss_blob_presigner_test.py
create mode 100644 paimon-python/pypaimon/tests/oss_user_agent_test.py
create mode 100644 paimon-python/pypaimon/tests/parquet_page_index_test.py
create mode 100644
paimon-python/pypaimon/tests/parquet_write_page_index_test.py
create mode 100644
paimon-python/pypaimon/tests/permission_management_json_test.py
create mode 100644 paimon-python/pypaimon/tests/polars_map_null_test.py
create mode 100644
paimon-python/pypaimon/tests/py36/named_variant_projection_test.py
create mode 100644 paimon-python/pypaimon/tests/py36/where_literal_cast_test.py
create mode 100644 paimon-python/pypaimon/tests/query_auth_validation_test.py
create mode 100644 paimon-python/pypaimon/tests/query_auth_widening_test.py
create mode 100644
paimon-python/pypaimon/tests/ray_batch_vector_refinement_test.py
create mode 100644 paimon-python/pypaimon/tests/ray_blob_affinity_test.py
create mode 100644 paimon-python/pypaimon/tests/ray_full_text_search_test.py
create mode 100644
paimon-python/pypaimon/tests/ray_search_result_metadata_test.py
create mode 100644
paimon-python/pypaimon/tests/ray_vector_filter_exactness_test.py
create mode 100644
paimon-python/pypaimon/tests/ray_vector_result_conversion_test.py
create mode 100644
paimon-python/pypaimon/tests/rest/rest_list_partitions_fallback_test.py
create mode 100644
paimon-python/pypaimon/tests/rest/rest_permission_management_test.py
create mode 100644 paimon-python/pypaimon/tests/rest_user_agent_test.py
create mode 100644 paimon-python/pypaimon/tests/scan_file_shard_test.py
create mode 100644
paimon-python/pypaimon/tests/search_query_auth_reject_test.py
create mode 100644 paimon-python/pypaimon/tests/search_result_metadata_test.py
create mode 100644 paimon-python/pypaimon/tests/shard_batch_reader_test.py
create mode 100644
paimon-python/pypaimon/tests/shared_shredding_full_restore_test.py
create mode 100644 paimon-python/pypaimon/tests/sorted_index_options_test.py
create mode 100644 paimon-python/pypaimon/tests/system/consumers_table_test.py
create mode 100644 paimon-python/pypaimon/tests/test_sequence_field_write.py
create mode 100644
paimon-python/pypaimon/tests/timestamp_key_serializer_test.py
create mode 100644 paimon-python/pypaimon/tests/vector_filter_exactness_test.py
create mode 100644 paimon-python/pypaimon/tests/vector_index_dv_build_test.py
create mode 100644 paimon-python/pypaimon/tests/video_keyframe_index_test.py
create mode 100644
paimon-python/pypaimon/tests/write/postpone_event_write_test.py
create mode 100644 paimon-python/pypaimon/utils/arrow_utils.py
create mode 100644 paimon-python/pypaimon/utils/path.py
create mode 100644
paimon-python/pypaimon/write/commit/fixed_bucket_commit_check.py
create mode 100644 paimon-python/pypaimon/write/commit_message_serializer.py
create mode 100644 paimon-python/pypaimon/write/native_commit.py
create mode 100644 paimon-python/pypaimon/write/native_merge_into.py
create mode 100644 paimon-python/pypaimon/write/native_update.py
create mode 100644 paimon-python/pypaimon/write/native_write.py
create mode 100644 paimon-python/pypaimon/write/row_id_file_index.py
create mode 100644 paimon-python/pypaimon/write/row_kind.py
copy paimon-python/pypaimon/{catalog/rest/table_metadata.py =>
write/writer/parquet_writer_options.py} (59%)
create mode 100644 paimon-python/pypaimon/write/writer/postpone_data_writer.py
create mode 100644 paimon-python/pyproject.toml
copy paimon-spark/{paimon-spark-common =>
paimon-spark-3.4}/src/main/scala/org/apache/paimon/spark/catalyst/Compatibility.scala
(81%)
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/paimon/spark/sql/BucketedScanCompatibilityTest.scala
(71%)
copy
paimon-common/src/main/java/org/apache/paimon/globalindex/SortedGlobalIndexer.java
=>
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/BucketedScanPlanningTest.scala
(71%)
create mode 100644
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/ExpireChangelogsProcedure.java
create mode 100644
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/ImportFilesProcedure.java
create mode 100644
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/RemoveUnexistingManifestsProcedure.java
create mode 100644
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/RollbackToAsLatestProcedure.java
create mode 100644
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonFormatOutputResolver.scala
create mode 100644
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonViewDependencyChecker.scala
delete mode 100644
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/adaptive/DisableUnnecessaryPaimonBucketedScan.scala
create mode 100644
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/metric/SparkMetricGroup.scala
create mode 100644
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/metrics/source/PaimonMetricsSource.scala
create mode 100644
paimon-spark/paimon-spark-common/src/test/java/org/apache/paimon/spark/catalog/FormatTableCatalogTest.java
create mode 100644
paimon-spark/paimon-spark-common/src/test/scala/org/apache/paimon/spark/util/SplitUtilsQueryAuthTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/CompositeBTreeIndexProcedureTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/ExpireChangelogsProcedureTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/ImportFilesProcedureTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/RemoveUnexistingManifestsProcedureTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/RollbackToAsLatestProcedureTest.scala
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/BucketedScanPlanningTestBase.scala
rename
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/{DisableUnnecessaryPaimonBucketedScanSuite.scala
=> BucketedScanPolicyTest.scala} (69%)
create mode 100644
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/CastPredicatePushDownTest.scala
create mode 100644
paimon-vortex/paimon-vortex-jni/src/test/java/dev/vortex/api/ExpressionLifetimeTest.java
create mode 100644 tools/ci/test_deploy_maven_repository.py
