This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a change to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
from 2acbe6514ee0 fix(build): restore imports dropped by earlier
cherry-picks
new f0fa7a198506 refactor(spark): consolidate the vendored AvroUtils fork
into hudi-spark-common (#19147)
new 8327a728d878 refactor(spark): share Spark 4.x partition-values and
mapping classes in hudi-spark4-common (#19148)
new 0ba508d05737 refactor(spark): dedup catalyst utils and analysis rules
across Spark version modules (#19149)
new ec8e678d8766 refactor(spark): consolidate duplicated small utils
across Spark version modules (#19150)
new 49736bf3bc48 refactor: Retire leftover Avro Schema usages in
AvroSchemaUtils and LSMTimelineWriter (#19153)
new a91e4443bcfd refactor: Migrate
reconcileSchema/reconcileSchemaRequirements to HoodieSchema (#19154)
new 5906248205cf perf(common): replace BitSet with a fixed word array in
the ported bloom filter (#19140)
new 2d84d0ca5540 test: handle expected OCC conflict in concurrent Java
writer test (#19124)
new 98598d6e885f refactor(flink): use SLF4J parameterized logging instead
of string concatenation (#19156)
new 185bfae35a49 refactor(hadoop-mr): use SLF4J parameterized logging
instead of string concatenation (#19157)
new eb67451ea600 refactor(hadoop-common): use SLF4J parameterized logging
instead of string concatenation (#19160)
new f05907925920 refactor(common): use SLF4J parameterized logging instead
of string concatenation (#19158)
new 4007c392ef94 fix(spark): correct self-recursive equals in
ProcedureParameterImpl (#19167)
new 71c368d3a472 refactor(cli): use SLF4J parameterized logging instead of
string concatenation (#19159)
new 75c587bd6372 test(spark): add streaming source and writer support
coverage (#19166)
new fd3823a42c3c refactor(spark): use SLF4J parameterized logging instead
of string concatenation (#19184)
new 16cdb7dc90cc refactor(aws): use SLF4J parameterized logging instead of
string concatenation (#19186)
new 51d1c8ade755 refactor(utilities): use SLF4J parameterized logging
instead of string concatenation (#19185)
new 6f8b046e137d refactor(examples): use SLF4J parameterized logging
instead of string concatenation (#19187)
new 18a2eb849fea fix(spark): make export_instants desc ordering work
(#19172)
new ab746b31fd3e refactor(sync): use SLF4J parameterized logging instead
of string concatenation (#19189)
new 5241c4860c9a refactor(client): use SLF4J parameterized logging instead
of string concatenation (#19155)
new 81fff774a5bd refactor(io): use SLF4J parameterized logging instead of
string concatenation (#19188)
new dc7eff474f16 test(trino): de-flake testRecordLevelFileSkipping by
setting the record-index wait timeout (#19213)
new 82c8fd9249da fix(flink): avoid AIOOBE in NestedColumnReader across the
read batch boundary (#19210)
new 72fee44b3e28 fix(reader): Remove redundant partition value conversion
in RecordContext (#19201)
new adebc4c95682 test(integ-test): add Testcontainers E2E for
VECTOR/BLOB/VARIANT Hive… (#19203)
new 82f519e5f5ea fix(spark): demote spurious per-write INFO logs to debug
(#19170)
new 86871e04030c test(client): cover low-coverage small classes across
hudi-client (#19224)
new 71b90a97e900 test(integ-test): add Spark 4.1.1 stack to hive-sync E2E
matrix (#19216)
new 61a3a496e5e6 fix(bundle): shade parquet-variant into common bundle
includes for Spark 4.1+ (#19235)
new dc5c845de223 test(spark): add extended SQL parser coverage for index
DDL and Hudi column types (#19218)
new fd3f51053b77 fix(spark): read Lance BLOB columns in <=512-row chunks
to avoid lance-core FFI abort (#19181)
new 51a52547568a fix(common): name the offending expression in
BindVisitor's unsupported-predicate error (#19241)
new 265768229f3c refactor(spark): consolidate the vendored 3.x Avro serde
forks into hudi-spark3-common (#19168)
new 96bb7b2ccc93 fix(flink): prevent data loss on global failover for
streaming writes (#19237)
new 46d3c8494aa2 fix(build): import OperationConverter from its
release-branch package
new 5b3fe07bc716 fix(spark): read INLINE blobs as CONTENT on internal
write-side Lance… (#19236)
new c19105f07e2d fix(build): import lock result enums from their
release-branch package
new e22eaadd06bf fix(build): import Spark Metadata/StructField types in
TestHoodieSparkLanceWriter
The 40 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/bot.yml | 82 +++
...r-compose_hadoop284_hive2310_spark353_amd64.yml | 76 ---
...r-compose_hadoop284_hive2310_spark353_arm64.yml | 72 ---
...-compose_hadoop340_hive2310_spark411_amd64.yml} | 14 +-
...-compose_hadoop340_hive2310_spark411_arm64.yml} | 14 +-
docker/demo/sparksql-blob-type-df.commands | 108 ++++
docker/demo/sparksql-blob-type-sql.commands | 126 ++++
docker/demo/sparksql-variant-type-df.commands | 80 +++
docker/demo/sparksql-variant-type-sql.commands | 89 +++
docker/demo/sparksql-vector-type-df.commands | 90 +++
docker/demo/sparksql-vector-type-sql.commands | 90 +++
docker/hoodie/hadoop/hive_base/entrypoint.sh | 2 +-
docker/hoodie/hadoop/sparkmaster/master.sh | 2 +-
.../hudi/aws/sync/AWSGlueCatalogSyncClient.java | 8 +-
.../apache/hudi/aws/utils/DynamoTableUtils.java | 4 +-
.../hudi/cli/commands/CompactionCommand.java | 2 +-
.../apache/hudi/cli/commands/ExportCommand.java | 2 +-
.../hudi/cli/commands/LockAuditingCommand.java | 6 +-
.../apache/hudi/cli/commands/MetadataCommand.java | 31 +-
.../apache/hudi/cli/commands/RepairsCommand.java | 6 +-
.../org/apache/hudi/cli/commands/TableCommand.java | 5 +-
.../apache/hudi/async/AsyncClusteringService.java | 2 +-
.../org/apache/hudi/async/AsyncCompactService.java | 6 +-
.../org/apache/hudi/async/HoodieAsyncService.java | 2 +-
.../hudi/client/BaseHoodieTableServiceClient.java | 2 +-
.../apache/hudi/client/BaseHoodieWriteClient.java | 25 +-
.../apache/hudi/client/CompactionAdminClient.java | 5 +-
.../client/HoodieTableServiceManagerClient.java | 2 +-
.../selector/BootstrapRegexModeSelector.java | 2 +-
.../client/embedded/EmbeddedTimelineService.java | 2 +-
.../timeline/versioning/v1/TimelineArchiverV1.java | 10 +-
.../timeline/versioning/v2/LSMTimelineWriter.java | 10 +-
...urrentFileWritesConflictResolutionStrategy.java | 3 +-
.../DirectMarkerTransactionManager.java | 10 +-
...urrentFileWritesConflictResolutionStrategy.java | 6 +-
.../lock/FileSystemBasedLockProvider.java | 2 +-
.../hudi/client/transaction/lock/LockManager.java | 2 +-
.../apache/hudi/client/utils/TransactionUtils.java | 3 +-
.../client/validator/StreamingOffsetValidator.java | 2 +-
.../FileMetadataWriteStatusConverter.java | 2 +-
.../index/bucket/ConsistentBucketIndexUtils.java | 4 +-
.../hudi/index/bucket/HoodieBucketIndex.java | 2 +-
.../java/org/apache/hudi/io/BaseCreateHandle.java | 4 +-
.../hudi/io/ExternalFileClusteringWriteHandle.java | 2 +-
.../hudi/io/FileGroupReaderBasedMergeHandle.java | 2 +-
.../org/apache/hudi/io/HoodieAppendHandle.java | 2 +-
.../org/apache/hudi/io/HoodieBinaryCopyHandle.java | 10 +-
.../src/main/java/org/apache/hudi/io/IOUtils.java | 2 +-
.../metadata/HoodieBackedTableMetadataWriter.java | 9 +
...ieBackedTableMetadataWriterTableVersionSix.java | 2 +-
.../java/org/apache/hudi/table/HoodieTable.java | 6 +-
.../CommitBasedClusteringPlanStrategy.java | 6 +-
.../PartitionAwareClusteringPlanStrategy.java | 10 +-
.../action/commit/BaseCommitActionExecutor.java | 8 +-
.../table/action/commit/HoodieMergeHelper.java | 2 +-
.../compact/ScheduleCompactionActionExecutor.java | 4 +-
.../generators/HoodieCompactionPlanGenerator.java | 2 +-
.../action/index/AbstractIndexingCatchupTask.java | 2 +-
.../action/restore/BaseRestoreActionExecutor.java | 4 +-
.../rollback/BaseRollbackActionExecutor.java | 6 +-
.../CopyOnWriteRollbackActionExecutor.java | 6 +-
.../rollback/ListingBasedRollbackStrategy.java | 2 +-
.../MergeOnReadRollbackActionExecutor.java | 8 +-
.../marker/TimelineServerBasedWriteMarkers.java | 6 +-
.../table/upgrade/FiveToSixUpgradeHandler.java | 4 +-
.../hudi/table/upgrade/UpgradeDowngrade.java | 2 +-
.../apache/hudi/client/TestClientStatsPojos.java | 94 +++
.../TestBootstrapPartitionPathTranslators.java | 46 ++
.../transaction/lock/TestLockResultEnums.java | 65 +++
.../lock/audit/TestAuditOperationState.java | 46 +-
.../utils/TestLegacyArchivedMetaEntryReader.java | 4 +-
.../hudi/exception/TestClientExceptions.java | 71 +++
.../bulkinsert/TestBulkInsertSortMode.java | 52 ++
.../hudi/index/bloom/TestBloomIndexFileInfo.java | 79 +++
.../TestHoodieBackedTableMetadataWriter.java | 27 +
.../table/action/commit/TestBucketTypeAndInfo.java | 72 +++
.../table/action/rollback/TestRollbackHelper.java | 2 +-
.../hudi/testutils/HoodieWriteableTestTable.java | 2 +-
.../apache/hudi/util/TestOperationConverter.java} | 37 +-
.../client/FlinkStreamingMetadataWriteHandler.java | 30 +-
.../hudi/client/HoodieFlinkTableServiceClient.java | 4 +-
.../apache/hudi/client/HoodieFlinkWriteClient.java | 25 +
.../java/org/apache/hudi/io/FlinkCreateHandle.java | 2 +-
.../io/storage/row/HoodieRowDataCreateHandle.java | 4 +-
.../commit/FlinkPartitionTTLActionExecutor.java | 2 +-
.../apache/hudi/client/TestFlinkWriteClient.java | 98 +++-
.../commit/BaseJavaCommitActionExecutor.java | 4 +-
.../table/action/commit/JavaUpsertPartitioner.java | 18 +-
.../BaseJavaDeltaCommitActionExecutor.java | 4 +-
.../hudi/client/TestJavaHoodieBackedMetadata.java | 12 +-
.../common/TestMultipleHoodieJavaWriteClient.java | 26 +-
...stJavaBulkInsertInternalPartitionerFactory.java | 53 ++
.../commit/TestJavaCopyOnWriteActionExecutor.java | 2 +-
.../testutils/HoodieJavaClientTestHarness.java | 2 +-
...SparkBinaryCopyClusteringExecutionStrategy.java | 2 +-
...arkExternalFileClusteringExecutionStrategy.java | 2 +-
.../client/common/SparkReaderContextFactory.java | 5 +
.../client/validator/SparkPreCommitValidator.java | 2 +-
.../SqlQueryEqualityPreCommitValidator.java | 8 +-
.../SqlQueryInequalityPreCommitValidator.java | 8 +-
.../SqlQuerySingleResultPreCommitValidator.java | 4 +-
.../hudi/io/storage/BlobDescriptorTransform.java | 24 +-
.../apache/hudi/io/storage/BlobStructLayout.java | 46 ++
.../hudi/io/storage/HoodieSparkLanceReader.java | 16 +-
.../hudi/io/storage/HoodieSparkLanceWriter.java | 58 +-
.../hudi/io/storage/LanceRecordIterator.java | 172 ++++--
.../hudi/io/storage/row/HoodieRowCreateHandle.java | 2 +-
.../bootstrap/BaseBootstrapMetadataHandler.java | 2 +-
.../SparkBootstrapCommitActionExecutor.java | 8 +-
.../hudi/BaseSparkInternalRecordContext.java | 11 -
.../TestHoodieSparkEngineDynamicRepartition.java | 2 +-
.../common/TestSparkReaderContextFactory.java | 6 +
.../bloom/TestHoodieBloomFilterProbingResult.java | 64 ++
.../apache/hudi/io/TestHoodieTimelineArchiver.java | 2 +-
.../action/commit/TestSparkBucketInfoGetter.java | 68 +++
.../TestTimelineServerBasedWriteMarkers.java | 4 +-
.../java/org/apache/hudi/avro/AvroSchemaUtils.java | 4 -
.../hudi/common/bloom/InternalBloomFilter.java | 89 +--
.../hudi/common/config/HoodieReaderConfig.java | 9 +-
.../apache/hudi/common/engine/RecordContext.java | 4 -
.../hudi/common/model/HoodieCommitMetadata.java | 2 +-
.../apache/hudi/common/schema/HoodieSchema.java | 7 +-
.../common/table/read/HoodieFileGroupReader.java | 2 +-
.../hudi/common/table/timeline/TimelineUtils.java | 2 +-
.../table/view/RocksDbBasedFileSystemView.java | 10 +-
.../org/apache/hudi/common/util/ConfigUtils.java | 2 +-
.../org/apache/hudi/expression/BindVisitor.java | 2 +-
.../org/apache/hudi/expression/Predicates.java | 10 +-
.../schema/utils/AvroSchemaEvolutionUtils.java | 26 +-
.../apache/hudi/metrics/JmxMetricsReporter.java | 8 +-
.../hudi/metrics/MetricsReporterFactory.java | 2 +-
.../apache/hudi/TestReportJvmConfiguration.java | 18 +-
.../common/bloom/InternalBloomFilterBenchmark.java | 127 ++++
.../hudi/common/bloom/TestInternalBloomFilter.java | 234 ++++++++
.../minicluster/ZookeeperTestService.java | 6 +-
.../apache/hudi/expression/TestBindVisitor.java | 53 ++
.../org/apache/hudi/expression/TestPredicates.java | 21 +
.../schema/utils/TestAvroSchemaEvolutionUtils.java | 6 +-
.../examples/spark/HoodieWriteClientExample.java | 8 +-
.../hudi/configuration/OptionsInference.java | 3 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 28 +-
.../sink/bucket/BucketBulkInsertWriterHelper.java | 2 +-
.../hudi/sink/bulk/BulkInsertWriterHelper.java | 6 +-
.../hudi/sink/clustering/ClusteringCommitSink.java | 2 +-
.../sink/clustering/ClusteringPlanOperator.java | 6 +-
.../sink/clustering/HoodieFlinkClusteringJob.java | 6 +-
.../hudi/sink/compact/HoodieFlinkCompactor.java | 8 +-
.../hudi/sink/partitioner/BucketAssigner.java | 2 +-
.../org/apache/hudi/sink/utils/EventBuffers.java | 10 +-
.../sink/v2/clustering/ClusteringCommitSinkV2.java | 2 +-
.../sink/v2/compact/CompactionCommitSinkV2.java | 2 +-
.../apache/hudi/source/IncrementalInputSplits.java | 7 +-
.../org/apache/hudi/table/HoodieTableSource.java | 2 +-
.../hudi/table/lookup/HoodieLookupFunction.java | 2 +-
.../main/java/org/apache/hudi/util/ClientIds.java | 2 +-
.../java/org/apache/hudi/util/ClusteringUtil.java | 2 +-
.../java/org/apache/hudi/util/CompactionUtil.java | 4 +-
.../apache/hudi/util/ViewStorageProperties.java | 2 +-
.../sink/TestStreamWriteOperatorCoordinator.java | 94 ++-
.../org/apache/hudi/sink/TestWriteMergeOnRead.java | 35 ++
.../apache/hudi/table/ITTestHoodieDataSource.java | 119 ++++
.../vector/reader/BaseVectorizedColumnReader.java | 7 +-
.../cow/vector/reader/NestedColumnReader.java | 33 +-
.../vector/reader/BaseVectorizedColumnReader.java | 7 +-
.../cow/vector/reader/NestedColumnReader.java | 33 +-
.../vector/reader/BaseVectorizedColumnReader.java | 7 +-
.../cow/vector/reader/NestedColumnReader.java | 33 +-
.../vector/reader/BaseVectorizedColumnReader.java | 7 +-
.../cow/vector/reader/NestedColumnReader.java | 33 +-
.../vector/reader/BaseVectorizedColumnReader.java | 7 +-
.../cow/vector/reader/NestedColumnReader.java | 33 +-
.../common/config/DFSPropertiesConfiguration.java | 4 +-
.../io/storage/hadoop/HoodieAvroHFileWriter.java | 2 +-
.../parquet/io/HoodieParquetFileBinaryCopier.java | 2 +-
.../table/view/TestHoodieTableFileSystemView.java | 2 +-
.../table/view/TestIncrementalFSViewSync.java | 10 +-
.../testutils/minicluster/HdfsTestService.java | 4 +-
.../BootstrapColumnStichingRecordReader.java | 2 +-
.../hadoop/HoodieCopyOnWriteTableInputFormat.java | 2 +-
.../hudi/hadoop/HoodieParquetInputFormat.java | 4 +-
.../org/apache/hudi/hadoop/InputPathHandler.java | 2 +-
.../hadoop/hive/HoodieCombineHiveInputFormat.java | 18 +-
.../realtime/AbstractRealtimeRecordReader.java | 10 +-
.../realtime/HoodieHFileRealtimeInputFormat.java | 7 +-
.../realtime/HoodieParquetRealtimeInputFormat.java | 7 +-
.../realtime/HoodieRealtimeRecordReader.java | 2 +-
.../realtime/RealtimeCompactedRecordReader.java | 4 +-
.../apache/hudi/hadoop/utils/HoodieHiveUtils.java | 6 +-
.../utils/HoodieRealtimeInputFormatUtils.java | 4 +-
hudi-integ-test/pom.xml | 48 +-
.../integ2/testcontainers/ContainerProvider.java | 22 +-
.../testcontainers/ITTestBaseTestcontainers.java | 236 ++++++++
.../testcontainers/ITTestCustomTypeHiveSync.java | 266 +++++++++
.../testcontainers/TestcontainersConfig.java | 110 ++++
.../testcontainers/command/CommandExecutor.java | 87 +++
.../testcontainers/command/CommandResult.java | 137 +++++
.../integ2/testcontainers/service/HiveService.java | 72 +++
.../testcontainers/service/SparkService.java | 71 +++
.../org/apache/hudi/common/metrics/Registry.java | 5 +-
.../apache/hudi/common/util/ReflectionUtils.java | 4 +-
.../apache/hudi/BaseHoodiePartitionValues.scala} | 11 +-
.../scala/org/apache/hudi/HoodieFileScanRDD.scala} | 10 +-
.../scala/org/apache/hudi/HoodieSchemaUtils.scala | 8 +-
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 12 +-
.../sql/BaseHoodieCatalystExpressionUtils.scala} | 49 +-
.../spark/sql/BaseHoodieCatalystPlanUtils.scala | 82 ++-
.../sql/HoodieSpark3CatalystExpressionUtils.scala | 37 --
.../org/apache/spark/sql/avro/AvroUtils.scala | 45 +-
...uning.scala => HoodieNestedSchemaPruning.scala} | 49 +-
.../datasources/lance/SparkLanceReaderBase.scala | 40 +-
.../datasources/orc/SparkOrcReaderBase.scala | 76 ++-
.../hudi/BaseResolveHudiAlterTableCommand.scala} | 43 +-
...stSparkFileFormatInternalRowReaderContext.scala | 26 +-
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 34 +-
.../procedures/ExportInstantsProcedure.scala | 14 +-
.../procedures/ProcedureParameterImpl.scala | 5 +-
.../hudi/client/TestHoodieClientMultiWriter.java | 12 +-
...DataValidationCheckForLogCompactionActions.java | 8 +-
.../io/storage/TestHoodieSparkLanceWriter.java | 95 +++
.../hudi/functional/TestLanceDataSource.scala | 644 ++++++++++++++++++++-
.../hudi/functional/TestStreamingSource.scala | 66 ++-
.../hudi/functional/TestStructuredStreaming.scala | 46 ++
.../spark/sql/hudi/ddl/TestCreateTable.scala | 10 +
.../sql/hudi/feature/TestDataSkippingQuery.scala | 62 ++
.../sql/hudi/feature/index/TestIndexSyntax.scala | 7 +-
.../procedure/TestExportInstantsProcedure.scala | 122 +++-
.../apache/hudi/Spark3HoodiePartitionValues.scala | 81 +--
.../spark/sql/HoodieSpark3CatalystPlanUtils.scala | 28 +-
.../spark/sql/HoodieSpark3SchemaUtils.scala} | 16 +-
.../spark/sql/adapter/BaseSpark3Adapter.scala | 12 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 7 +-
.../org/apache/spark/sql/avro/AvroSerializer.scala | 9 +-
.../hudi/Spark3ResolveHudiAlterTableCommand.scala | 43 ++
.../org/apache/hudi/Spark33HoodieFileScanRDD.scala | 36 --
.../sql/HoodieSpark33CatalystExpressionUtils.scala | 81 +--
.../spark/sql/HoodieSpark33CatalystPlanUtils.scala | 103 +---
.../spark/sql/HoodieSpark33SchemaUtils.scala | 14 +-
.../apache/spark/sql/adapter/Spark3_3Adapter.scala | 16 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 531 -----------------
.../org/apache/spark/sql/avro/AvroSerializer.scala | 490 ----------------
.../datasources/Spark33NestedSchemaPruning.scala | 62 --
.../datasources/orc/Spark33OrcReader.scala | 85 ---
.../parquet/Spark33DataSourceUtils.scala | 77 ---
.../hudi/Spark33ResolveHudiAlterTableCommand.scala | 67 ---
.../org/apache/hudi/Spark34HoodieFileScanRDD.scala | 36 --
.../sql/HoodieSpark34CatalystExpressionUtils.scala | 81 +--
.../spark/sql/HoodieSpark34CatalystPlanUtils.scala | 94 +--
.../spark/sql/HoodieSpark34SchemaUtils.scala | 14 +-
.../apache/spark/sql/adapter/Spark3_4Adapter.scala | 16 +-
.../org/apache/spark/sql/avro/AvroUtils.scala | 227 --------
.../datasources/Spark34NestedSchemaPruning.scala | 62 --
.../datasources/orc/Spark34OrcReader.scala | 91 ---
.../parquet/Spark34DataSourceUtils.scala | 77 ---
.../hudi/Spark34ResolveHudiAlterTableCommand.scala | 67 ---
.../org/apache/hudi/Spark35HoodieFileScanRDD.scala | 36 --
.../sql/HoodieSpark35CatalystExpressionUtils.scala | 81 +--
.../spark/sql/HoodieSpark35CatalystPlanUtils.scala | 103 +---
.../spark/sql/HoodieSpark35SchemaUtils.scala | 16 +-
.../apache/spark/sql/adapter/Spark3_5Adapter.scala | 16 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 531 -----------------
.../org/apache/spark/sql/avro/AvroSerializer.scala | 489 ----------------
.../org/apache/spark/sql/avro/AvroUtils.scala | 227 --------
.../datasources/Spark35NestedSchemaPruning.scala | 61 --
.../datasources/orc/Spark35OrcReader.scala | 92 ---
.../parquet/Spark35DataSourceUtils.scala | 76 ---
...Spark4HoodiePartitionCDCFileGroupMapping.scala} | 11 +-
.../Spark4HoodiePartitionFileSliceMapping.scala} | 13 +-
.../apache/hudi/Spark4HoodiePartitionValues.scala} | 21 +-
.../client/model/Spark4HoodieInternalRow.scala} | 29 +-
.../sql/HoodieSpark4CatalystExpressionUtils.scala | 45 +-
.../spark/sql/HoodieSpark4CatalystPlanUtils.scala | 66 +++
.../spark/sql/HoodieSpark4SchemaUtils.scala} | 4 +-
.../spark/sql/adapter/BaseSpark4Adapter.scala | 25 +-
.../HoodieSpark4PartitionedFileUtils.scala} | 4 +-
.../hudi/Spark4ResolveHudiAlterTableCommand.scala | 29 +-
.../sql/hudi/analysis/HoodieSpark4Analysis.scala} | 110 +---
...Spark40HoodiePartitionCDCFileGroupMapping.scala | 11 +-
.../Spark40HoodiePartitionFileSliceMapping.scala | 11 +-
.../apache/hudi/Spark40HoodiePartitionValues.scala | 85 +--
.../client/model/Spark40HoodieInternalRow.scala | 25 +-
.../sql/HoodieSpark40CatalystExpressionUtils.scala | 99 +---
.../spark/sql/HoodieSpark40CatalystPlanUtils.scala | 123 +---
.../spark/sql/HoodieSpark40SchemaUtils.scala | 31 +-
.../apache/spark/sql/adapter/Spark4_0Adapter.scala | 32 +-
.../org/apache/spark/sql/avro/AvroUtils.scala | 227 --------
.../datasources/Spark40NestedSchemaPruning.scala | 61 --
.../datasources/orc/Spark40OrcReader.scala | 92 ---
.../parquet/Spark40DataSourceUtils.scala | 76 ---
.../hudi/Spark40ResolveHudiAlterTableCommand.scala | 68 ---
.../sql/hudi/analysis/HoodieSpark40Analysis.scala | 122 +---
.../org/apache/hudi/Spark41HoodieFileScanRDD.scala | 36 --
...Spark41HoodiePartitionCDCFileGroupMapping.scala | 9 +-
.../Spark41HoodiePartitionFileSliceMapping.scala | 11 +-
.../apache/hudi/Spark41HoodiePartitionValues.scala | 86 +--
.../client/model/Spark41HoodieInternalRow.scala | 19 +-
.../sql/HoodieSpark41CatalystExpressionUtils.scala | 99 +---
.../spark/sql/HoodieSpark41CatalystPlanUtils.scala | 123 +---
.../spark/sql/HoodieSpark41SchemaUtils.scala | 31 +-
.../apache/spark/sql/adapter/Spark4_1Adapter.scala | 33 +-
.../org/apache/spark/sql/avro/AvroUtils.scala | 227 --------
.../HoodieSpark41PartitionedFileUtils.scala | 68 ---
.../datasources/Spark41NestedSchemaPruning.scala | 61 --
.../datasources/orc/Spark41OrcReader.scala | 92 ---
.../parquet/Spark41DataSourceUtils.scala | 76 ---
.../hudi/Spark41ResolveHudiAlterTableCommand.scala | 68 ---
.../sql/hudi/analysis/HoodieSpark41Analysis.scala | 121 +---
.../java/org/apache/hudi/hive/HiveSyncTool.java | 2 +-
.../hudi/hive/ddl/QueryBasedDDLExecutor.java | 4 +-
.../io/trino/plugin/hudi/TestHudiSmokeTest.java | 2 +-
.../hudi/utilities/HoodieDataTableValidator.java | 4 +-
.../utilities/sources/helpers/DFSPathSelector.java | 5 +-
.../sources/helpers/S3EventsMetaSelector.java | 2 +-
.../functional/TestHoodieSnapshotExporter.java | 2 +-
.../offlinejob/HoodieOfflineJobTestBase.java | 6 +-
pom.xml | 3 +
315 files changed, 5856 insertions(+), 7489 deletions(-)
copy docker/compose/{docker-compose_hadoop340_hive2310_spark402_amd64.yml =>
docker-compose_hadoop340_hive2310_spark411_amd64.yml} (97%)
copy docker/compose/{docker-compose_hadoop340_hive2310_spark402_amd64.yml =>
docker-compose_hadoop340_hive2310_spark411_arm64.yml} (97%)
create mode 100644 docker/demo/sparksql-blob-type-df.commands
create mode 100644 docker/demo/sparksql-blob-type-sql.commands
create mode 100644 docker/demo/sparksql-variant-type-df.commands
create mode 100644 docker/demo/sparksql-variant-type-sql.commands
create mode 100644 docker/demo/sparksql-vector-type-df.commands
create mode 100644 docker/demo/sparksql-vector-type-sql.commands
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/TestClientStatsPojos.java
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/bootstrap/translator/TestBootstrapPartitionPathTranslators.java
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/lock/TestLockResultEnums.java
copy
hudi-common/src/test/java/org/apache/hudi/common/util/TestBase64CodecUtil.java
=>
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/lock/audit/TestAuditOperationState.java
(52%)
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/exception/TestClientExceptions.java
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/execution/bulkinsert/TestBulkInsertSortMode.java
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/index/bloom/TestBloomIndexFileInfo.java
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/action/commit/TestBucketTypeAndInfo.java
copy
hudi-client/hudi-client-common/src/{main/java/org/apache/hudi/table/storage/HoodieDefaultLayout.java
=> test/java/org/apache/hudi/util/TestOperationConverter.java} (51%)
create mode 100644
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/execution/bulkinsert/TestJavaBulkInsertInternalPartitionerFactory.java
create mode 100644
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/BlobStructLayout.java
create mode 100644
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/index/bloom/TestHoodieBloomFilterProbingResult.java
create mode 100644
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestSparkBucketInfoGetter.java
create mode 100644
hudi-common/src/test/java/org/apache/hudi/common/bloom/InternalBloomFilterBenchmark.java
create mode 100644
hudi-common/src/test/java/org/apache/hudi/common/bloom/TestInternalBloomFilter.java
create mode 100644
hudi-common/src/test/java/org/apache/hudi/expression/TestBindVisitor.java
copy
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieProducer.java
=>
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/ContainerProvider.java
(58%)
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/ITTestBaseTestcontainers.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/ITTestCustomTypeHiveSync.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/TestcontainersConfig.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/command/CommandExecutor.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/command/CommandResult.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/service/HiveService.java
create mode 100644
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/service/SparkService.java
copy
hudi-spark-datasource/{hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionValues.scala
=>
hudi-spark-common/src/main/scala/org/apache/hudi/BaseHoodiePartitionValues.scala}
(91%)
rename
hudi-spark-datasource/{hudi-spark4.0.x/src/main/scala/org/apache/hudi/Spark40HoodieFileScanRDD.scala
=> hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileScanRDD.scala}
(77%)
copy
hudi-spark-datasource/{hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark40CatalystExpressionUtils.scala
=>
hudi-spark-common/src/main/scala/org/apache/spark/sql/BaseHoodieCatalystExpressionUtils.scala}
(74%)
delete mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala
rename hudi-spark-datasource/{hudi-spark3.3.x =>
hudi-spark-common}/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
(85%)
rename
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/{BaseHoodieNestedSchemaPruning.scala
=> HoodieNestedSchemaPruning.scala} (77%)
rename
hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/hudi/Spark35ResolveHudiAlterTableCommand.scala
=>
hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/BaseResolveHudiAlterTableCommand.scala}
(52%)
copy
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/parser/HoodieExtendedParserInterface.scala
=>
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala
(53%)
copy
hudi-spark-datasource/{hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/HoodieSpark34SchemaUtils.scala
=>
hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3SchemaUtils.scala}
(70%)
rename hudi-spark-datasource/{hudi-spark3.4.x =>
hudi-spark3-common}/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
(97%)
rename hudi-spark-datasource/{hudi-spark3.4.x =>
hudi-spark3-common}/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
(97%)
create mode 100644
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/hudi/Spark3ResolveHudiAlterTableCommand.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/hudi/Spark33HoodieFileScanRDD.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark33NestedSchemaPruning.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark33OrcReader.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark33DataSourceUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/hudi/Spark33ResolveHudiAlterTableCommand.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/hudi/Spark34HoodieFileScanRDD.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark34NestedSchemaPruning.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark34OrcReader.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark34DataSourceUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/hudi/Spark34ResolveHudiAlterTableCommand.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/hudi/Spark35HoodieFileScanRDD.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark35NestedSchemaPruning.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark35OrcReader.scala
delete mode 100644
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark35DataSourceUtils.scala
copy
hudi-spark-datasource/{hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionCDCFileGroupMapping.scala
=>
hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionCDCFileGroupMapping.scala}
(75%)
copy
hudi-spark-datasource/{hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionFileSliceMapping.scala
=>
hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionFileSliceMapping.scala}
(77%)
copy
hudi-spark-datasource/{hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionFileSliceMapping.scala
=>
hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionValues.scala}
(62%)
copy
hudi-spark-datasource/{hudi-spark4.1.x/src/main/scala/org/apache/hudi/client/model/Spark41HoodieInternalRow.scala
=>
hudi-spark4-common/src/main/scala/org/apache/hudi/client/model/Spark4HoodieInternalRow.scala}
(67%)
create mode 100644
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/HoodieSpark4CatalystPlanUtils.scala
copy
hudi-spark-datasource/{hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark40SchemaUtils.scala
=>
hudi-spark4-common/src/main/scala/org/apache/spark/sql/HoodieSpark4SchemaUtils.scala}
(93%)
rename
hudi-spark-datasource/{hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/HoodieSpark40PartitionedFileUtils.scala
=>
hudi-spark4-common/src/main/scala/org/apache/spark/sql/execution/datasources/HoodieSpark4PartitionedFileUtils.scala}
(96%)
copy
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/spark/internal/ReflectUtil.java
=>
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/Spark4ResolveHudiAlterTableCommand.scala
(50%)
copy
hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark35Analysis.scala
=>
hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark4Analysis.scala}
(55%)
delete mode 100644
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark40NestedSchemaPruning.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark40OrcReader.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40DataSourceUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/Spark40ResolveHudiAlterTableCommand.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/hudi/Spark41HoodieFileScanRDD.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/HoodieSpark41PartitionedFileUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark41NestedSchemaPruning.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark41OrcReader.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41DataSourceUtils.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/hudi/Spark41ResolveHudiAlterTableCommand.scala
