This is an automated email from the ASF dual-hosted git repository. ashvin pushed a change to branch 345-read-and-translate-the-deletion-vectors-in-delta-source-table-to-xtables-internal-representation in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
omit 2e234ae9 Refactor: rename to clariy delete ordinal only omit b65080ee Add support for inline delete vectors omit 92712a6b Fix spotless error, unused import omit 7786c5bd Handle optional offset information omit 5f4f8515 Extract Delta Lake deletion vectors add 14967dd6 Add community sync page for XTable add b463ae6e Update GMeet link and add today's recording add c9deec1a [590] Add RFC for XCatalogSync - Synchronize tables across catalogs add 9fa812e3 [590] Add RunCatalogSync utility for synchronizing tables across catalogs add d3bbe5fe Delta source: add record count for DataFile add 79c68def [site] Update blogs add 8b0b5eaf [590] Add Iceberg Glue Catalog Sync implementation add 30f3e544 [590] Add Iceberg HMS Catalog Sync implementation add f2045be9 [590] Add Delta Glue Catalog Sync implementation add e79a2d34 [590] Add Delta HMS Catalog Sync implementation add 2e71e156 [590] Changes in xtable-core for Hudi Catalog Sync add 67946b98 Update conversion sources add b3e03e55 [40] Add commit-level metadata and source-target id mapping add f194f4cb [590] Add Hudi HMS Catalog Sync Implementation add 03e5b022 [590] Add Hudi Glue Catalog Sync Implementation add dc209962 Add change proposal for conversion of deletion vectors add c76692b8 Add base representation of storage files new 5036f3b5 Extract Delta Lake deletion vectors This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2e234ae9) \ N -- N -- N refs/heads/345-read-and-translate-the-deletion-vectors-in-delta-source-table-to-xtables-internal-representation (5036f3b5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: README.md | 2 +- assets/images/catalog_sync_flow.jpg | Bin 0 -> 133357 bytes pom.xml | 20 +- rfc/rfc-1/rfc-1.md | 139 ++++ rfc/rfc-2/2 - Deletion Info Conversion.md | 227 ++++++ website/README.md | 4 + website/blog/OneTable-is-now-Apache-XTable.md | 1 + website/blog/xtable-in-aws-lambda.mdx | 19 + website/community/sync.md | 34 + website/docusaurus.config.js | 9 + .../xtable-in-aws-lambda/xtable-in-aws-lambda.png | Bin 0 -> 115121 bytes website/static/index.html | 2 + .../apache/xtable/conversion/ConversionConfig.java | 12 +- .../xtable/conversion/ExternalCatalogConfig.java | 65 ++ .../xtable/conversion/TargetCatalogConfig.java | 21 +- .../org/apache/xtable/model/InternalSnapshot.java | 3 + .../java/org/apache/xtable/model/TableChange.java | 23 +- .../apache/xtable/model/exception/ErrorCode.java | 3 +- .../xtable/model/metadata/TableSyncMetadata.java | 22 +- .../org/apache/xtable/model/stat/FileStats.java | 15 +- .../storage/{FileFormat.java => CatalogType.java} | 10 +- .../org/apache/xtable/model/storage/FilesDiff.java | 6 +- .../xtable/model/storage/InternalDataFile.java | 19 +- .../model/storage/InternalDeletionVector.java | 26 +- .../apache/xtable/model/storage/InternalFile.java | 56 ++ .../{DataFilesDiff.java => InternalFilesDiff.java} | 38 +- .../xtable/model/storage/PartitionFileGroup.java | 12 +- .../spi/extractor/CatalogConversionSource.java | 9 + .../xtable/spi/extractor/ConversionSource.java | 14 + .../org/apache/xtable/spi/sync/CatalogSync.java | 12 +- .../apache/xtable/spi/sync/CatalogSyncClient.java | 9 + .../apache/xtable/spi/sync/ConversionTarget.java | 16 +- .../apache/xtable/spi/sync/TableFormatSync.java | 23 +- .../model/metadata/TestTableSyncMetadata.java | 20 +- .../xtable/model/storage/TestDataFilesDiff.java | 8 +- .../apache/xtable/model/storage/TestFilesDiff.java | 2 +- .../spi/extractor/TestExtractFromSource.java | 20 +- .../xtable/spi/sync/TestTableFormatSync.java | 170 +++-- {xtable-api => xtable-aws}/pom.xml | 65 +- .../xtable/glue/DefaultGlueClientFactory.java | 80 +++ .../org/apache/xtable/glue/GlueCatalogConfig.java | 102 +++ .../xtable/glue/GlueCatalogConversionSource.java | 100 +++ .../glue/GlueCatalogPartitionSyncOperations.java | 331 +++++++++ .../apache/xtable/glue/GlueCatalogSyncClient.java | 268 +++++++ .../glue/GlueCatalogTableBuilderFactory.java | 48 ++ .../apache/xtable/glue/GlueCatalogTableUtils.java | 56 ++ .../org/apache/xtable/glue/GlueClientFactory.java | 23 +- .../apache/xtable/glue/GlueSchemaExtractor.java | 272 +++++++ .../glue/table/DeltaGlueCatalogTableBuilder.java | 116 +++ .../glue/table/HudiGlueCatalogTableBuilder.java | 185 +++++ .../glue/table/IcebergGlueCatalogTableBuilder.java | 121 ++++ ...he.xtable.spi.extractor.CatalogConversionSource | 5 +- .../org.apache.xtable.spi.sync.CatalogSyncClient | 5 +- .../xtable/glue/GlueCatalogSyncTestBase.java | 238 ++++++ .../apache/xtable/glue/TestGlueCatalogConfig.java | 125 ++++ .../glue/TestGlueCatalogConversionSource.java | 193 +++++ .../TestGlueCatalogPartitionSyncOperations.java | 519 +++++++++++++ .../xtable/glue/TestGlueCatalogSyncClient.java | 463 ++++++++++++ .../xtable/glue/TestGlueSchemaExtractor.java | 799 +++++++++++++++++++++ .../table/TestDeltaGlueCatalogTableBuilder.java | 106 +++ .../table/TestHudiGlueCatalogTableBuilder.java | 125 ++++ .../table/TestIcebergGlueCatalogTableBuilder.java | 150 ++++ xtable-core/pom.xml | 30 + .../xtable/catalog/CatalogConversionFactory.java | 102 +++ .../apache/xtable/catalog/CatalogPartition.java | 31 +- .../xtable/catalog/CatalogPartitionEvent.java | 49 ++ .../catalog/CatalogPartitionSyncOperations.java | 99 +++ .../xtable/catalog/CatalogPartitionSyncTool.java | 25 +- .../apache/xtable/catalog/CatalogTableBuilder.java | 20 +- .../org/apache/xtable/catalog/CatalogUtils.java | 69 ++ .../java/org/apache/xtable/catalog/Constants.java | 49 ++ .../apache/xtable/catalog/TableFormatUtils.java | 89 +++ .../xtable/conversion/ConversionController.java | 212 ++++-- .../apache/xtable/conversion/ConversionUtils.java | 23 +- .../apache/xtable/delta/DeltaActionsConverter.java | 19 +- .../apache/xtable/delta/DeltaConversionSource.java | 24 +- .../apache/xtable/delta/DeltaConversionTarget.java | 74 +- .../delta/DeltaDataFileUpdatesExtractor.java | 19 +- .../apache/xtable/delta/DeltaSchemaExtractor.java | 96 --- .../apache/xtable/delta/DeltaStatsExtractor.java | 52 +- ...eadException.java => CatalogSyncException.java} | 20 +- .../xtable/hudi/BaseFileUpdatesExtractor.java | 17 +- .../apache/xtable/hudi/HudiConversionSource.java | 7 + .../apache/xtable/hudi/HudiConversionTarget.java | 76 +- .../apache/xtable/hudi/HudiDataFileExtractor.java | 6 +- .../org/apache/xtable/hudi/HudiTableManager.java | 4 +- .../hudi/catalog/HudiCatalogPartitionSyncTool.java | 471 ++++++++++++ .../HudiCatalogTablePropertiesExtractor.java | 148 ++++ .../xtable/hudi/catalog/HudiInputFormatUtils.java | 112 +++ .../xtable/iceberg/IcebergCatalogConfig.java | 8 +- .../xtable/iceberg/IcebergConversionSource.java | 21 +- .../xtable/iceberg/IcebergConversionTarget.java | 45 +- .../xtable/iceberg/IcebergDataFileUpdatesSync.java | 40 +- .../apache/xtable/reflection/ReflectionUtils.java | 42 +- .../apache/xtable/schema/SparkSchemaExtractor.java | 133 ++++ .../org/apache/xtable/ValidationTestHelper.java | 14 +- .../catalog/TestCatalogConversionFactory.java | 108 +++ .../apache/xtable/catalog/TestCatalogUtils.java | 65 ++ .../xtable/catalog/TestSchemaExtractorBase.java | 92 +++ .../xtable/catalog/TestTableFormatUtils.java | 102 +++ .../conversion/TestConversionController.java | 231 +++++- .../xtable/delta/ITDeltaConversionSource.java | 2 +- .../xtable/delta/ITDeltaDeleteVectorConvert.java | 12 +- .../xtable/delta/TestDeltaActionsConverter.java | 18 +- .../xtable/delta/TestDeltaSchemaExtractor.java | 134 ---- .../xtable/delta/TestDeltaStatsExtractor.java | 29 +- .../org/apache/xtable/delta/TestDeltaSync.java | 78 +- .../apache/xtable/hudi/ITHudiConversionTarget.java | 174 ++++- .../xtable/hudi/TestBaseFileUpdatesExtractor.java | 6 +- .../xtable/hudi/TestHudiConversionTarget.java | 7 +- .../catalog/TestHudiCatalogPartitionSyncTool.java | 323 +++++++++ .../TestHudiCatalogTablePropertiesExtractor.java | 182 +++++ .../hudi/catalog/TestHudiInputFormatUtils.java | 118 +++ .../iceberg/TestIcebergConversionSource.java | 4 +- .../org/apache/xtable/iceberg/TestIcebergSync.java | 123 +++- .../TestSparkSchemaExtractor.java} | 258 +------ .../org/apache/xtable/testutil/ITTestUtils.java | 117 +++ ...e.xtable.spi.extractor.CatalogConversionSource} | 5 +- .../org.apache.xtable.spi.sync.CatalogSyncClient} | 5 +- xtable-hive-metastore/pom.xml | 207 ++++++ .../org/apache/xtable/hms/HMSCatalogConfig.java | 65 ++ .../xtable/hms/HMSCatalogConversionSource.java | 111 +++ .../hms/HMSCatalogPartitionSyncOperations.java | 221 ++++++ .../apache/xtable/hms/HMSCatalogSyncClient.java | 240 +++++++ .../xtable/hms/HMSCatalogTableBuilderFactory.java | 77 ++ .../org/apache/xtable/hms/HMSClientProvider.java | 65 ++ .../org/apache/xtable/hms/HMSSchemaExtractor.java | 143 ++++ .../hms/table/DeltaHMSCatalogTableBuilder.java | 95 +++ .../hms/table/HudiHMSCatalogTableBuilder.java | 185 +++++ .../hms/table/IcebergHMSCatalogTableBuilder.java | 126 ++++ ...he.xtable.spi.extractor.CatalogConversionSource | 5 +- .../org.apache.xtable.spi.sync.CatalogSyncClient | 5 +- .../apache/xtable/hms/HMSCatalogSyncTestBase.java | 188 +++++ .../apache/xtable/hms/TestHMSCatalogConfig.java | 64 ++ .../xtable/hms/TestHMSCatalogConversionSource.java | 181 +++++ .../hms/TestHMSCatalogPartitionSyncOperations.java | 470 ++++++++++++ .../xtable/hms/TestHMSCatalogSyncClient.java | 420 +++++++++++ .../hms/TestHMSCatalogTableBuilderFactory.java | 64 ++ .../apache/xtable/hms/TestHMSSchemaExtractor.java | 493 +++++++++++++ .../hms/table/TestDeltaHMSCatalogTableBuilder.java | 134 ++++ .../hms/table/TestHudiHMSCatalogTableBuilder.java | 107 +++ .../table/TestIcebergHMSCatalogTableBuilder.java | 185 +++++ xtable-utilities/pom.xml | 49 ++ .../apache/xtable/utilities/RunCatalogSync.java | 381 ++++++++++ .../java/org/apache/xtable/utilities/RunSync.java | 4 +- .../apache/xtable/utilities/ITRunCatalogSync.java | 142 ++++ .../xtable/utilities/TestRunCatalogSync.java | 21 +- .../src/test/resources/catalogConfig.yaml | 72 ++ 148 files changed, 13568 insertions(+), 977 deletions(-) create mode 100644 assets/images/catalog_sync_flow.jpg create mode 100644 rfc/rfc-1/rfc-1.md create mode 100644 rfc/rfc-2/2 - Deletion Info Conversion.md create mode 100644 website/blog/xtable-in-aws-lambda.mdx create mode 100644 website/community/sync.md create mode 100644 website/static/images/blog/xtable-in-aws-lambda/xtable-in-aws-lambda.png create mode 100644 xtable-api/src/main/java/org/apache/xtable/conversion/ExternalCatalogConfig.java copy xtable-core/src/main/java/org/apache/xtable/iceberg/IcebergCatalogConfig.java => xtable-api/src/main/java/org/apache/xtable/conversion/TargetCatalogConfig.java (64%) copy xtable-core/src/main/java/org/apache/xtable/hudi/HudiFileStats.java => xtable-api/src/main/java/org/apache/xtable/model/stat/FileStats.java (82%) copy xtable-api/src/main/java/org/apache/xtable/model/storage/{FileFormat.java => CatalogType.java} (80%) create mode 100644 xtable-api/src/main/java/org/apache/xtable/model/storage/InternalFile.java rename xtable-api/src/main/java/org/apache/xtable/model/storage/{DataFilesDiff.java => InternalFilesDiff.java} (62%) copy {xtable-api => xtable-aws}/pom.xml (64%) create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/DefaultGlueClientFactory.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogConfig.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogConversionSource.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogPartitionSyncOperations.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogSyncClient.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogTableBuilderFactory.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogTableUtils.java copy xtable-api/src/main/java/org/apache/xtable/model/storage/DataLayoutStrategy.java => xtable-aws/src/main/java/org/apache/xtable/glue/GlueClientFactory.java (65%) create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/GlueSchemaExtractor.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/table/DeltaGlueCatalogTableBuilder.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/table/HudiGlueCatalogTableBuilder.java create mode 100644 xtable-aws/src/main/java/org/apache/xtable/glue/table/IcebergGlueCatalogTableBuilder.java copy xtable-core/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => xtable-aws/src/main/resources/META-INF/services/org.apache.xtable.spi.extractor.CatalogConversionSource (86%) copy xtable-core/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => xtable-aws/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.CatalogSyncClient (86%) create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/GlueCatalogSyncTestBase.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/TestGlueCatalogConfig.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/TestGlueCatalogConversionSource.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/TestGlueCatalogPartitionSyncOperations.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/TestGlueCatalogSyncClient.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/TestGlueSchemaExtractor.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/table/TestDeltaGlueCatalogTableBuilder.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/table/TestHudiGlueCatalogTableBuilder.java create mode 100644 xtable-aws/src/test/java/org/apache/xtable/glue/table/TestIcebergGlueCatalogTableBuilder.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/CatalogConversionFactory.java copy xtable-api/src/main/java/org/apache/xtable/model/storage/DataLayoutStrategy.java => xtable-core/src/main/java/org/apache/xtable/catalog/CatalogPartition.java (52%) create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/CatalogPartitionEvent.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/CatalogPartitionSyncOperations.java copy xtable-api/src/main/java/org/apache/xtable/spi/extractor/CatalogConversionSource.java => xtable-core/src/main/java/org/apache/xtable/catalog/CatalogPartitionSyncTool.java (50%) copy xtable-api/src/main/java/org/apache/xtable/model/validation/ValidationChecker.java => xtable-core/src/main/java/org/apache/xtable/catalog/CatalogTableBuilder.java (62%) create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/CatalogUtils.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/Constants.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java copy xtable-api/src/main/java/org/apache/xtable/model/storage/TableFormat.java => xtable-core/src/main/java/org/apache/xtable/conversion/ConversionUtils.java (67%) copy xtable-core/src/main/java/org/apache/xtable/exception/{ReadException.java => CatalogSyncException.java} (63%) create mode 100644 xtable-core/src/main/java/org/apache/xtable/hudi/catalog/HudiCatalogPartitionSyncTool.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/hudi/catalog/HudiCatalogTablePropertiesExtractor.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/hudi/catalog/HudiInputFormatUtils.java create mode 100644 xtable-core/src/main/java/org/apache/xtable/schema/SparkSchemaExtractor.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/catalog/TestCatalogConversionFactory.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/catalog/TestCatalogUtils.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/catalog/TestSchemaExtractorBase.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/catalog/TestTableFormatUtils.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/hudi/catalog/TestHudiCatalogPartitionSyncTool.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/hudi/catalog/TestHudiCatalogTablePropertiesExtractor.java create mode 100644 xtable-core/src/test/java/org/apache/xtable/hudi/catalog/TestHudiInputFormatUtils.java copy xtable-core/src/test/java/org/apache/xtable/{delta/TestDeltaSchemaExtractor.java => schema/TestSparkSchemaExtractor.java} (73%) copy xtable-core/src/{main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => test/resources/META-INF/services/org.apache.xtable.spi.extractor.CatalogConversionSource} (86%) copy xtable-core/src/{main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => test/resources/META-INF/services/org.apache.xtable.spi.sync.CatalogSyncClient} (86%) create mode 100644 xtable-hive-metastore/pom.xml create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSCatalogConfig.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSCatalogConversionSource.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSCatalogPartitionSyncOperations.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSCatalogSyncClient.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSCatalogTableBuilderFactory.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSClientProvider.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/HMSSchemaExtractor.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/table/DeltaHMSCatalogTableBuilder.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/table/HudiHMSCatalogTableBuilder.java create mode 100644 xtable-hive-metastore/src/main/java/org/apache/xtable/hms/table/IcebergHMSCatalogTableBuilder.java copy xtable-core/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => xtable-hive-metastore/src/main/resources/META-INF/services/org.apache.xtable.spi.extractor.CatalogConversionSource (86%) copy xtable-core/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.ConversionTarget => xtable-hive-metastore/src/main/resources/META-INF/services/org.apache.xtable.spi.sync.CatalogSyncClient (86%) create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/HMSCatalogSyncTestBase.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSCatalogConfig.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSCatalogConversionSource.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSCatalogPartitionSyncOperations.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSCatalogSyncClient.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSCatalogTableBuilderFactory.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/TestHMSSchemaExtractor.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/table/TestDeltaHMSCatalogTableBuilder.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/table/TestHudiHMSCatalogTableBuilder.java create mode 100644 xtable-hive-metastore/src/test/java/org/apache/xtable/hms/table/TestIcebergHMSCatalogTableBuilder.java create mode 100644 xtable-utilities/src/main/java/org/apache/xtable/utilities/RunCatalogSync.java create mode 100644 xtable-utilities/src/test/java/org/apache/xtable/utilities/ITRunCatalogSync.java copy xtable-api/src/test/java/org/apache/xtable/conversion/TestTargetTable.java => xtable-utilities/src/test/java/org/apache/xtable/utilities/TestRunCatalogSync.java (64%) create mode 100644 xtable-utilities/src/test/resources/catalogConfig.yaml