[GitHub] [hudi] atmabdalla commented on issue #7145: [SUPPORT] `org.apache.avro.SchemaParseException: Can't redefine: array` when an Array containing a Struct is the only field in a Struct

2023-01-24 Thread via GitHub
atmabdalla commented on issue #7145: URL: https://github.com/apache/hudi/issues/7145#issuecomment-1402838875 @ssdong I think you have covered this issue on your [blog](https://susudong.wordpress.com/2019/03/18/issues-with-avro-schema-and-parquet-schema/)? -- This is an automated message f

[jira] [Updated] (HUDI-4937) Fix HoodieTable injecting HoodieBackedTableMetadata not reusing underlying MT readers

2023-01-24 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-4937: -- Fix Version/s: 0.13.1 (was: 0.13.0) > Fix HoodieTable injecting HoodieBac

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1402841570 ## CI report: * 6ab358154bb350a68340c9e8b9cafd0de260252c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1189

[GitHub] [hudi] hudi-bot commented on pull request #7576: [HUDI-4991] Allow kafka-like configs to set truststore and keystore for the SchemaProvider

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7576: URL: https://github.com/apache/hudi/pull/7576#issuecomment-1402842300 ## CI report: * 777e992653f094345bcadb5d0ad2125663f5340c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459

[GitHub] [hudi] yihua commented on a diff in pull request #7744: [HUDI-5276] Fix getting partition paths under relative paths

2023-01-24 Thread via GitHub
yihua commented on code in PR #7744: URL: https://github.com/apache/hudi/pull/7744#discussion_r1086059373 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java: ## @@ -150,9 +152,11 @@ protected Option> getRecordByKey(String key, } @Overri

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1402846258 ## CI report: * 6ab358154bb350a68340c9e8b9cafd0de260252c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1189

[GitHub] [hudi] hudi-bot commented on pull request #7576: [HUDI-4991] Allow kafka-like configs to set truststore and keystore for the SchemaProvider

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7576: URL: https://github.com/apache/hudi/pull/7576#issuecomment-1402847315 ## CI report: * 777e992653f094345bcadb5d0ad2125663f5340c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459

[GitHub] [hudi] yihua commented on a diff in pull request #7744: [HUDI-5276] Fix getting partition paths under relative paths

2023-01-24 Thread via GitHub
yihua commented on code in PR #7744: URL: https://github.com/apache/hudi/pull/7744#discussion_r1086084801 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadata.java: ## @@ -138,12 +138,15 @@ static HoodieBackedTableMetadata createHoodieBackedTableMetadata(Ho

[jira] [Created] (HUDI-5611) Revisit metadata-table-based file listing calls

2023-01-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5611: --- Summary: Revisit metadata-table-based file listing calls Key: HUDI-5611 URL: https://issues.apache.org/jira/browse/HUDI-5611 Project: Apache Hudi Issue Type: Improveme

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls by using batch lookup

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Summary: Revisit metadata-table-based file listing calls by using batch lookup (was: Revisit metadata-table

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls and use batch lookup instead

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Summary: Revisit metadata-table-based file listing calls and use batch lookup instead (was: Revisit metadat

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7379: [HUDI-5323] Support virtual keys in Bloom Index and always write bloom filters to parquet files

2023-01-24 Thread via GitHub
alexeykudinkin commented on code in PR #7379: URL: https://github.com/apache/hudi/pull/7379#discussion_r1086091096 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkParquetReader.java: ## @@ -79,8 +82,8 @@ public BloomFilter readBloomFilter() {

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls and use batch lookup instead

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Description: We discover a performance issue with savepoint when the metadata table is enabled. It is due t

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls and use batch lookup instead

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Description: We discover a performance issue with savepoint when the metadata table is enabled. It is due t

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls and use batch lookup instead

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Priority: Critical (was: Major) > Revisit metadata-table-based file listing calls and use batch lookup inst

[jira] [Updated] (HUDI-5611) Revisit metadata-table-based file listing calls and use batch lookup instead

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5611: Fix Version/s: 0.13.1 > Revisit metadata-table-based file listing calls and use batch lookup instead > -

[jira] [Created] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5612: --- Summary: Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView Key: HUDI-5612 URL: https://issues.apache.org/jira/browse/HUDI-5612 Projec

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Priority: Critical (was: Major) > Integrate metadata table with SpillableMapBasedFileSystemView and > Rock

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Fix Version/s: 0.13.1 > Integrate metadata table with SpillableMapBasedFileSystemView and > RocksDbBasedFil

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Description: Currently, metadata-table-based file listing is integrated through `HoodieMetadataFileSystemVi

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Description: Currently, metadata-table-based file listing is integrated through `HoodieMetadataFileSystemVie

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Description: Currently, metadata-table-based file listing is integrated through `HoodieMetadataFileSystemVi

[jira] [Updated] (HUDI-5612) Integrate metadata table with SpillableMapBasedFileSystemView and RocksDbBasedFileSystemView

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5612: Description: Currently, metadata-table-based file listing is integrated through HoodieMetadataFileSystemView

[GitHub] [hudi] yihua commented on a diff in pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
yihua commented on code in PR #7690: URL: https://github.com/apache/hudi/pull/7690#discussion_r1086092057 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java: ## @@ -84,15 +84,32 @@ public HoodieSavepointMetadata ex

[GitHub] [hudi] abhishekkh commented on issue #3835: Hudi deltastreamer using avro schema parser when using jsonKafkaSource

2023-01-24 Thread via GitHub
abhishekkh commented on issue #3835: URL: https://github.com/apache/hudi/issues/3835#issuecomment-1402905587 Sure will give it a try -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[jira] [Created] (HUDI-5613) Improve the abstraction of the table file system view

2023-01-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5613: --- Summary: Improve the abstraction of the table file system view Key: HUDI-5613 URL: https://issues.apache.org/jira/browse/HUDI-5613 Project: Apache Hudi Issue Type: Imp

[jira] [Updated] (HUDI-5613) Improve the abstraction of the table file system view

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5613: Description: AbstractTableFileSystemView does not contain HoodieEngineContext instance and instead relies o

[GitHub] [hudi] hudi-bot commented on pull request #7379: [HUDI-5323] Support virtual keys in Bloom Index and always write bloom filters to parquet files

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7379: URL: https://github.com/apache/hudi/pull/7379#issuecomment-1402914135 ## CI report: * aaf5fe979daf2820bc80f0dc7a57ae52d92b5c95 UNKNOWN * 2441c3cce239d422a32e250eafefb4601fa04bb5 UNKNOWN * 80ffa35ffacb35e53e97190dff387d4820dd78e0 Azure: [FAILUR

[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7668: URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402914764 ## CI report: * b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN * be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7740: [HUDI-5464] Suffix instant time for metadata partition init deltacommit

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7740: URL: https://github.com/apache/hudi/pull/7740#issuecomment-1402915037 ## CI report: * 405071a2cab10908738a7c54269aff189ac13f47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7379: [HUDI-5323] Support virtual keys in Bloom Index and always write bloom filters to parquet files

2023-01-24 Thread via GitHub
nsivabalan commented on code in PR #7379: URL: https://github.com/apache/hudi/pull/7379#discussion_r1086104736 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileWriterFactory.java: ## @@ -69,10 +66,8 @@ protected HoodieFileWriter newParquet

[GitHub] [hudi] hudi-bot commented on pull request #7746: [HUDI-5610] Fix hudi-cli-bundle startup conflict for spark 3.2.x

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7746: URL: https://github.com/apache/hudi/pull/7746#issuecomment-1402919687 ## CI report: * 3ad814fb1a4116b0256fe338b45618206d5efb24 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] hudi-bot commented on pull request #7642: [HUDI-5534] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7642: URL: https://github.com/apache/hudi/pull/7642#issuecomment-1402919384 ## CI report: * b11fa6b2246e4f02f1da12487093a9b5bfaf2149 UNKNOWN * 4437e6302deb849200d61526b61330e23f918162 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7709: URL: https://github.com/apache/hudi/pull/7709#issuecomment-1402919580 ## CI report: * 704d59a3c83e12862107f3f5904c5a90d4f771c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1456

[GitHub] [hudi] hudi-bot commented on pull request #7744: [HUDI-5276] Fix getting partition paths under relative paths

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7744: URL: https://github.com/apache/hudi/pull/7744#issuecomment-1402919653 ## CI report: * d20577002d7b9cb59fa6cdfb45ab1a7d63c91215 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460

[GitHub] [hudi] yihua commented on a diff in pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
yihua commented on code in PR #7690: URL: https://github.com/apache/hudi/pull/7690#discussion_r1086114031 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -81,6 +86,8 @@ public abstract class AbstractTableFileSystemView imple

[GitHub] [hudi] yihua commented on a diff in pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
yihua commented on code in PR #7690: URL: https://github.com/apache/hudi/pull/7690#discussion_r1086116287 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -289,6 +296,123 @@ private void clear() { */ protected abstract

[GitHub] [hudi] hudi-bot commented on pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7709: URL: https://github.com/apache/hudi/pull/7709#issuecomment-1402925015 ## CI report: * 704d59a3c83e12862107f3f5904c5a90d4f771c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1456

[GitHub] [hudi] hudi-bot commented on pull request #7744: [HUDI-5276] Fix getting partition paths under relative paths

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7744: URL: https://github.com/apache/hudi/pull/7744#issuecomment-1402925093 ## CI report: * d20577002d7b9cb59fa6cdfb45ab1a7d63c91215 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7738: [HUDI-5569] Fixing TableFileSystemView to detect early failed commits

2023-01-24 Thread via GitHub
nsivabalan commented on code in PR #7738: URL: https://github.com/apache/hudi/pull/7738#discussion_r1084642273 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java: ## @@ -295,4 +295,9 @@ public static Option getEarliestInstantForMetadataArchiva

[GitHub] [hudi] rahil-c commented on pull request #7746: [HUDI-5610] Fix hudi-cli-bundle startup conflict for spark 3.2.x

2023-01-24 Thread via GitHub
rahil-c commented on PR #7746: URL: https://github.com/apache/hudi/pull/7746#issuecomment-1402983415 Thanks Ethan I left a note in the special note section that the logging for spark 3.2.x is still not working correctly, whereas logging for spark 3.3.1 is working fine. -- This is an aut

[jira] [Updated] (HUDI-4937) Fix HoodieTable injecting HoodieBackedTableMetadata not reusing underlying MT readers

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4937: - Priority: Critical (was: Blocker) > Fix HoodieTable injecting HoodieBackedTableMetadata not reusing under

[jira] [Updated] (HUDI-5534) Optimize Bloom Index lookup DAG

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5534: - Priority: Blocker (was: Critical) > Optimize Bloom Index lookup DAG > --- > >

[jira] [Updated] (HUDI-5323) Decouple virtual key with writing bloom filters to parquet files

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5323: - Priority: Critical (was: Blocker) > Decouple virtual key with writing bloom filters to parquet files > --

[jira] [Updated] (HUDI-3636) Clustering fails due to marker creation failure

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3636: - Priority: Blocker (was: Critical) > Clustering fails due to marker creation failure > ---

[jira] [Updated] (HUDI-2681) Make hoodie record_key and preCombine_key optional

2023-01-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2681: -- Status: Patch Available (was: In Progress) > Make hoodie record_key and preCombine_key optional > -

[jira] [Updated] (HUDI-3636) Clustering fails due to marker creation failure

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3636: - Priority: Critical (was: Blocker) > Clustering fails due to marker creation failure > ---

[jira] [Assigned] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5610: --- Assignee: Rahil Chertara > Hudi cli bundle Spark 3.2.0 startup issue. >

[jira] [Updated] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5610: Sprint: 0.13.0 Final Sprint 3 > Hudi cli bundle Spark 3.2.0 startup issue. > --

[jira] [Updated] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5610: Story Points: 2 > Hudi cli bundle Spark 3.2.0 startup issue. > ---

[jira] [Updated] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5610: Reviewers: Ethan Guo > Hudi cli bundle Spark 3.2.0 startup issue. > ---

[jira] [Updated] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5610: Status: In Progress (was: Open) > Hudi cli bundle Spark 3.2.0 startup issue. > ---

[jira] [Updated] (HUDI-5610) Hudi cli bundle Spark 3.2.0 startup issue.

2023-01-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5610: Status: Patch Available (was: In Progress) > Hudi cli bundle Spark 3.2.0 startup issue. >

[jira] [Updated] (HUDI-5352) Jackson fails to serialize LocalDate when updating Delta Commit metadata

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5352: - Priority: Critical (was: Blocker) > Jackson fails to serialize LocalDate when updating Delta Commit metad

[jira] [Assigned] (HUDI-2681) Make hoodie record_key and preCombine_key optional

2023-01-24 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-2681: Assignee: sivabalan narayanan (was: Lokesh Jain) > Make hoodie record_key and preCombine_key optio

[GitHub] [hudi] alexeykudinkin commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock

2023-01-24 Thread via GitHub
alexeykudinkin commented on PR #7307: URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403015540 @TengHuo can you please rebase to the latest master and verify whether fix is still relevant? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] hudi-bot commented on pull request #7576: [HUDI-4991] Allow kafka-like configs to set truststore and keystore for the SchemaProvider

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7576: URL: https://github.com/apache/hudi/pull/7576#issuecomment-1403020957 ## CI report: * 0a0d47dacbb1815dc0f22e91ab39125a1fcc1993 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403021176 ## CI report: * 8c3cc74c973ae760ebd1803dfdc638e9821f181a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1403024849 ## CI report: * 4b13e187f9bf8ab53644ec93cc8e12eec122ce9c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7668: URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403025622 ## CI report: * b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN * be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403025684 ## CI report: * 8c3cc74c973ae760ebd1803dfdc638e9821f181a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7668: URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403029740 ## CI report: * b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN * be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403029815 ## CI report: * 8c3cc74c973ae760ebd1803dfdc638e9821f181a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7722: URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403034764 ## CI report: * 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN * 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] danny0405 commented on a diff in pull request #7738: [HUDI-5569] Fixing TableFileSystemView to detect early failed commits

2023-01-24 Thread via GitHub
danny0405 commented on code in PR #7738: URL: https://github.com/apache/hudi/pull/7738#discussion_r1086166592 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java: ## @@ -125,8 +133,21 @@ private boolean isFileSliceCommitted(FileSlice slice) { if (

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
nsivabalan commented on code in PR #7690: URL: https://github.com/apache/hudi/pull/7690#discussion_r1086169007 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -289,6 +292,113 @@ private void clear() { */ protected abs

[jira] [Closed] (HUDI-5182) Create Hudi CLI Bundle

2023-01-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-5182. Resolution: Fixed > Create Hudi CLI Bundle > -- > > Key: HUDI-5182 >

[jira] [Closed] (HUDI-4666) Investigate Hudi CLI out of box support

2023-01-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-4666. Resolution: Fixed > Investigate Hudi CLI out of box support > > >

[jira] [Updated] (HUDI-5271) Inconsistent reader and writer schema in HoodieAvroDataBlock cause exception

2023-01-24 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5271: -- Priority: Critical (was: Major) > Inconsistent reader and writer schema in HoodieAvroDataBlock

[jira] [Updated] (HUDI-5271) Inconsistent reader and writer schema in HoodieAvroDataBlock cause exception

2023-01-24 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5271: -- Fix Version/s: 0.13.1 > Inconsistent reader and writer schema in HoodieAvroDataBlock cause excep

[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock

2023-01-24 Thread via GitHub
TengHuo commented on PR #7307: URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403059007 > @TengHuo can you please rebase to the latest master and verify whether fix is still relevant? @alexeykudinkin got it, np. Let me rebase it to the latest master branch. -- This is

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
alexeykudinkin commented on code in PR #6830: URL: https://github.com/apache/hudi/pull/6830#discussion_r1086178778 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java: ## @@ -282,45 +285,50 @@ private HoodieLogBlock createCorruptBlock(long blo

[GitHub] [hudi] yihua commented on a diff in pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
yihua commented on code in PR #7690: URL: https://github.com/apache/hudi/pull/7690#discussion_r1086184617 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -289,6 +292,113 @@ private void clear() { */ protected abstract

[GitHub] [hudi] nsivabalan commented on pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
nsivabalan commented on PR #7709: URL: https://github.com/apache/hudi/pull/7709#issuecomment-1403071049 CI is green https://user-images.githubusercontent.com/513218/214478113-0a6e3d32-5ea5-4ded-8496-85724a902a27.png";> -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] hudi-bot commented on pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7709: URL: https://github.com/apache/hudi/pull/7709#issuecomment-1403074931 ## CI report: * 9367d417e91a5f43140e1a7bde07e1c0372a43dc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock

2023-01-24 Thread via GitHub
TengHuo commented on PR #7307: URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403077872 Hi @alexeykudinkin Before rebase, I have one thing want to check with you. Last week, there was an issue about a similar exception about Avro schema namespace, https://github.c

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403079877 ## CI report: * 8c3cc74c973ae760ebd1803dfdc638e9821f181a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7722: URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403079994 ## CI report: * 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN * 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] yihua commented on issue #7541: [SUPPORT] It's very slow to savepoint a table which has many (75k) partitions

2023-01-24 Thread via GitHub
yihua commented on issue #7541: URL: https://github.com/apache/hudi/issues/7541#issuecomment-1403080868 Hi @haoxie-aws the fix to this issue (#7690) is ready to land. I benchmarked the savepoint performance on a Hudi table with 20k partitions on a small EMR cluster and got the following im

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1403084872 ## CI report: * 4b13e187f9bf8ab53644ec93cc8e12eec122ce9c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403085537 ## CI report: * a708a002f3bb5dc237e430cff5ce4f43e8a97b51 UNKNOWN * 3b3000d68db2b47b46bb3f39d127b4655cfe53b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7722: URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403085610 ## CI report: * 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN * 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1403088098 ## CI report: * 4b13e187f9bf8ab53644ec93cc8e12eec122ce9c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] vinothchandar commented on pull request #7726: [HUDI-5575] Adding/Fixing auto generation of record keys w/ hudi

2023-01-24 Thread via GitHub
vinothchandar commented on PR #7726: URL: https://github.com/apache/hudi/pull/7726#issuecomment-1403090108 @nsivabalan are @alexeykudinkin 's review feedback addressed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
alexeykudinkin commented on code in PR #6830: URL: https://github.com/apache/hudi/pull/6830#discussion_r1086202983 ## hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java: ## @@ -914,6 +917,34 @@ public void testAppendAndReadOnCorruptedLog() throw

[GitHub] [hudi] codope commented on a diff in pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
codope commented on code in PR #7709: URL: https://github.com/apache/hudi/pull/7709#discussion_r1086209554 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java: ## @@ -497,8 +497,8 @@ public void testMetadataTableArchival

[GitHub] [hudi] nsivabalan merged pull request #7709: [HUDI-5582] Do not let users override internal metadata configs

2023-01-24 Thread via GitHub
nsivabalan merged PR #7709: URL: https://github.com/apache/hudi/pull/7709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

[hudi] branch master updated: [HUDI-5582] Do not let users override internal metadata configs (#7709)

2023-01-24 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new a79f8093755 [HUDI-5582] Do not let users overri

[jira] [Closed] (HUDI-5582) Do not let users override internal metadata configs

2023-01-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-5582. - Resolution: Fixed > Do not let users override internal metadata configs >

[jira] [Created] (HUDI-5614) Use custom (mocked) FS w/ custom scheme in TestHoodieLogFormat

2023-01-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-5614: - Summary: Use custom (mocked) FS w/ custom scheme in TestHoodieLogFormat Key: HUDI-5614 URL: https://issues.apache.org/jira/browse/HUDI-5614 Project: Apache Hudi I

[GitHub] [hudi] codope commented on a diff in pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
codope commented on code in PR #6830: URL: https://github.com/apache/hudi/pull/6830#discussion_r1086212761 ## hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java: ## @@ -914,6 +917,34 @@ public void testAppendAndReadOnCorruptedLog() throws IOExc

[GitHub] [hudi] hudi-bot commented on pull request #7744: [HUDI-5276] Fix getting partition paths under relative paths

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7744: URL: https://github.com/apache/hudi/pull/7744#issuecomment-1403123633 ## CI report: * 8a65ac3fee10325d26a3f23623dbd07bf9e25c51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1461

[GitHub] [hudi] codope commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled

2023-01-24 Thread via GitHub
codope commented on code in PR #7668: URL: https://github.com/apache/hudi/pull/7668#discussion_r1086228289 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala: ## @@ -18,7 +18,7 @@ package org.apache.spark.sql.

[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
xushiyan commented on code in PR #7722: URL: https://github.com/apache/hudi/pull/7722#discussion_r1086228641 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java: ## @@ -44,7 +44,7 @@ public class HoodieInstantTimeGenerator { publ

[GitHub] [hudi] subinsonynaukri commented on issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30

2023-01-24 Thread via GitHub
subinsonynaukri commented on issue #7732: URL: https://github.com/apache/hudi/issues/7732#issuecomment-1403157805 Thanks @alexeykudinkin for the reply. Even I tried it with DecimalType(18,0) and still getting this exception. Somehow if the number of digits is more than 15, this exception oc

[GitHub] [hudi] fengjian428 commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

2023-01-24 Thread via GitHub
fengjian428 commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1403158748 > the source code fix may not be required. #7739 we found an issue w/ locking. OK, I test master branch with this UT, it succeed -- This is an automated message from the Apache

[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
xushiyan commented on code in PR #7722: URL: https://github.com/apache/hudi/pull/7722#discussion_r1086260267 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java: ## @@ -752,17 +749,23 @@ public Builder setLayoutVersion(Option layoutVersion) {

[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation

2023-01-24 Thread via GitHub
xushiyan commented on code in PR #7722: URL: https://github.com/apache/hudi/pull/7722#discussion_r1086266100 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java: ## @@ -680,15 +680,12 @@ public void initializeBootstrapDirsIfNotExists() throws IO

[GitHub] [hudi] hudi-bot commented on pull request #6830: [HUDI-2118] Skip checking corrupt log blocks for transactional write file systems

2023-01-24 Thread via GitHub
hudi-bot commented on PR #6830: URL: https://github.com/apache/hudi/pull/6830#issuecomment-1403185385 ## CI report: * e4092aa1a724cf820113cef5223cfad011c53e26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1462

[GitHub] [hudi] hudi-bot commented on pull request #7690: [HUDI-5485] Add File System View API for batch listing and improve savepoint performance with metadata table

2023-01-24 Thread via GitHub
hudi-bot commented on PR #7690: URL: https://github.com/apache/hudi/pull/7690#issuecomment-1403186587 ## CI report: * a708a002f3bb5dc237e430cff5ce4f43e8a97b51 UNKNOWN * 98ff092b7f0142a017132604fd8b3c81fb0fb4d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

  1   2   3   >