[GitHub] [hudi] nsivabalan commented on issue #6373: [SUPPORT] Incremental cleaning never used during insert

2022-09-04 Thread GitBox
nsivabalan commented on issue #6373: URL: https://github.com/apache/hudi/issues/6373#issuecomment-1236276077 yes, sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] nsivabalan commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
nsivabalan commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236276782 @dongkelun : sorry, I dropped the ball on this. can you rebase w/ latest master. I can review it and we can get it in for 0.12.1. -- This is an automated message from the Apache Gi

[GitHub] [hudi] nsivabalan commented on a diff in pull request #5091: [HUDI-3453] Fix HoodieBackedTableMetadata concurrent reading issue

2022-09-04 Thread GitBox
nsivabalan commented on code in PR #5091: URL: https://github.com/apache/hudi/pull/5091#discussion_r962265102 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java: ## @@ -92,6 +100,52 @@ public void testTableOperatio

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236295076 ## CI report: * cece46a6cd515e141779415a499e8942f3589375 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8660

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236295886 ## CI report: * cece46a6cd515e141779415a499e8942f3589375 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8660

[GitHub] [hudi] hechao-ustc opened a new pull request, #6582: [DOCS] Add Flink DataStream API demo in Flink Guide.

2022-09-04 Thread GitBox
hechao-ustc opened a new pull request, #6582: URL: https://github.com/apache/hudi/pull/6582 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performa

[GitHub] [hudi] imtheguna opened a new issue, #6583: [SUPPORT] ERROR Executor: Exception in task 0.0 in stage 173.0 (TID 192)

2022-09-04 Thread GitBox
imtheguna opened a new issue, #6583: URL: https://github.com/apache/hudi/issues/6583 Hi, I trying to insert first time but getting below error. my code dff = my data frame val tableName = "hudi_trips_cow" val basePath = "file:///tmp/hudi_trips_cow" dff.

[GitHub] [hudi] Zouxxyy commented on issue #6578: [SUPPORT]: Potential bug with HiveSyncTool

2022-09-04 Thread GitBox
Zouxxyy commented on issue #6578: URL: https://github.com/apache/hudi/issues/6578#issuecomment-1236308318 I think you are right, and I found a new problem, I found that the active timeline is used [here](https://github.com/apache/hudi/blob/c0e15879668537d6de96db2994fc68209ad0baa5/hudi-sync/

[GitHub] [hudi] QuChunhe commented on issue #6560: [SUPPORT]Hudi java client throws Error upserting bucketType UPDATE for partition :0

2022-09-04 Thread GitBox
QuChunhe commented on issue #6560: URL: https://github.com/apache/hudi/issues/6560#issuecomment-1236309577 The above errors appear again. So schema evolution is not the cause of the errors. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236318064 ## CI report: * 239bcf292d2838a3a221bb388150c5a4b12f473d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1113

[GitHub] [hudi] QuChunhe opened a new issue, #6584: [SUPPORT]Hudi Java client don't support Multi Writing

2022-09-04 Thread GitBox
QuChunhe opened a new issue, #6584: URL: https://github.com/apache/hudi/issues/6584 Hudi Java client don't support Multi Writing, and throw errors: "Cannot resolve conflicts for overlapping writes" 1. Hudi version: 0.12.0, aliyun oss file system, flink 1.13.6, and hudi sink parallel

[GitHub] [hudi] QuChunhe opened a new issue, #6585: [SUPPORT]Hudi Java client writes data very very slowly

2022-09-04 Thread GitBox
QuChunhe opened a new issue, #6585: URL: https://github.com/apache/hudi/issues/6585 Every row data is about 0.72K. Every insert commit with 300 rows takes about 2 minutes. But increasing the rows of a commit, the time of every commit did not increase much. Event if each insert commit has 4

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236337450 ## CI report: * 239bcf292d2838a3a221bb388150c5a4b12f473d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1113

[GitHub] [hudi] QuChunhe commented on issue #6585: [SUPPORT]Hudi Java client writes data very very slowly

2022-09-04 Thread GitBox
QuChunhe commented on issue #6585: URL: https://github.com/apache/hudi/issues/6585#issuecomment-1236340053 Furthermore, the performance does not improve when increasing the parallelism, for example setting parallelism to 400 or 1000. ```java .withDeleteParallelism(parallelism)

[GitHub] [hudi] easonwood commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11

2022-09-04 Thread GitBox
easonwood commented on issue #6543: URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236347008 @Zouxxyy I try your method in https://github.com/apache/hudi/issues/6563. The classloader result looks correct [arthas@14524]$ classloader -t +-BootstrapClassLoader

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236348226 ## CI report: * 239bcf292d2838a3a221bb388150c5a4b12f473d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1113

[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11

2022-09-04 Thread GitBox
Zouxxyy commented on issue #6543: URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236348594 > customer.jar where do you add your customer jar? You can try putting it where your hudi-spark-bundle.jar is -- This is an automated message from the Apache Git Service. To re

[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11

2022-09-04 Thread GitBox
Zouxxyy commented on issue #6543: URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236348644 > @Zouxxyy I try your method in #6563. The classloader result looks correct, AppClassLoader load the special class > > [arthas@14524]$ classloader -t +-BootstrapClassLoader +-sun

[GitHub] [hudi] hudi-bot commented on pull request #5478: [HUDI-3998] Fix getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread GitBox
hudi-bot commented on PR #5478: URL: https://github.com/apache/hudi/pull/5478#issuecomment-1236361386 ## CI report: * 9b10ad3fb80db31e34e46abbd5d0b3ba9f179a8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114

[GitHub] [hudi] easonwood commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11

2022-09-04 Thread GitBox
easonwood commented on issue #6543: URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236362534 @Zouxxyy ReflectionUtils can be found in AppClassLoader. spark-submit \ --master yarn \ --deploy-mode cluster \ --jars /usr/lib/spark/jars/hudi-spark-bundle.j

[GitHub] [hudi] xushiyan opened a new pull request, #6586: [HUDI-4694] Print testcase running time for CI jobs

2022-09-04 Thread GitBox
xushiyan opened a new pull request, #6586: URL: https://github.com/apache/hudi/pull/6586 ### Change Logs Add a CI task for each UT/FT task to print top 100 long-running testcases. ### Impact **Risk level: none** ### Contributor's checklist - [ ] Read through

[jira] [Updated] (HUDI-4694) Analyze the latest UT/FT runtime

2022-09-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4694: - Labels: pull-request-available (was: ) > Analyze the latest UT/FT runtime > -

[jira] [Updated] (HUDI-4694) Analyze the latest UT/FT runtime

2022-09-04 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4694: - Status: Patch Available (was: In Progress) > Analyze the latest UT/FT runtime > -

[jira] [Updated] (HUDI-4694) Analyze the latest UT/FT runtime

2022-09-04 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4694: - Status: In Progress (was: Open) > Analyze the latest UT/FT runtime > > >

[GitHub] [hudi] hudi-bot commented on pull request #6586: [HUDI-4694] Print testcase running time for CI jobs

2022-09-04 Thread GitBox
hudi-bot commented on PR #6586: URL: https://github.com/apache/hudi/pull/6586#issuecomment-1236364498 ## CI report: * 4a77d16c6250dedf71405735952d24671506a8ad UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6586: [HUDI-4694] Print testcase running time for CI jobs

2022-09-04 Thread GitBox
hudi-bot commented on PR #6586: URL: https://github.com/apache/hudi/pull/6586#issuecomment-1236373425 ## CI report: * 4a77d16c6250dedf71405735952d24671506a8ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114

[GitHub] [hudi] nsivabalan commented on issue #6055: Hudi Partial Update not working by using MERGE statement on Hudi External Table

2022-09-04 Thread GitBox
nsivabalan commented on issue #6055: URL: https://github.com/apache/hudi/issues/6055#issuecomment-1236382839 @rishabhbandi : gentle ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [hudi] nsivabalan commented on issue #6503: [SUPPORT] Hudi Merge Into with larger volume

2022-09-04 Thread GitBox
nsivabalan commented on issue #6503: URL: https://github.com/apache/hudi/issues/6503#issuecomment-1236383069 @maduraitech : can you respond to Ethan's request above. would help us investigate further. -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [hudi] nsivabalan closed issue #5540: [SUPPORT]HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:

2022-09-04 Thread GitBox
nsivabalan closed issue #5540: [SUPPORT]HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:492) URL: https://github.com/apache/hudi/issues/5540 -- This is an automated message from the Apache Git Ser

[GitHub] [hudi] nsivabalan commented on issue #5540: [SUPPORT]HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync

2022-09-04 Thread GitBox
nsivabalan commented on issue #5540: URL: https://github.com/apache/hudi/issues/5540#issuecomment-1236383185 oh ok ok, got it. thanks. may I know which issue you are referring to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nleena123 opened a new issue, #5540: [SUPPORT]HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync

2022-09-04 Thread GitBox
nleena123 opened a new issue, #5540: URL: https://github.com/apache/hudi/issues/5540 Hi Team, our hudi jobs are failing with below exception. Can you please help me on this. HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastre

[GitHub] [hudi] nsivabalan commented on issue #5540: [SUPPORT]HoodieException: Commit 20220509105215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync

2022-09-04 Thread GitBox
nsivabalan commented on issue #5540: URL: https://github.com/apache/hudi/issues/5540#issuecomment-1236383285 @nleena123 : are you still facing issues or are we good to close it out ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [hudi] hudi-bot commented on pull request #6586: [HUDI-4694] Print testcase running time for CI jobs

2022-09-04 Thread GitBox
hudi-bot commented on PR #6586: URL: https://github.com/apache/hudi/pull/6586#issuecomment-1236385285 ## CI report: * 4a77d16c6250dedf71405735952d24671506a8ad Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114

[jira] [Created] (HUDI-4774) Support multi-writer/OCC for java write client

2022-09-04 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-4774: - Summary: Support multi-writer/OCC for java write client Key: HUDI-4774 URL: https://issues.apache.org/jira/browse/HUDI-4774 Project: Apache Hudi Is

[jira] [Updated] (HUDI-4774) Support multi-writer/OCC for java write client

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4774: -- Component/s: multi-writer > Support multi-writer/OCC for java write client > ---

[jira] [Updated] (HUDI-4774) Support multi-writer/OCC for java write client

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4774: -- Epic Link: HUDI-1456 > Support multi-writer/OCC for java write client >

[GitHub] [hudi] nsivabalan commented on issue #6584: [SUPPORT]Hudi Java client don't support Multi Writing

2022-09-04 Thread GitBox
nsivabalan commented on issue #6584: URL: https://github.com/apache/hudi/issues/6584#issuecomment-1236387535 yes, we don't have multi-writer support yet w/ java client. https://issues.apache.org/jira/browse/HUDI-4774 Let us know if you will have bandwidth to take a stab at it.

[jira] [Assigned] (HUDI-4774) Support multi-writer/OCC for java write client

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4774: - Assignee: sivabalan narayanan > Support multi-writer/OCC for java write client >

[jira] [Created] (HUDI-4775) Incremental source for MOR fails

2022-09-04 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-4775: - Summary: Incremental source for MOR fails Key: HUDI-4775 URL: https://issues.apache.org/jira/browse/HUDI-4775 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] nsivabalan opened a new pull request, #6587: [HUDI-4775] Fixing incremental source for MOR table

2022-09-04 Thread GitBox
nsivabalan opened a new pull request, #6587: URL: https://github.com/apache/hudi/pull/6587 ### Change Logs Incremental Source for a hudi table of type MOR fails since the commit timeline fetched works only for COW table. Fixing the timeline call in this patch. ### Impact

[jira] [Updated] (HUDI-4775) Incremental source for MOR fails

2022-09-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4775: - Labels: pull-request-available (was: ) > Incremental source for MOR fails >

[GitHub] [hudi] nsivabalan commented on a diff in pull request #6587: [HUDI-4775] Fixing incremental source for MOR table

2022-09-04 Thread GitBox
nsivabalan commented on code in PR #6587: URL: https://github.com/apache/hudi/pull/6587#discussion_r962360399 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java: ## @@ -73,7 +73,7 @@ public static Pair> calculateBeginAndEndInstants(J

[jira] [Assigned] (HUDI-4775) Incremental source for MOR fails

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4775: - Assignee: sivabalan narayanan > Incremental source for MOR fails > -

[jira] [Updated] (HUDI-4775) Incremental source for MOR fails

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4775: -- Fix Version/s: 0.12.1 > Incremental source for MOR fails >

[GitHub] [hudi] hudi-bot commented on pull request #6587: [HUDI-4775] Fixing incremental source for MOR table

2022-09-04 Thread GitBox
hudi-bot commented on PR #6587: URL: https://github.com/apache/hudi/pull/6587#issuecomment-1236406344 ## CI report: * 9c996aa5881d2a9e341b5181ef635750a7f4c926 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6587: [HUDI-4775] Fixing incremental source for MOR table

2022-09-04 Thread GitBox
hudi-bot commented on PR #6587: URL: https://github.com/apache/hudi/pull/6587#issuecomment-1236407142 ## CI report: * 9c996aa5881d2a9e341b5181ef635750a7f4c926 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114

[GitHub] [hudi] nsivabalan commented on issue #5253: Hudi execution plan not generated properly [SUPPORT]

2022-09-04 Thread GitBox
nsivabalan commented on issue #5253: URL: https://github.com/apache/hudi/issues/5253#issuecomment-1236416014 @njalan : can you let us know if we are good to close this one out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[jira] [Assigned] (HUDI-3646) The Hudi update syntax should not modify the nullability attribute of a column

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3646: - Assignee: Alexey Kudinkin > The Hudi update syntax should not modify the nullabil

[jira] [Updated] (HUDI-4631) Enhance retries for failed writes w/ write conflicts in a multi writer scenarios

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4631: -- Sprint: 2022/09/19 (was: 2022/09/05) > Enhance retries for failed writes w/ write confl

[jira] [Assigned] (HUDI-3055) Make sure that Compression Codec configuration is respected across the board

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3055: - Assignee: Alexey Kudinkin > Make sure that Compression Codec configuration is res

[jira] [Updated] (HUDI-4123) HoodieDeltaStreamer throws exception due to SqlSource return null checkpoint

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4123: -- Reviewers: sivabalan narayanan > HoodieDeltaStreamer throws exception due to SqlSource r

[jira] [Assigned] (HUDI-3775) Allow for offline compaction of MOR tables via spark streaming

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3775: - Assignee: sivabalan narayanan (was: Rajesh) > Allow for offline compaction of MO

[jira] [Assigned] (HUDI-3780) improve drop partitions

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3780: - Assignee: sivabalan narayanan > improve drop partitions > ---

[jira] [Assigned] (HUDI-3796) Implement layout to filter out uncommitted log files without reading the log blocks

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3796: - Assignee: sivabalan narayanan > Implement layout to filter out uncommitted log fi

[jira] [Updated] (HUDI-3881) Implement index syntax for spark sql

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3881: -- Sprint: 2022/09/19 (was: 2022/09/05) > Implement index syntax for spark sql > -

[jira] [Assigned] (HUDI-3915) Error upserting bucketType UPDATE for partition :0

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3915: - Assignee: Alexey Kudinkin > Error upserting bucketType UPDATE for partition :0 >

[jira] [Updated] (HUDI-3994) HoodieDeltaStreamer - Spark master shouldn't have a default

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3994: -- Reviewers: Raymond Xu > HoodieDeltaStreamer - Spark master shouldn't have a default > --

[jira] [Assigned] (HUDI-3959) Rename class name for spark rdd reader

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3959: - Assignee: Raymond Xu (was: Simon Su) > Rename class name for spark rdd reader >

[jira] [Assigned] (HUDI-3954) Don't keep the last commit before the earliest commit to retain

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3954: - Assignee: sivabalan narayanan (was: 董可伦) > Don't keep the last commit before the

[jira] [Assigned] (HUDI-4199) Clean up row writer path for url encoding, consistent logical timestamp

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4199: - Assignee: sivabalan narayanan > Clean up row writer path for url encoding, consis

[jira] [Assigned] (HUDI-4201) Add tooling to delete empty non-completed instants from timeline

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4201: - Assignee: Sagar Sumit > Add tooling to delete empty non-completed instants from t

[jira] [Updated] (HUDI-4282) Throws IOException in method HoodieLogFileReader.isBlockCorrupted()

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4282: -- Reviewers: sivabalan narayanan > Throws IOException in method HoodieLogFileReader.isBloc

[jira] [Updated] (HUDI-4281) Using hudi to build a large number of tables in spark on hive causes OOM

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4281: -- Reviewers: Ethan Guo > Using hudi to build a large number of tables in spark on hive cau

[jira] [Assigned] (HUDI-4493) Fix handling of corrupt avro files properly

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4493: - Assignee: sivabalan narayanan > Fix handling of corrupt avro files properly > ---

[jira] [Updated] (HUDI-4526) improve spillableMapBasePath disk directory is full

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4526: -- Reviewers: sivabalan narayanan > improve spillableMapBasePath disk directory is full > -

[jira] [Assigned] (HUDI-4526) improve spillableMapBasePath disk directory is full

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4526: - Assignee: sivabalan narayanan (was: Forward Xu) > improve spillableMapBasePath d

[jira] [Assigned] (HUDI-3648) Failed to execute rollback due to HoodieIOException: Could not delete instant

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3648: - Assignee: Ethan Guo > Failed to execute rollback due to HoodieIOException: Could

[jira] [Updated] (HUDI-3648) Failed to execute rollback due to HoodieIOException: Could not delete instant

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3648: -- Story Points: 2 > Failed to execute rollback due to HoodieIOException: Could not delete

[jira] [Updated] (HUDI-4613) Avoid the use of regex expressions when call hoodieFileGroup#addLogFile function

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4613: -- Story Points: 0.5 > Avoid the use of regex expressions when call hoodieFileGroup#addLogF

[jira] [Updated] (HUDI-4613) Avoid the use of regex expressions when call hoodieFileGroup#addLogFile function

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4613: -- Reviewers: Sagar Sumit > Avoid the use of regex expressions when call hoodieFileGroup#ad

[jira] [Updated] (HUDI-3818) hudi doesn't support bytes column as primary key

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3818: -- Sprint: 2022/09/19 (was: 2022/09/05) > hudi doesn't support bytes column as primary key

[jira] [Updated] (HUDI-3617) MOR compact improve

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3617: -- Reviewers: sivabalan narayanan > MOR compact improve > --- > >

[jira] [Updated] (HUDI-3617) MOR compact improve

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3617: -- Story Points: 0.5 (was: 1.5) > MOR compact improve > --- > >

[jira] [Assigned] (HUDI-3342) MOR Delta Block Rollbacks not applied if Lazy Block reading is disabled

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3342: - Assignee: Raymond Xu > MOR Delta Block Rollbacks not applied if Lazy Block readin

[jira] [Updated] (HUDI-3998) getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3998: -- Story Points: 1 (was: 1.5) > getCommitsSinceLastCleaning failed when async cleaning > -

[jira] [Assigned] (HUDI-4330) NPE when trying to upsert into a dataset with no Meta Fields

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4330: - Assignee: Raymond Xu > NPE when trying to upsert into a dataset with no Meta Fiel

[jira] [Updated] (HUDI-3998) getCommitsSinceLastCleaning failed when async cleaning

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3998: -- Reviewers: sivabalan narayanan > getCommitsSinceLastCleaning failed when async cleaning

[jira] [Updated] (HUDI-3407) Make sure Restore operation is Not Concurrent w/ Writes in Multi-Writer scenario

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3407: -- Sprint: 2022/09/19 (was: 2022/09/05) > Make sure Restore operation is Not Concurrent w/

[jira] [Updated] (HUDI-4666) Investigate Hudi CLI out of box support

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4666: -- Reviewers: Sagar Sumit > Investigate Hudi CLI out of box support >

[jira] [Assigned] (HUDI-4624) Make sure all DeltaStreamer Sources are Closeable

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4624: - Assignee: sivabalan narayanan > Make sure all DeltaStreamer Sources are Closeable

[jira] [Updated] (HUDI-4363) Support Clustering row writer to improve performance

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4363: -- Reviewers: sivabalan narayanan > Support Clustering row writer to improve performance >

[jira] [Updated] (HUDI-4363) Support Clustering row writer to improve performance

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4363: -- Story Points: 2 > Support Clustering row writer to improve performance > ---

[jira] [Updated] (HUDI-4755) INSERT_OVERWRITE(/TABLE) in spark sql should not fail time travel queries for older timestamps

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4755: -- Reviewers: Raymond Xu > INSERT_OVERWRITE(/TABLE) in spark sql should not fail time trave

[jira] [Assigned] (HUDI-4761) Test using spark listeners that guards any changes to DAG

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4761: - Assignee: sivabalan narayanan > Test using spark listeners that guards any change

[jira] [Updated] (HUDI-4761) Test using spark listeners that guards any changes to DAG

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4761: -- Story Points: 3 > Test using spark listeners that guards any changes to DAG > --

[jira] [Updated] (HUDI-4761) Test using spark listeners that guards any changes to DAG

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4761: -- Fix Version/s: 0.12.1 > Test using spark listeners that guards any changes to DAG >

[jira] [Updated] (HUDI-2369) Blog on bulk insert sort modes

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2369: -- Story Points: 3 > Blog on bulk insert sort modes > -- > >

[jira] [Updated] (HUDI-3067) "Table already exists" error with multiple writers and dynamodb

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3067: -- Story Points: 1 > "Table already exists" error with multiple writers and dynamodb >

[jira] [Updated] (HUDI-3067) "Table already exists" error with multiple writers and dynamodb

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3067: -- Reviewers: sivabalan narayanan > "Table already exists" error with multiple writers and

[jira] [Updated] (HUDI-4620) No expected exception is thrown when create hudi table without primaryKey

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4620: -- Story Points: 0.5 > No expected exception is thrown when create hudi table without prima

[jira] [Assigned] (HUDI-4136) Run snapshot query in hive throw ' IOException: java.lang.IllegalArgumentException: HoodieRealtimeRecordReader can only work on RealtimeSplit and not with a empty file'.

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4136: - Assignee: Sagar Sumit > Run snapshot query in hive throw ' IOException: > java.l

[jira] [Updated] (HUDI-3892) Add HoodieReadClient with java

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3892: -- Sprint: 2022/09/19 (was: 2022/09/05) > Add HoodieReadClient with java > ---

[jira] [Updated] (HUDI-4620) No expected exception is thrown when create hudi table without primaryKey

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4620: -- Reviewers: Ethan Guo > No expected exception is thrown when create hudi table without pr

[jira] [Updated] (HUDI-3646) The Hudi update syntax should not modify the nullability attribute of a column

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3646: -- Story Points: 1.5 > The Hudi update syntax should not modify the nullability attribute o

[jira] [Updated] (HUDI-2786) Failed to connect to namenode in Docker Demo on Apple M1 chip

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2786: -- Story Points: 8 > Failed to connect to namenode in Docker Demo on Apple M1 chip > --

[jira] [Updated] (HUDI-2786) Failed to connect to namenode in Docker Demo on Apple M1 chip

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2786: -- Story Points: 6 (was: 8) > Failed to connect to namenode in Docker Demo on Apple M1 chi

[jira] [Updated] (HUDI-4281) Using hudi to build a large number of tables in spark on hive causes OOM

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4281: -- Sprint: 2022/09/19 (was: 2022/09/05) > Using hudi to build a large number of tables in

[jira] [Updated] (HUDI-3425) Clean up spill path created by Hudi during uneventful shutdown

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3425: -- Story Points: 2 > Clean up spill path created by Hudi during uneventful shutdown > -

[jira] [Updated] (HUDI-1885) Support Delete/Update Non-Pk Table

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1885: -- Story Points: 4 > Support Delete/Update Non-Pk Table > -

[jira] [Updated] (HUDI-3055) Make sure that Compression Codec configuration is respected across the board

2022-09-04 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3055: -- Story Points: 2 > Make sure that Compression Codec configuration is respected across the

  1   2   >