[jira] [Closed] (HUDI-3100) Hive Conditional sync cannot be set from deltastreamer
[ https://issues.apache.org/jira/browse/HUDI-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3100. Fix Version/s: (was: 0.11.0) Reviewers: sivabalan narayanan Resolution: Fixed > Hive Conditional sync cannot be set from deltastreamer > -- > > Key: HUDI-3100 > URL: https://issues.apache.org/jira/browse/HUDI-3100 > Project: Apache Hudi > Issue Type: Bug > Components: DeltaStreamer, Hive Integration >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available > Fix For: 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[hudi] branch master updated (b2b23f5 -> 2467c13)
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from b2b23f5 [HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter (#4521) add 2467c13 [HUDI-3100] Add config for hive conditional sync (#4440) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/hudi/DataSourceUtils.java | 2 ++ .../src/main/scala/org/apache/hudi/DataSourceOptions.scala | 10 ++ 2 files changed, 12 insertions(+)
[GitHub] [hudi] xushiyan merged pull request #4440: [HUDI-3100] Add config for hive conditional sync
xushiyan merged pull request #4440: URL: https://github.com/apache/hudi/pull/4440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?
xushiyan commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1007188747 After some discussions, we think that we should keep cloud provider's jars out of open source bundle jars. Any cloud provider can create its own specific hudi module and hudi bundle jars. (like `hudi-aws` and `hudi-spark-aws-bundle` for example) But open source bundle jars should stay neutral. cc @danny0405 @nsivabalan @codope @vinothchandar @zhedoubushishi @umehrot2 I've pivoted this ticket to removing bundle deps to align with flink bundle changes. https://issues.apache.org/jira/browse/HUDI-3157 > If we just directly remove the shading for aws jars, then we need to manually pass aws jars in the Spark/Flink classpath when the users are using AWS Dynamodb/cloudwatch features. @zhedoubushishi If to help users use the bundle a bit easier, as I suggested above, please consider adding an aws specific hudi bundle to resolve dependency problem. Hope this could align with your thoughts too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
dongkelun commented on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007187752 @xushiyan @nsivabalan Hello,can you please take a review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
hudi-bot removed a comment on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007166026 ## CI report: * 63cee6177c62cf267849d4f9379eaad88fd5f584 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4965) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
hudi-bot commented on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007186261 ## CI report: * 63cee6177c62cf267849d4f9379eaad88fd5f584 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4965) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar edited a comment on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
vinothchandar edited a comment on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1007172166 @leesf few questions. 1. What do the hudi-spark2-extensions/hudi-spark3-extensions do? What code would these have in the future? 2. Users may have spark jobs that depend directly on `hudi-spark` . any impact to these from the changes? 3. I feel there are way too many packages now - hudi-spark*, then it depends hudi-spark-common* . Is there a way for us to merge hudi-spark-common into hudi-spark* ? is the hudi-spark3-common so we can share code across the different 3.x versions? Can you add a README that explains what code should go where going forward 4. Any updates to release notes or README? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
vinothchandar commented on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1007172166 @leesf few questions. 1. What do the hudi-spark2-extensions/hudi-spark3-extensions do? What code would these have in the future? 2. Users may have spark jobs that depend directly on `hudi-spark` . any impact to these from the changes? 3. I feel there are way too many packages now - hudi-spark*, then it depends hudi-spark-common* . Is there a way for us to merge hudi-spark-common into hudi-spark* ? is the hudi-spark3-common so we can share code across the different 3.x versions? Can you add a README that explains what code should go where going forward -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4532: [Minor]Fix some code style based on check-sytle plugin
hudi-bot removed a comment on pull request #4532: URL: https://github.com/apache/hudi/pull/4532#issuecomment-1007152604 ## CI report: * 59e48916ff71ca86523e40558a2e15418a3fffcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4964) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3193) Move hudi-aws dependency out of hudi-client-common
Raymond Xu created HUDI-3193: Summary: Move hudi-aws dependency out of hudi-client-common Key: HUDI-3193 URL: https://issues.apache.org/jira/browse/HUDI-3193 Project: Apache Hudi Issue Type: Improvement Reporter: Raymond Xu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4532: [Minor]Fix some code style based on check-sytle plugin
hudi-bot commented on pull request #4532: URL: https://github.com/apache/hudi/pull/4532#issuecomment-1007171945 ## CI report: * 59e48916ff71ca86523e40558a2e15418a3fffcb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4964) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3157) Remove aws jars from hudi bundles
[ https://issues.apache.org/jira/browse/HUDI-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3157: - Summary: Remove aws jars from hudi bundles (was: Shade aws-dependencies to avoid class conflicts) > Remove aws jars from hudi bundles > - > > Key: HUDI-3157 > URL: https://issues.apache.org/jira/browse/HUDI-3157 > Project: Apache Hudi > Issue Type: Bug >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Critical > Labels: sev:critical, user-support-issues > Fix For: 0.11.0, 0.10.1 > > > ref: > [https://github.com/apache/hudi/issues/4474] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3157) Shade aws-dependencies to avoid class conflicts
[ https://issues.apache.org/jira/browse/HUDI-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3157: Assignee: Raymond Xu (was: Wenning Ding) > Shade aws-dependencies to avoid class conflicts > --- > > Key: HUDI-3157 > URL: https://issues.apache.org/jira/browse/HUDI-3157 > Project: Apache Hudi > Issue Type: Bug >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Critical > Labels: sev:critical, user-support-issues > Fix For: 0.11.0, 0.10.1 > > > ref: > [https://github.com/apache/hudi/issues/4474] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
hudi-bot commented on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007166026 ## CI report: * 63cee6177c62cf267849d4f9379eaad88fd5f584 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4965) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot commented on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007166010 ## CI report: * f67f3a7d1a3de2bd170fed67031a37ad1da5115a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4962) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
hudi-bot removed a comment on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007164625 ## CI report: * 63cee6177c62cf267849d4f9379eaad88fd5f584 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot removed a comment on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007136180 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) * f67f3a7d1a3de2bd170fed67031a37ad1da5115a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4962) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
leesf commented on a change in pull request #4514: URL: https://github.com/apache/hudi/pull/4514#discussion_r780047010 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -177,7 +177,7 @@ class DefaultSource extends RelationProvider outputMode) } - override def shortName(): String = "hudi" + override def shortName(): String = "hudi_v1" Review comment: it is because in hudi-spark-bundle module. I used ` META-INF/services/org.apache.spark.sql.sources.DataSourceRegister ` to append the formats(hudi_v1 and hudi) in `DataSourceRegister` file, so it will conflict if not change the format. As to the PR itself, we do not need to change the format to `hudi_v1` and not use AppendingTransformer. But when implementing V2 codepath, I find it difficult to handle the incremental bootstrap table(https://github.com/apache/hudi/blob/ac8d014a0602e3c499771f3313f0f88de57cdda1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala#L175) as it will pass schema to Spark and difficult to handle in v2 codepath, after finding a good way to handle it, we would definitely delete the hudi_v1 format here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
leesf commented on a change in pull request #4514: URL: https://github.com/apache/hudi/pull/4514#discussion_r780047010 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -177,7 +177,7 @@ class DefaultSource extends RelationProvider outputMode) } - override def shortName(): String = "hudi" + override def shortName(): String = "hudi_v1" Review comment: it is because in hudi-spark-bundle module. I used ` META-INF/services/org.apache.spark.sql.sources.DataSourceRegister ` to append the formats(hudi_v1 and hudi) in `DataSourceRegister` file, so it will conflict if not change the format. As to the PR itself, we do not need to change the format to `hudi_v1` and not use AppendingTransformer. But when implementing V2 codepath, I find it difficult to handle the incremental bootstrap table(https://github.com/apache/hudi/blob/ac8d014a0602e3c499771f3313f0f88de57cdda1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala#L174) as it will pass schema to Spark and difficult to handle in v2 codepath, after finding a good way to handle it, we would definitely delete the hudi_v1 format here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
hudi-bot commented on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007164625 ## CI report: * 63cee6177c62cf267849d4f9379eaad88fd5f584 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
dongkelun commented on pull request #4533: URL: https://github.com/apache/hudi/pull/4533#issuecomment-1007164107 This PR is to solve this [issue](https://github.com/apache/hudi/issues/4525) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3192) Spark metastore schema evolution broken
[ https://issues.apache.org/jira/browse/HUDI-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3192: - Labels: pull-request-available (was: ) > Spark metastore schema evolution broken > --- > > Key: HUDI-3192 > URL: https://issues.apache.org/jira/browse/HUDI-3192 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Blocker > Labels: pull-request-available > Fix For: 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] dongkelun opened a new pull request #4533: [HUDI-3192] Spark metastore schema evolution broken
dongkelun opened a new pull request #4533: URL: https://github.com/apache/hudi/pull/4533 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
vinothchandar commented on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1007160076 2 more days should be okay? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3192) Spark metastore schema evolution broken
董可伦 created HUDI-3192: - Summary: Spark metastore schema evolution broken Key: HUDI-3192 URL: https://issues.apache.org/jira/browse/HUDI-3192 Project: Apache Hudi Issue Type: Improvement Components: Spark Integration Reporter: 董可伦 Assignee: 董可伦 Fix For: 0.10.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] zhangyue19921010 commented on pull request #4078: [HUDI-2833] Clean up unused archive files instead of expanding indefinitely.
zhangyue19921010 commented on pull request #4078: URL: https://github.com/apache/hudi/pull/4078#issuecomment-1007158124 @nsivabalan and @yihua Thanks a lot for your help :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
vinothchandar commented on a change in pull request #4514: URL: https://github.com/apache/hudi/pull/4514#discussion_r780041223 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -177,7 +177,7 @@ class DefaultSource extends RelationProvider outputMode) } - override def shortName(): String = "hudi" + override def shortName(): String = "hudi_v1" Review comment: would it conflict? Given we are extending `DefaultSource` and overriding `shortName()`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope closed pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path
codope closed pull request #3877: URL: https://github.com/apache/hudi/pull/3877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot commented on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007154511 ## CI report: * 26a7987a7b002b5596ba627ee6d7dc15dfc6c922 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4961) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot removed a comment on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007136167 ## CI report: * 6917597395b1f70c4953833554d9b584d4645297 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4955) * 26a7987a7b002b5596ba627ee6d7dc15dfc6c922 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4961) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
xushiyan commented on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1007153782 > @xushiyan given we are almost winding down for 0.10.1, I suggest we land this sooner than later. That way we can focus on stabilizing master for 0.11.0 - that's not too far away. wdyt? it won't be too far away. @nsivabalan is cherry picking for 0.10.1 which will complete by Jan 9. Holding this off for 2 more days can avoid conflicts from some Spark fixes merged after this. Seeing 2 more fixes coming. After that we should be able to land this right away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4532: [Minor]Fix some code style based on check-sytle plugin
hudi-bot commented on pull request #4532: URL: https://github.com/apache/hudi/pull/4532#issuecomment-1007152604 ## CI report: * 59e48916ff71ca86523e40558a2e15418a3fffcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4964) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4532: [Minor]Fix some code style based on check-sytle plugin
hudi-bot removed a comment on pull request #4532: URL: https://github.com/apache/hudi/pull/4532#issuecomment-1007151644 ## CI report: * 59e48916ff71ca86523e40558a2e15418a3fffcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4532: [Minor]Fix some code style based on check-sytle plugin
hudi-bot commented on pull request #4532: URL: https://github.com/apache/hudi/pull/4532#issuecomment-1007151644 ## CI report: * 59e48916ff71ca86523e40558a2e15418a3fffcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4521: [HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
zhangyue19921010 commented on pull request #4521: URL: https://github.com/apache/hudi/pull/4521#issuecomment-1007151532 Hi @nsivabalan Thanks a lot for your help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 opened a new pull request #4532: [Minor]Fix some code style based on check-sytle plugin
zhangyue19921010 opened a new pull request #4532: URL: https://github.com/apache/hudi/pull/4532 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007148515 ## CI report: * fde98bb87aca9963a377d036f458d83f23a90ad6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4960) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007135175 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) * fde98bb87aca9963a377d036f458d83f23a90ad6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4960) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4531: [WIP][HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot removed a comment on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1007136190 ## CI report: * 16c7ef660e054e35bda2c981c3a5dd9d9c60e975 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4963) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4531: [WIP][HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot commented on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1007147660 ## CI report: * 16c7ef660e054e35bda2c981c3a5dd9d9c60e975 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4963) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4531: [WIP][HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot removed a comment on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1007135339 ## CI report: * 16c7ef660e054e35bda2c981c3a5dd9d9c60e975 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4531: [WIP][HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot commented on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1007136190 ## CI report: * 16c7ef660e054e35bda2c981c3a5dd9d9c60e975 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4963) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot removed a comment on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007135331 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) * f67f3a7d1a3de2bd170fed67031a37ad1da5115a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot removed a comment on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007135310 ## CI report: * 6917597395b1f70c4953833554d9b584d4645297 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4955) * 26a7987a7b002b5596ba627ee6d7dc15dfc6c922 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot commented on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007136180 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) * f67f3a7d1a3de2bd170fed67031a37ad1da5115a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4962) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot commented on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007136167 ## CI report: * 6917597395b1f70c4953833554d9b584d4645297 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4955) * 26a7987a7b002b5596ba627ee6d7dc15dfc6c922 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4961) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot commented on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007135331 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) * f67f3a7d1a3de2bd170fed67031a37ad1da5115a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4531: [WIP][HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot commented on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1007135339 ## CI report: * 16c7ef660e054e35bda2c981c3a5dd9d9c60e975 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot removed a comment on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007055013 ## CI report: * 6917597395b1f70c4953833554d9b584d4645297 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4955) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4520: [HUDI-3179][Stacked on 4417] Extracted common `AbstractHoodieTableFileIndex` to be shared across engines
hudi-bot commented on pull request #4520: URL: https://github.com/apache/hudi/pull/4520#issuecomment-1007135310 ## CI report: * 6917597395b1f70c4953833554d9b584d4645297 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4955) * 26a7987a7b002b5596ba627ee6d7dc15dfc6c922 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot removed a comment on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007099737 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007134338 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) * fde98bb87aca9963a377d036f458d83f23a90ad6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007135175 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) * fde98bb87aca9963a377d036f458d83f23a90ad6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4960) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007129547 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007134338 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) * fde98bb87aca9963a377d036f458d83f23a90ad6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3191) Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex
[ https://issues.apache.org/jira/browse/HUDI-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3191: - Labels: pull-request-available (was: ) > Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex > --- > > Key: HUDI-3191 > URL: https://issues.apache.org/jira/browse/HUDI-3191 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Alexey Kudinkin >Priority: Major > Labels: pull-request-available > > There are multiple control flows that would require accurate re-mapping to > start leveraging `AbstractHoodieTableFileIndex` > # Snapshot Query mode > # Incremental Query mode > This task would focus mostly on rebasing Snapshot Mode -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] alexeykudinkin opened a new pull request #4531: [WIP][HUDI-3191] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
alexeykudinkin opened a new pull request #4531: URL: https://github.com/apache/hudi/pull/4531 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex` ## Brief change log - Moving `AbstractHoodieTableFileIndex` to "hudi-spark-common" (temporarily, will be migrated to "hudi-common") - Bootstrapping `HiveHoodieTableFileIndex` impl of `AbstractHoodieTableFileIndex` for Hive - Rebasing `HiveFileInputFormatBase` onto `HiveHoodieTableFileIndex` ## Verify this pull request This pull request is already covered by existing tests, such as *(please describe tests)*. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3191) Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex
[ https://issues.apache.org/jira/browse/HUDI-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3191: -- Status: In Progress (was: Open) > Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex > --- > > Key: HUDI-3191 > URL: https://issues.apache.org/jira/browse/HUDI-3191 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Alexey Kudinkin >Priority: Major > > There are multiple control flows that would require accurate re-mapping to > start leveraging `AbstractHoodieTableFileIndex` > # Snapshot Query mode > # Incremental Query mode > This task would focus mostly on rebasing Snapshot Mode -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3191) Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex
Alexey Kudinkin created HUDI-3191: - Summary: Rebase Hive's FileInputFormat onto AbstractHoodieTableFileIndex Key: HUDI-3191 URL: https://issues.apache.org/jira/browse/HUDI-3191 Project: Apache Hudi Issue Type: Sub-task Reporter: Alexey Kudinkin There are multiple control flows that would require accurate re-mapping to start leveraging `AbstractHoodieTableFileIndex` # Snapshot Query mode # Incremental Query mode This task would focus mostly on rebasing Snapshot Mode -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3179) Extract common Hudi Table File Index implementation
[ https://issues.apache.org/jira/browse/HUDI-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-3179: - Assignee: Alexey Kudinkin > Extract common Hudi Table File Index implementation > > > Key: HUDI-3179 > URL: https://issues.apache.org/jira/browse/HUDI-3179 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Extract common Hudi Table File Index implementation from Spark's > `HoodieFileIndex`, to leverage common file indexing functionality across > Spark/Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3179) Extract common Hudi Table File Index implementation
[ https://issues.apache.org/jira/browse/HUDI-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3179: -- Status: Patch Available (was: In Progress) > Extract common Hudi Table File Index implementation > > > Key: HUDI-3179 > URL: https://issues.apache.org/jira/browse/HUDI-3179 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Extract common Hudi Table File Index implementation from Spark's > `HoodieFileIndex`, to leverage common file indexing functionality across > Spark/Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3179) Extract common Hudi Table File Index implementation
[ https://issues.apache.org/jira/browse/HUDI-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3179: -- Status: In Progress (was: Open) > Extract common Hudi Table File Index implementation > > > Key: HUDI-3179 > URL: https://issues.apache.org/jira/browse/HUDI-3179 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Extract common Hudi Table File Index implementation from Spark's > `HoodieFileIndex`, to leverage common file indexing functionality across > Spark/Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007107627 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007129547 ## CI report: * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3082) [Phase 1] Unify MOR table access across Spark, Hive
[
https://issues.apache.org/jira/browse/HUDI-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-3082:
-
Status: Patch Available (was: In Progress)
> [Phase 1] Unify MOR table access across Spark, Hive
> ---
>
> Key: HUDI-3082
> URL: https://issues.apache.org/jira/browse/HUDI-3082
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> This is Phase 1 of what outlined in HUDI-3081
>
> The goal is
> * Unify Hive’s RecordReaders (`RealtimeCompactedRecordReader`,
> {{{}RealtimeUnmergedRecordReader{}}})
> ** _These Readers should only differ in the way they handle the payload,
> everything else should remain constant_
> * Abstract w/in common component (name TBD)
> ** Listing current file-slices at the requested instant (handling the
> timeline)
> ** Creating Record Iterator for the provided file-slice
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hudi] vinothchandar commented on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
vinothchandar commented on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1007120240 @xushiyan given we are almost winding down for 0.10.1, I suggest we land this sooner than later. That way we can focus on stabilizing master for 0.11.0 - that's not too far away. wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007106330 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) * cb2d16ca92d7867d43c405f2babc9a85aaeca64e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007107627 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) * cb2d16ca92d7867d43c405f2babc9a85aaeca64e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007100483 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007106330 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) * cb2d16ca92d7867d43c405f2babc9a85aaeca64e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1896) [UMBRELLA] Implement DeltaStreamer Source for cloud object stores
[ https://issues.apache.org/jira/browse/HUDI-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1896: -- Epic Name: Implement DeltaStreamer Source for cloud object stores > [UMBRELLA] Implement DeltaStreamer Source for cloud object stores > - > > Key: HUDI-1896 > URL: https://issues.apache.org/jira/browse/HUDI-1896 > Project: Apache Hudi > Issue Type: Epic > Components: DeltaStreamer >Reporter: Raymond Xu >Assignee: Rajesh Mahindra >Priority: Critical > Labels: hudi-umbrellas, pull-request-available > Fix For: 1.0.0 > > > As discussed in HUDI-1723, we need a better implementation for Cloud object > storage like AWS S3 or GCS, leveraging on change notification. > Also consider > [https://docs.databricks.com/spark/latest/structured-streaming/sqs.html] > > We need to look into current *DFSSource classes and see if we can add a new > `DFSPathSelector` implementation, that fetech new files on cloud storage > after a given point in time. The timestamp based approach used by existing > path selector, largely works, but has corner cases as mentioned in HUDI-1723 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1628) [Umbrella] Improve data locality during ingestion
[ https://issues.apache.org/jira/browse/HUDI-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1628: -- Epic Name: Improve data locality during ingestion Description: Today the upsert partitioner does the file sizing/bin-packing etc for inserts and then sends some inserts over to existing file groups to maintain file size. We can abstract all of this into strategies and some kind of pipeline abstractions and have it also consider "affinity" to an existing file group based on say information stored in the metadata table? See http://mail-archives.apache.org/mod_mbox/hudi-dev/202102.mbox/browser for more details was: Today the upsert partitioner does the file sizing/bin-packing etc for inserts and then sends some inserts over to existing file groups to maintain file size. We can abstract all of this into strategies and some kind of pipeline abstractions and have it also consider "affinity" to an existing file group based on say information stored in the metadata table? See http://mail-archives.apache.org/mod_mbox/hudi-dev/202102.mbox/browser for more details > [Umbrella] Improve data locality during ingestion > - > > Key: HUDI-1628 > URL: https://issues.apache.org/jira/browse/HUDI-1628 > Project: Apache Hudi > Issue Type: Epic > Components: Writer Core >Reporter: satish >Assignee: Ethan Guo >Priority: Major > Labels: hudi-umbrellas > Fix For: 0.11.0 > > > Today the upsert partitioner does the file sizing/bin-packing etc for > inserts and then sends some inserts over to existing file groups to > maintain file size. > We can abstract all of this into strategies and some kind of pipeline > abstractions and have it also consider "affinity" to an existing file group > based > on say information stored in the metadata table? > See http://mail-archives.apache.org/mod_mbox/hudi-dev/202102.mbox/browser > for more details -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1387) [UMBRELLA] Support Apache Calcite for writing/querying Hudi datasets
[ https://issues.apache.org/jira/browse/HUDI-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1387: -- Epic Name: Support Apache Calcite for writing/querying Hudi datasets > [UMBRELLA] Support Apache Calcite for writing/querying Hudi datasets > > > Key: HUDI-1387 > URL: https://issues.apache.org/jira/browse/HUDI-1387 > Project: Apache Hudi > Issue Type: Epic > Components: Common Core, Writer Core >Reporter: Raymond Xu >Priority: Major > Labels: gsoc, gsoc2021, hudi-umbrellas, mentor > > (More details to be added) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1390) [UMBRELLA] Support schema inference for unstructured data
[ https://issues.apache.org/jira/browse/HUDI-1390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1390: -- Epic Name: Support schema inference for unstructured data > [UMBRELLA] Support schema inference for unstructured data > - > > Key: HUDI-1390 > URL: https://issues.apache.org/jira/browse/HUDI-1390 > Project: Apache Hudi > Issue Type: Epic > Components: bootstrap >Reporter: Raymond Xu >Priority: Major > Labels: gsoc, gsoc2021, hudi-umbrellas, mentor > > (More details to be added) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1385) [UMBRELLA] Improve source ingestion support in DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1385: -- Epic Name: Improve source ingestion support in DeltaStreamer > [UMBRELLA] Improve source ingestion support in DeltaStreamer > > > Key: HUDI-1385 > URL: https://issues.apache.org/jira/browse/HUDI-1385 > Project: Apache Hudi > Issue Type: Epic > Components: DeltaStreamer >Reporter: Raymond Xu >Assignee: Rajesh Mahindra >Priority: Major > Labels: gsoc, gsoc2021, hudi-umbrellas, mentor > > (More details to be added) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1250) [UMBRELLA] Test coverage
[ https://issues.apache.org/jira/browse/HUDI-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1250: -- Epic Name: Test coverage > [UMBRELLA] Test coverage > > > Key: HUDI-1250 > URL: https://issues.apache.org/jira/browse/HUDI-1250 > Project: Apache Hudi > Issue Type: Epic > Components: Testing >Reporter: sivabalan narayanan >Priority: Major > Labels: hudi-umbrellas > > I found a handful of tickets related to adding more tests. Creating this > umbrella ticket to track all of them together. > https://issues.apache.org/jira/browse/HUDI-987 : integration tests for MOR > table of decimal type > https://issues.apache.org/jira/browse/HUDI-778 : adding code cov badge > https://issues.apache.org/jira/browse/HUDI-699 : Add unit test for > CompactionCommand > https://issues.apache.org/jira/browse/HUDI-693: Add unit test for hudi-cli > module > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1249) [UMBRELLA] refactor tests for ease of development
[ https://issues.apache.org/jira/browse/HUDI-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1249: -- Epic Name: refactor tests for ease of development > [UMBRELLA] refactor tests for ease of development > - > > Key: HUDI-1249 > URL: https://issues.apache.org/jira/browse/HUDI-1249 > Project: Apache Hudi > Issue Type: Epic > Components: Testing >Reporter: sivabalan narayanan >Priority: Major > Labels: hudi-umbrellas > > Creating an umbrella ticket to track efforts to refactor test and test utils > for ease of development. > > https://issues.apache.org/jira/browse/HUDI-996: shared spark session > provider. > https://issues.apache.org/jira/browse/HUDI-995 Organize test utils methods > and classes > https://issues.apache.org/jira/browse/HUDI-994 : Identify functional tests > that are convertible to unit tests with mocks > https://issues.apache.org/jira/browse/HUDI-736 : Simplify > ReflectionUtils#getTopLevelClasses > https://issues.apache.org/jira/browse/HUDI-488 : Refactor Source classes in > hudi-utilities > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1248) [UMBRELLA] Tests cleanup and fixes
[ https://issues.apache.org/jira/browse/HUDI-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1248: -- Epic Name: Tests cleanup and fixes > [UMBRELLA] Tests cleanup and fixes > -- > > Key: HUDI-1248 > URL: https://issues.apache.org/jira/browse/HUDI-1248 > Project: Apache Hudi > Issue Type: Epic > Components: Testing >Reporter: sivabalan narayanan >Assignee: Raymond Xu >Priority: Critical > Labels: hudi-umbrellas, pull-request-available > > There are quite few tickets that requires some fixes to tests. Creating this > umbrella ticket to track all efforts. > > https://issues.apache.org/jira/browse/HUDI-1055 remove .parquet from tests. > https://issues.apache.org/jira/browse/HUDI-1033 ITTestRepairsCommand and > TestRepairsCommand > https://issues.apache.org/jira/browse/HUDI-1010 memory leak. > https://issues.apache.org/jira/browse/HUDI-997 memory leak > https://issues.apache.org/jira/browse/HUDI-664 : Adjust Logging levels to > reduce verbose log msgs in hudi-client > https://issues.apache.org/jira/browse/HUDI-623: Remove > UpgradePayloadFromUberToApache > https://issues.apache.org/jira/browse/HUDI-541: Replace variables/comments > named "data files" to "base file" > https://issues.apache.org/jira/browse/HUDI-347: Fix > TestHoodieClientOnCopyOnWriteStorage Tests with modular private methods > https://issues.apache.org/jira/browse/HUDI-323: Docker demo/integ-test > stdout/stderr output only available on process exit > https://issues.apache.org/jira/browse/HUDI-284: Need Tests for Hudi handling > of schema evolution > https://issues.apache.org/jira/browse/HUDI-154: Enable Rollback case in > HoodieRealtimeRecordReaderTest.testReader > https://issues.apache.org/jira/browse/HUDI-1143 timestamp micros. > https://issues.apache.org/jira/browse/HUDI-1989: flaky tests in > TestHoodieMergeOnReadTable -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1237) [UMBRELLA] Checkstyle, formatting, warnings, spotless
[ https://issues.apache.org/jira/browse/HUDI-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1237: -- Epic Name: Checkstyle, formatting, warnings, spotless > [UMBRELLA] Checkstyle, formatting, warnings, spotless > - > > Key: HUDI-1237 > URL: https://issues.apache.org/jira/browse/HUDI-1237 > Project: Apache Hudi > Issue Type: Epic > Components: Code Cleanup >Reporter: sivabalan narayanan >Assignee: leesf >Priority: Major > Labels: gsoc, gsoc2021, hudi-umbrellas, mentor > > Umbrella ticket to track all tickets related to checkstyle, spotless, > warnings etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1239) [UMBRELLA] Config clean up
[ https://issues.apache.org/jira/browse/HUDI-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1239: -- Epic Name: Config clean up > [UMBRELLA] Config clean up > -- > > Key: HUDI-1239 > URL: https://issues.apache.org/jira/browse/HUDI-1239 > Project: Apache Hudi > Issue Type: Epic > Components: Code Cleanup >Reporter: sivabalan narayanan >Assignee: Vinoth Chandar >Priority: Major > Labels: hudi-umbrellas > > Tracks all efforts to clean up configs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1238) [UMBRELLA] Perf test env
[ https://issues.apache.org/jira/browse/HUDI-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1238: -- Epic Name: Perf test env > [UMBRELLA] Perf test env > > > Key: HUDI-1238 > URL: https://issues.apache.org/jira/browse/HUDI-1238 > Project: Apache Hudi > Issue Type: Epic > Components: Testing >Reporter: sivabalan narayanan >Assignee: Rajesh Mahindra >Priority: Major > Labels: hudi-umbrellas > > We need to build a perf test environment which monitors metrics from a long > running test suite and displays via dashboards. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1236) [UMBRELLA] Integ Test suite infra
[ https://issues.apache.org/jira/browse/HUDI-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-1236: -- Epic Name: Integ Test suite infra > [UMBRELLA] Integ Test suite infra > -- > > Key: HUDI-1236 > URL: https://issues.apache.org/jira/browse/HUDI-1236 > Project: Apache Hudi > Issue Type: Epic > Components: Testing >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: Nishith Agarwal >Priority: Major > Labels: hudi-umbrellas > > Long running test suite that checks for correctness across all deployment > modes (batch/streaming) and writers (deltastreamer/spark) and readers (hive, > presto, spark) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-868) [UMBRELLA] Insert Overwrite API
[ https://issues.apache.org/jira/browse/HUDI-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-868: - Epic Name: Insert Overwrite API > [UMBRELLA] Insert Overwrite API > --- > > Key: HUDI-868 > URL: https://issues.apache.org/jira/browse/HUDI-868 > Project: Apache Hudi > Issue Type: Epic >Affects Versions: 0.9.0 >Reporter: satish >Assignee: satish >Priority: Major > Labels: hudi-umbrellas > > Usecases: > - Tables where the majority of records change every cycle. So it is likely > efficient to write new data instead of doing upserts. > - Operational tasks to fix a specific corrupted partition. We can do 'insert > overwrite' on that partition with records from the source. This can be much > faster than restore and replay for some data sources. > The functionality will be similar to hive definition of 'insert overwite'. > But, doing this in Hoodie will provide better isolation between writer and > readers. I can share possible implementation choices and some nuances if the > community thinks this is a useful feature to add. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-538) [UMBRELLA] Restructuring hudi client module for multi engine support
[ https://issues.apache.org/jira/browse/HUDI-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-538: - Epic Name: Restructuring hudi client module for multi engine support > [UMBRELLA] Restructuring hudi client module for multi engine support > > > Key: HUDI-538 > URL: https://issues.apache.org/jira/browse/HUDI-538 > Project: Apache Hudi > Issue Type: Epic > Components: Code Cleanup >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: hudi-umbrellas > > Hudi is currently tightly coupled with the Spark framework. It caused the > integration with other computing engine more difficult. We plan to decouple > it with Spark. This umbrella issue used to track this work. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-270) [UMBRELLA] Improve Hudi website UI and documentation
[ https://issues.apache.org/jira/browse/HUDI-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-270: - Epic Name: Improve Hudi website UI and documentation > [UMBRELLA] Improve Hudi website UI and documentation > > > Key: HUDI-270 > URL: https://issues.apache.org/jira/browse/HUDI-270 > Project: Apache Hudi > Issue Type: Epic > Components: Docs >Reporter: Bhavani Sudha Saktheeswaran >Assignee: Kyle Weller >Priority: Minor > Labels: hudi-umbrellas, pull-request-available > > This is an umbrella task of multiple tasks that aim to improve the website -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-466) [Umbrella] Record level, global low-latency index implementation
[ https://issues.apache.org/jira/browse/HUDI-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-466: - Epic Name: Record level, global low-latency index implementation > [Umbrella] Record level, global low-latency index implementation > > > Key: HUDI-466 > URL: https://issues.apache.org/jira/browse/HUDI-466 > Project: Apache Hudi > Issue Type: Epic > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: hudi-umbrellas > > Improve record indexing using record -> partitionpath, fileId look up. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-60) [UMBRELLA] Support Apache Beam for incremental tailing
[ https://issues.apache.org/jira/browse/HUDI-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-60: Epic Name: Support Apache Beam for incremental tailing > [UMBRELLA] Support Apache Beam for incremental tailing > -- > > Key: HUDI-60 > URL: https://issues.apache.org/jira/browse/HUDI-60 > Project: Apache Hudi > Issue Type: Epic > Components: Spark Integration, Utilities >Reporter: Vinoth Chandar >Priority: Major > Labels: gsoc, gsoc2021, hudi-umbrellas, mentor > > (More details to be added) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-57) [UMBRELLA] Support ORC Storage
[ https://issues.apache.org/jira/browse/HUDI-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha Saktheeswaran updated HUDI-57: Epic Name: Support ORC Storage > [UMBRELLA] Support ORC Storage > -- > > Key: HUDI-57 > URL: https://issues.apache.org/jira/browse/HUDI-57 > Project: Apache Hudi > Issue Type: Epic > Components: Hive Integration, Writer Core >Affects Versions: 0.9.0 >Reporter: Vinoth Chandar >Assignee: Teresa Kang >Priority: Major > Labels: hudi-umbrellas, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/uber/hudi/issues/68] > https://github.com/uber/hudi/issues/155 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007100483 ## CI report: * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007081241 ## CI report: * eecd338f6aa8c22150cc3a3abc28eb5c2535ef1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4945) * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot removed a comment on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007080163 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4530: [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data
hudi-bot commented on pull request #4530: URL: https://github.com/apache/hudi/pull/4530#issuecomment-1007099737 ## CI report: * 33af796c394961a4d9b16dcba8950e68ee018ea5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4956) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-3183) Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
[ https://issues.apache.org/jira/browse/HUDI-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3183: - Assignee: Yue Zhang > Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter > > > Key: HUDI-3183 > URL: https://issues.apache.org/jira/browse/HUDI-3183 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Assignee: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3183) Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
[ https://issues.apache.org/jira/browse/HUDI-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3183: -- Fix Version/s: 0.11.0 > Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter > > > Key: HUDI-3183 > URL: https://issues.apache.org/jira/browse/HUDI-3183 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Assignee: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0, 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HUDI-3183) Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
[ https://issues.apache.org/jira/browse/HUDI-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-3183. --- > Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter > > > Key: HUDI-3183 > URL: https://issues.apache.org/jira/browse/HUDI-3183 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Assignee: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] nsivabalan merged pull request #4521: [HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
nsivabalan merged pull request #4521: URL: https://github.com/apache/hudi/pull/4521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3183) Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter
[ https://issues.apache.org/jira/browse/HUDI-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3183: -- Fix Version/s: 0.10.1 > Fix wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter > > > Key: HUDI-3183 > URL: https://issues.apache.org/jira/browse/HUDI-3183 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.1 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[hudi] branch master updated (d7afc58 -> b2b23f5)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from d7afc58 [HUDI-3118] Add default HUDI_DIR in setupKafka.sh (#4460) add b2b23f5 [HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter (#4521) No new revisions were added by this update. Summary of changes: .../hudi/cli/commands/TestCommitsCommand.java | 47 ++ .../table/timeline/HoodieArchivedTimeline.java | 2 +- 2 files changed, 48 insertions(+), 1 deletion(-)
[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-1007081241 ## CI report: * eecd338f6aa8c22150cc3a3abc28eb5c2535ef1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4945) * fb90b5e4b8b1311eb029595d9d7133a8da593bec Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4957) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
