[GitHub] [hudi] hudi-bot commented on pull request #7559: [HUDI-5447] Adding read support for record level index in metadata table

2023-01-03 Thread GitBox
hudi-bot commented on PR #7559: URL: https://github.com/apache/hudi/pull/7559#issuecomment-1370375541 ## CI report: * bdb5a418d12df6413bf94f3ff5149224139d6892 Azure:

[GitHub] [hudi] yihua commented on issue #7539: [SUPPORT]IllegalStateException: Trying to access closed classloader

2023-01-03 Thread GitBox
yihua commented on issue #7539: URL: https://github.com/apache/hudi/issues/7539#issuecomment-1370374380 @hbgstc123 Thanks for raising the issue. @danny0405 could you provide help here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hudi] yihua commented on issue #7541: [SUPPORT] It's very slow to savepoint a table which has many (75k) partitions

2023-01-03 Thread GitBox
yihua commented on issue #7541: URL: https://github.com/apache/hudi/issues/7541#issuecomment-1370374025 Hi @haoxie-aws Thanks for raising this issue. I confirm that the issue is due to unnecessary scanning of the metadata table when the number of partitions is large. When the metadata

[jira] [Updated] (HUDI-5486) Update 0.12.x release notes with Long Term Support

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5486: Story Points: 0 > Update 0.12.x release notes with Long Term Support >

[jira] [Updated] (HUDI-5485) Improve performance of savepoint with MDT

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5485: Story Points: 3 > Improve performance of savepoint with MDT > - > >

[GitHub] [hudi] hudi-bot commented on pull request #7559: [HUDI-5447] Adding read support for record level index in metadata table

2023-01-03 Thread GitBox
hudi-bot commented on PR #7559: URL: https://github.com/apache/hudi/pull/7559#issuecomment-1370372976 ## CI report: * 6857bb863668c8a6e83755a5d8a27812425ab586 Azure:

[GitHub] [hudi] yihua commented on issue #7565: [SUPPORT] Memory Exception when building BuildProfile

2023-01-03 Thread GitBox
yihua commented on issue #7565: URL: https://github.com/apache/hudi/issues/7565#issuecomment-1370371179 Hi @jomach Thanks for raising the issue. If you haven't, please check out the [Tuning Guide](https://hudi.apache.org/docs/tuning-guide/) for writing data to a Hudi table through a Spark

[GitHub] [hudi] hudi-bot commented on pull request #7597: [HUDI-5192] Prevent GH actions from running on trivial file changes

2023-01-03 Thread GitBox
hudi-bot commented on PR #7597: URL: https://github.com/apache/hudi/pull/7597#issuecomment-1370370194 ## CI report: * 9bdf2992ab2664d81eba0136b687949a7afdaa0d Azure:

[GitHub] [hudi] yihua commented on issue #7577: [SUPPORT]

2023-01-03 Thread GitBox
yihua commented on issue #7577: URL: https://github.com/apache/hudi/issues/7577#issuecomment-1370366726 Hi @Shagish thanks for raising this issue. Could you share the write configs of your Hudi Spark job? One possibility is that the metadata table might be out of sync with the data table

[jira] [Assigned] (HUDI-5293) Schema on read + reconcile schema fails w/ 0.12.1

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5293: - Assignee: Jonathan Vexler > Schema on read + reconcile schema fails w/ 0.12.1 >

[jira] [Assigned] (HUDI-5356) Call close on SparkRDDWriteClient several places

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5356: - Assignee: Jonathan Vexler > Call close on SparkRDDWriteClient several places >

[jira] [Assigned] (HUDI-5349) Clean up partially failed restore if any

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5349: - Assignee: Jonathan Vexler (was: sivabalan narayanan) > Clean up partially

[jira] [Closed] (HUDI-5370) Properly close file handles for Metadata writer

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-5370. - Resolution: Fixed > Properly close file handles for Metadata writer >

[jira] [Assigned] (HUDI-5361) Propagate Hudi properties set in Spark's SQLConf to Hudi

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5361: - Assignee: Jonathan Vexler (was: Alexey Kudinkin) > Propagate Hudi properties

[jira] [Assigned] (HUDI-5322) Bulk-insert (row-writing) is not rewriting incoming dataset into Writer's schema

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5322: - Assignee: Jonathan Vexler > Bulk-insert (row-writing) is not rewriting incoming

[jira] [Assigned] (HUDI-5372) Fix NPE caused by alter table add column

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5372: - Assignee: Jonathan Vexler > Fix NPE caused by alter table add column >

[jira] [Updated] (HUDI-5370) Properly close file handles for Metadata writer

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5370: -- Fix Version/s: 0.12.2 > Properly close file handles for Metadata writer >

[jira] [Updated] (HUDI-5375) Fix re-using of file readers w/ metadata table in FileIndex

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5375: -- Fix Version/s: 0.12.2 > Fix re-using of file readers w/ metadata table in FileIndex >

[jira] [Closed] (HUDI-5375) Fix re-using of file readers w/ metadata table in FileIndex

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-5375. - Resolution: Fixed > Fix re-using of file readers w/ metadata table in FileIndex >

[jira] [Closed] (HUDI-5383) Test 0.12.2 release branch

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-5383. - Assignee: sivabalan narayanan Resolution: Fixed > Test 0.12.2 release branch >

[jira] [Assigned] (HUDI-5386) Cleaning conflicts in occ mode

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5386: - Assignee: Jonathan Vexler > Cleaning conflicts in occ mode >

[jira] [Updated] (HUDI-5383) Test 0.12.2 release branch

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5383: -- Fix Version/s: 0.12.2 > Test 0.12.2 release branch > --- > >

[jira] [Assigned] (HUDI-5461) Upsert after renaming the table fails due to table props validation

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5461: - Assignee: Jonathan Vexler > Upsert after renaming the table fails due to table

[jira] [Assigned] (HUDI-5462) Spark-sql certain commands are only allowed with v2 tables

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5462: - Assignee: Jonathan Vexler > Spark-sql certain commands are only allowed with v2

[jira] [Assigned] (HUDI-5457) Configuration documentation for hoodie.datasource.write.operation needs to be updated

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5457: - Assignee: Jonathan Vexler > Configuration documentation for

[jira] [Updated] (HUDI-31) MOR - Allow partitioner to pick more than one small file for inserting new data #494

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-31: Fix Version/s: 0.11.0 > MOR - Allow partitioner to pick more than one small file for

[jira] [Assigned] (HUDI-4755) INSERT_OVERWRITE(/TABLE) in spark sql should not fail time travel queries for older timestamps

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4755: - Assignee: Jonathan Vexler (was: XiaoyuGeng) > INSERT_OVERWRITE(/TABLE) in spark

[jira] [Updated] (HUDI-31) MOR - Allow partitioner to pick more than one small file for inserting new data #494

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-31: Status: Open (was: In Progress) > MOR - Allow partitioner to pick more than one small file

[jira] [Closed] (HUDI-31) MOR - Allow partitioner to pick more than one small file for inserting new data #494

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-31. --- Resolution: Fixed > MOR - Allow partitioner to pick more than one small file for inserting new

[jira] [Assigned] (HUDI-5460) Spark-sql ALTER TABLE SET TBLPROPERTIES never fails

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5460: - Assignee: Jonathan Vexler > Spark-sql ALTER TABLE SET TBLPROPERTIES never fails

[jira] [Updated] (HUDI-3775) Allow for offline compaction of MOR tables via spark streaming

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3775: -- Sprint: 2022/09/05, 2023-01-09 (was: 2022/09/05) > Allow for offline compaction of MOR

[jira] [Assigned] (HUDI-3775) Allow for offline compaction of MOR tables via spark streaming

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3775: - Assignee: Jonathan Vexler (was: sivabalan narayanan) > Allow for offline

[GitHub] [hudi] yihua commented on issue #7589: Keep only clustered file(all) after cleaning

2023-01-03 Thread GitBox
yihua commented on issue #7589: URL: https://github.com/apache/hudi/issues/7589#issuecomment-1370361242 Hi @maheshguptags Thanks for the question. To clarify, are you asking to keep the new parquet files after clustering, which replace the compacted file groups that have parquet files?

[GitHub] [hudi] yihua commented on issue #7590: Failed to rollback s3://s3_bucket/xml commits 20221231041647333

2023-01-03 Thread GitBox
yihua commented on issue #7590: URL: https://github.com/apache/hudi/issues/7590#issuecomment-1370358130 Hi @koochiswathiTR Thanks for raising the issue. Could you share the Hudi write configs for this job? It looks like that the timeline server failed to start due to the underlying

[jira] [Updated] (HUDI-5463) Apply rollback commits from data table as rollbacks in MDT instead of Delta commit

2023-01-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5463: -- Sprint: 0.13.0 Final Sprint (was: 2023-01-09) > Apply rollback commits from data table

[GitHub] [hudi] yihua commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
yihua commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1370346300 @umehrot2 @rahil-c do you folks have any information on this question? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] yihua commented on issue #7596: [SUPPORT] java.lang.NoSuchMethodException: org.apache.hudi.utilities.sources.AvroKafkaSource when running HoodieDeltaStreamer

2023-01-03 Thread GitBox
yihua commented on issue #7596: URL: https://github.com/apache/hudi/issues/7596#issuecomment-1370343588 Hi @afuyo thanks for reporting this issue. This might be a configuration issue. Have you tried the same Deltastreamer job in the [Docker Demo](https://hudi.apache.org/docs/docker_demo)

[GitHub] [hudi] maddy2u commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
maddy2u commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1370299596 Thanks for the information @tooptoop4 ! I will keep an eye on the PR for updates. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hudi] hudi-bot commented on pull request #7597: [HUDI-5192] Prevent GH actions from running on trivial file changes

2023-01-03 Thread GitBox
hudi-bot commented on PR #7597: URL: https://github.com/apache/hudi/pull/7597#issuecomment-1370269415 ## CI report: * 9bdf2992ab2664d81eba0136b687949a7afdaa0d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7597: [HUDI-5192] Prevent GH actions from running on trivial file changes

2023-01-03 Thread GitBox
hudi-bot commented on PR #7597: URL: https://github.com/apache/hudi/pull/7597#issuecomment-1370264023 ## CI report: * 9bdf2992ab2664d81eba0136b687949a7afdaa0d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] yihua commented on issue #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

2023-01-03 Thread GitBox
yihua commented on issue #7595: URL: https://github.com/apache/hudi/issues/7595#issuecomment-1370256718 Hi @BalaMahesh Thanks for raising the issue. To better triage this, could you provide more details about the Hudi table, partitioned or non-partitioned table, how many partitions if

[jira] [Updated] (HUDI-5192) GH actions and azure ci tests run even for trivial fixes

2023-01-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5192: - Labels: pull-request-available (was: ) > GH actions and azure ci tests run even for trivial

[GitHub] [hudi] jonvex opened a new pull request, #7597: [HUDI-5192] Prevent GH actions from running on trivial file changes

2023-01-03 Thread GitBox
jonvex opened a new pull request, #7597: URL: https://github.com/apache/hudi/pull/7597 ### Change Logs If a pr only has changes to files with extensions of `bmp, gif, jpg, jpeg, md, pdf, png, svg` do not run GH actions. ### Impact Reduce waiting time before gh actions

[GitHub] [hudi] tooptoop4 commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
tooptoop4 commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1370192167 not sure when. for 2nd qn, at least in trino itself I think u can test that PR on a local patch build, I assume it will work as long as u define the right IAM permissions -- This

[GitHub] [hudi] maddy2u commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
maddy2u commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1370181877 @tooptoop4 : Any expectation on when this will be merged to Trinodb? I think it takes a while before it is made available within Athena post this. Not sure if this is the

[jira] [Updated] (HUDI-5493) Revisit the archival process wrt clustering

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5493: Description: [https://github.com/apache/hudi/pull/7568]   The above PR fixes the case where the archival

[jira] [Updated] (HUDI-5493) Revisit the archival process wrt clustering

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5493: Fix Version/s: 0.14.0 > Revisit the archival process wrt clustering >

[jira] [Updated] (HUDI-5493) Revisit the archival process wrt clustering

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5493: Priority: Critical (was: Major) > Revisit the archival process wrt clustering >

[GitHub] [hudi] yihua commented on pull request #7568: [HUDI-5341] CleanPlanner retains earliest commits must not be later than earliest pending commit

2023-01-03 Thread GitBox
yihua commented on PR #7568: URL: https://github.com/apache/hudi/pull/7568#issuecomment-1370168033 [HUDI-5493](https://issues.apache.org/jira/browse/HUDI-5493) for revisiting the logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Created] (HUDI-5493) Revisit the archival process wrt clustering

2023-01-03 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5493: --- Summary: Revisit the archival process wrt clustering Key: HUDI-5493 URL: https://issues.apache.org/jira/browse/HUDI-5493 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] soumilshah1995 commented on issue #7591: [SUPPORT] Kinesis Data Analytics Flink1.13 to HUDI

2023-01-03 Thread GitBox
soumilshah1995 commented on issue #7591: URL: https://github.com/apache/hudi/issues/7591#issuecomment-1370164850 Here is details again i have tried again this morning Please note This time i am on US-WEST-2 previously i was trying on US-EAST-1 Kinesis Streams

[jira] [Updated] (HUDI-3411) Incorrect Record Key Field property Handling

2023-01-03 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-3411: -- Status: Open (was: In Progress) > Incorrect Record Key Field property Handling >

[jira] [Updated] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5023: Status: Patch Available (was: In Progress) > Add new Executor avoiding Queueing in the write-path >

[jira] [Updated] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5023: Status: In Progress (was: Reopened) > Add new Executor avoiding Queueing in the write-path >

[jira] [Updated] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5023: Sprint: 2022/11/15, 2022/11/29, 0.13.0 Final Sprint (was: 2022/11/15, 2022/11/29) > Add new Executor

[jira] [Updated] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5023: Reviewers: Ethan Guo, sivabalan narayanan (was: sivabalan narayanan) > Add new Executor avoiding Queueing

[jira] [Reopened] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reopened HUDI-5023: - Assignee: Alexey Kudinkin (was: Yue Zhang) > Add new Executor avoiding Queueing in the write-path >

[GitHub] [hudi] yihua commented on a diff in pull request #7476: [HUDI-5023] Switching default Write Executor type to `SIMPLE`

2023-01-03 Thread GitBox
yihua commented on code in PR #7476: URL: https://github.com/apache/hudi/pull/7476#discussion_r1060867754 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -2449,12 +2450,17 @@ public Builder withWriteBufferLimitBytes(int

[GitHub] [hudi] hudi-bot commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
hudi-bot commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1370124079 ## CI report: * 96c0d86652aa342ad9b13bccc30020830ef8d204 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1370118321 ## CI report: * 1ac267ba9af690ecd47f74f60c34851387aee9eb Azure:

[GitHub] [hudi] jonvex commented on issue #5687: [SUPPORT]hudi sql parser ignores all exceptions of spark sql parser

2023-01-03 Thread GitBox
jonvex commented on issue #5687: URL: https://github.com/apache/hudi/issues/5687#issuecomment-1370096424 I was not able to reproduce the error by running ``` select CAST(-123456789 AS TIMESTAMP) as de; ``` In spark-sql with or without hudi. I was able to confirm that I did

[GitHub] [hudi] rahil-c commented on pull request #7584: [HUDI-5205] Support Flink 1.16.0

2023-01-03 Thread GitBox
rahil-c commented on PR #7584: URL: https://github.com/apache/hudi/pull/7584#issuecomment-1370087863 @stayrascal @danny0405 Thanks for making this change, just to confirm this is targeted for Hudi 0.13.0? -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Updated] (HUDI-5442) Fix HiveHoodieTableFileIndex to use lazy listing

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5442: Status: In Progress (was: Open) > Fix HiveHoodieTableFileIndex to use lazy listing >

[jira] [Updated] (HUDI-5442) Fix HiveHoodieTableFileIndex to use lazy listing

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5442: Status: Open (was: Patch Available) > Fix HiveHoodieTableFileIndex to use lazy listing >

[jira] [Updated] (HUDI-5485) Improve performance of savepoint with MDT

2023-01-03 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5485: Status: In Progress (was: Open) > Improve performance of savepoint with MDT >

[GitHub] [hudi] jonvex commented on issue #7494: FileNotFoundException while writing dataframe to local file system

2023-01-03 Thread GitBox
jonvex commented on issue #7494: URL: https://github.com/apache/hudi/issues/7494#issuecomment-1370067142 Here are the steps that I tried: 1. Download [spark-3.3.1-bin-hadoop3.tgz](https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz) 2. Set the environment

[GitHub] [hudi] xushiyan commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
xushiyan commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1370045864 can you also edit PR description as per the template? it's failing the validate pr check. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] xushiyan commented on a diff in pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
xushiyan commented on code in PR #7572: URL: https://github.com/apache/hudi/pull/7572#discussion_r1060804456 ## hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java: ## @@ -69,12 +69,12 @@ public RetryHelper(long maxRetryIntervalMs, int maxRetryNumbers, long

[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager

2023-01-03 Thread GitBox
xushiyan commented on code in PR #6732: URL: https://github.com/apache/hudi/pull/6732#discussion_r1060687626 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java: ## @@ -85,6 +91,8 @@ protected BaseHoodieClient(HoodieEngineContext

[GitHub] [hudi] jonvex commented on issue #7494: FileNotFoundException while writing dataframe to local file system

2023-01-03 Thread GitBox
jonvex commented on issue #7494: URL: https://github.com/apache/hudi/issues/7494#issuecomment-1369997447 @idatya I'm still looking into this, but it would be helpful to know if are you using one of the Hudi release branches or if you are using master? -- This is an automated message from

[GitHub] [hudi] nsivabalan commented on issue #7574: [SUPPORT] Upsert job failing while upgrading from 0.7 to 0.10.1

2023-01-03 Thread GitBox
nsivabalan commented on issue #7574: URL: https://github.com/apache/hudi/issues/7574#issuecomment-1369967197 @amitbans : can you paste the write configs you are using. also, screen short of jobs and stages page from sparkUI as well. For the particular job and stage thats failing, if

[GitHub] [hudi] hudi-bot commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
hudi-bot commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1369961264 ## CI report: * f6a856197ee645a4dcb155a7616c6363829c5d37 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
hudi-bot commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1369953527 ## CI report: * f6a856197ee645a4dcb155a7616c6363829c5d37 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1369945551 ## CI report: * 1ac267ba9af690ecd47f74f60c34851387aee9eb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
hudi-bot commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1369945409 ## CI report: * f6a856197ee645a4dcb155a7616c6363829c5d37 Azure:

[GitHub] [hudi] afuyo opened a new issue, #7596: [SUPPORT] java.lang.NoSuchMethodException: org.apache.hudi.utilities.sources.AvroKafkaSource when running HoodieDeltaStreamer

2023-01-03 Thread GitBox
afuyo opened a new issue, #7596: URL: https://github.com/apache/hudi/issues/7596 - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Yes **Describe the problem you faced** Exception when running HoodieDeltaStreamer: Could not load class

[GitHub] [hudi] hudi-bot commented on pull request #7584: [HUDI-5205] Support Flink 1.16.0

2023-01-03 Thread GitBox
hudi-bot commented on PR #7584: URL: https://github.com/apache/hudi/pull/7584#issuecomment-1369862519 ## CI report: * efd000d200790d748acb49ad79cd2ff09db64d73 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7584: [HUDI-5205] Support Flink 1.16.0

2023-01-03 Thread GitBox
hudi-bot commented on PR #7584: URL: https://github.com/apache/hudi/pull/7584#issuecomment-1369855337 ## CI report: * 1e9fa1cc5c993845a338616532ebc189772a181c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7584: [HUDI-5205] Support Flink 1.16.0

2023-01-03 Thread GitBox
hudi-bot commented on PR #7584: URL: https://github.com/apache/hudi/pull/7584#issuecomment-1369848321 ## CI report: * 1e9fa1cc5c993845a338616532ebc189772a181c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7593: [HUDI-5492] spark call command show_compaction doesn't return the com…

2023-01-03 Thread GitBox
hudi-bot commented on PR #7593: URL: https://github.com/apache/hudi/pull/7593#issuecomment-1369841163 ## CI report: * 8dac276274844f65a48d2e877a3cb1ed1d4ec3e3 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1369841004 ## CI report: * 1ac267ba9af690ecd47f74f60c34851387aee9eb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
hudi-bot commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1369748183 ## CI report: * f6a856197ee645a4dcb155a7616c6363829c5d37 Azure:

[GitHub] [hudi] tooptoop4 commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
tooptoop4 commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1369727346 https://github.com/trinodb/trino/pull/15084 hasn't been merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] BalaMahesh opened a new issue, #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

2023-01-03 Thread GitBox
BalaMahesh opened a new issue, #7595: URL: https://github.com/apache/hudi/issues/7595 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at

[GitHub] [hudi] minihippo commented on a diff in pull request #5064: [HUDI-3654] Add new module `hudi-metaserver`

2023-01-03 Thread GitBox
minihippo commented on code in PR #5064: URL: https://github.com/apache/hudi/pull/5064#discussion_r1060528770 ## hudi-platform-service/hudi-metaserver/src/main/java/org/apache/hudi/metaserver/client/HoodieMetaserverClientImp.java: ## @@ -108,35 +108,42 @@ public void

[GitHub] [hudi] minihippo commented on a diff in pull request #5064: [HUDI-3654] Add new module `hudi-metaserver`

2023-01-03 Thread GitBox
minihippo commented on code in PR #5064: URL: https://github.com/apache/hudi/pull/5064#discussion_r1060528770 ## hudi-platform-service/hudi-metaserver/src/main/java/org/apache/hudi/metaserver/client/HoodieMetaserverClientImp.java: ## @@ -108,35 +108,42 @@ public void

[GitHub] [hudi] soumilshah1995 commented on issue #7591: [SUPPORT] Kinesis Data Analytics Flink1.13 to HUDI

2023-01-03 Thread GitBox
soumilshah1995 commented on issue #7591: URL: https://github.com/apache/hudi/issues/7591#issuecomment-1369696915 Hey Danny Yes I did try that yesterday I could not get it to work. I keep getting this same error message On Tue, Jan 3, 2023 at 12:08 AM Danny Chan

[GitHub] [hudi] danny0405 commented on pull request #6524: [HUDI-4717] CompactionCommitEvent message corrupted when sent by compact_task

2023-01-03 Thread GitBox
danny0405 commented on PR #6524: URL: https://github.com/apache/hudi/pull/6524#issuecomment-1369691323 > > Does #7399 solve your problem here ? > > Yeah, we cherry-pick [#7399](https://github.com/apache/hudi/pull/7399). But if user enable latency-marker, there is still thread-safety

[GitHub] [hudi] minihippo commented on pull request #7572: [HUDI-5483]Make retryhelper more suitable for common use.

2023-01-03 Thread GitBox
minihippo commented on PR #7572: URL: https://github.com/apache/hudi/pull/7572#issuecomment-1369688545 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1369686205 ## CI report: * 92b8c60d309978d24aa33badba2cd4d9f0640b18 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1369681168 ## CI report: * 70796357cee7f956d7ad595f27fa2a8e8524d798 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7593: [HUDI-5492] spark call command show_compaction doesn't return the com…

2023-01-03 Thread GitBox
hudi-bot commented on PR #7593: URL: https://github.com/apache/hudi/pull/7593#issuecomment-1369676071 ## CI report: * e7dad5ff4526bd3ce1f93c0a6143f919eeb57bb4 Azure:

[jira] [Updated] (HUDI-5490) Investigate test failures w/ record level index for existing tests

2023-01-03 Thread Lokesh Jain (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HUDI-5490: -- Description: Enable record level index for some of the chosen tests (30 to 40) and ensure they

[GitHub] [hudi] IvyIvy1109 commented on issue #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
IvyIvy1109 commented on issue #7594: URL: https://github.com/apache/hudi/issues/7594#issuecomment-1369635307 > Hi, > > Simple question - Does AWS Athena support Hudi Time Travel? > > We see good support for Iceberg tables

[GitHub] [hudi] hudi-bot commented on pull request #7593: [HUDI-5492] spark call command show_compaction doesn't return the com…

2023-01-03 Thread GitBox
hudi-bot commented on PR #7593: URL: https://github.com/apache/hudi/pull/7593#issuecomment-1369614570 ## CI report: * e7dad5ff4526bd3ce1f93c0a6143f919eeb57bb4 Azure:

[GitHub] [hudi] maddy2u opened a new issue, #7594: [SUPPORT] Hudi Time Travel from Athena

2023-01-03 Thread GitBox
maddy2u opened a new issue, #7594: URL: https://github.com/apache/hudi/issues/7594 Hi, Simple question - Does AWS Athena support Hudi Time Travel? We see good support for Iceberg tables [here](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-table-data.html) I

[GitHub] [hudi] hudi-bot commented on pull request #7355: [HUDI-5308] Hive query returns null when the where clause has a partition field

2023-01-03 Thread GitBox
hudi-bot commented on PR #7355: URL: https://github.com/apache/hudi/pull/7355#issuecomment-1369595486 ## CI report: * efcb91b1f4a577a016a82bd4a6e0a203d04c251f Azure:

[GitHub] [hudi] yabha-isomap commented on issue #7381: [SUPPORT] Dependencies required for running flink hudi quickstart.

2023-01-03 Thread GitBox
yabha-isomap commented on issue #7381: URL: https://github.com/apache/hudi/issues/7381#issuecomment-1369565883 Thanks @danny0405 . When I use following dependency ```xml org.apache.hadoop hadoop-common

[GitHub] [hudi] yyar commented on issue #7472: [SUPPORT] Too many metadata timeline file caused by old rollback active timeline

2023-01-03 Thread GitBox
yyar commented on issue #7472: URL: https://github.com/apache/hudi/issues/7472#issuecomment-1369538405 @yihua Okay. Since I'm using 0.11.1 version, I'll cherry-pick [these two commits](https://github.com/apache/hudi/pull/7580/commits) based on 0.11.1 release version. -- This is an

[GitHub] [hudi] hudi-bot commented on pull request #7573: [HUDI-5484] Avoid using GenericRecord in ColumnStatMetadata

2023-01-03 Thread GitBox
hudi-bot commented on PR #7573: URL: https://github.com/apache/hudi/pull/7573#issuecomment-1369524807 ## CI report: * 70796357cee7f956d7ad595f27fa2a8e8524d798 Azure:

<    1   2   3   >