[GitHub] [hudi] jackiehff opened a new issue #2641: [SUPPORT]Build Hudi source with spark 3.1.1 compile error

jackiehff opened a new issue #2641: URL: https://github.com/apache/hudi/issues/2641 **Describe the problem you faced** When I build hudi source with newest spark 3.1.1 version, the terminal prints compile error : /Users/jackiehff/github/hudi/hudi-client/hudi-spark-client/src/main/sc

[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table

codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=desc) (d7d94c2) in

[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table

codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=desc) (d7d94c2) in

[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table

codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=desc) (d7d94c2) in

[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table

codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=desc) (338a5e0) in

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588933140 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -359,10 +388,21 @@ public abstract O b

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588922630 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/lock/LockManager.java ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache S

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588918517 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/lock/SimpleConcurrentFileWritesConflictResolutionStrategy.java ## @@ -0,0 +

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588918353 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java ## @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache S

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588918353 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java ## @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache S

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588918281 ## File path: hudi-common/src/main/java/org/apache/hudi/common/lock/LockProvider.java ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588918149 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java ## @@ -222,6 +225,17

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588917850 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/SimpleConcurrentFileWritesConflictResolutionStrategy.java ## @@

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588917643 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/SimpleConcurrentFileWritesConflictResolutionStrategy.java ## @@

[GitHub] [hudi] vburenin commented on a change in pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem

vburenin commented on a change in pull request #2500: URL: https://github.com/apache/hudi/pull/2500#discussion_r588912086 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java ## @@ -74,20 +75,21 @@ public HoodieLogFileReader(Fi

[GitHub] [hudi] codejoyan edited a comment on issue #2592: [SUPPORT] Does latest versions of Hudi (0.7.0, 0.6.0) work with Spark 2.3.0 when reading orc files?

codejoyan edited a comment on issue #2592: URL: https://github.com/apache/hudi/issues/2592#issuecomment-791988738 Hi @umehrot2 I tried using `databricks-avro` along with `avro 1.7.7` and `avro 1.8.2` but I still get `NoSuchMethod` exceptions. I am yet to try and build Hudi. Will

[GitHub] [hudi] codejoyan edited a comment on issue #2592: [SUPPORT] Does latest versions of Hudi (0.7.0, 0.6.0) work with Spark 2.3.0 when reading orc files?

codejoyan edited a comment on issue #2592: URL: https://github.com/apache/hudi/issues/2592#issuecomment-791988738 Hi @umehrot2 I tried using `databricks-avro` along with `avro 1.7.7` and `avro 1.8.2` but I still get `NoSuchMethod` exceptions. I am yet to try and build Hudi. Will

[GitHub] [hudi] codejoyan edited a comment on issue #2592: [SUPPORT] Does latest versions of Hudi (0.7.0, 0.6.0) work with Spark 2.3.0 when reading orc files?

codejoyan edited a comment on issue #2592: URL: https://github.com/apache/hudi/issues/2592#issuecomment-791988738 Hi @umehrot2 I tried using `databricks-avro` along with `avro 1.7.7` and `avro 1.8.2` but I still get `NoSuchMethod` exceptions still. I am yet to try and build Hudi.

[GitHub] [hudi] codejoyan commented on issue #2592: [SUPPORT] Does latest versions of Hudi (0.7.0, 0.6.0) work with Spark 2.3.0 when reading orc files?

codejoyan commented on issue #2592: URL: https://github.com/apache/hudi/issues/2592#issuecomment-791988738 Hi @umehrot2 I tried using `databricks-avro` along with `avro 1.7.7` and [avro 1.8.2](url) but I still get `NoSuchMethod` exceptions still. I am yet to try and build Hudi. W

[GitHub] [hudi] codejoyan edited a comment on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes

codejoyan edited a comment on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-791961553 Thanks @bvaradar and @nsivabalan. Please let me know how to improve the performance or if you need any further details to investigate. I used the below configurations (SIMPLE

[GitHub] [hudi] codecov-io edited a comment on pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem

codecov-io edited a comment on pull request #2500: URL: https://github.com/apache/hudi/pull/2500#issuecomment-776932935 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2500?src=pr&el=h1) Report > Merging [#2500](https://codecov.io/gh/apache/hudi/pull/2500?src=pr&el=desc) (3987af2) in

[GitHub] [hudi] nsivabalan edited a comment on pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem

nsivabalan edited a comment on pull request #2500: URL: https://github.com/apache/hudi/pull/2500#issuecomment-791962579 Finally able to get hold of it. I set up a cluster w/ gcloud dataproc and tested it out. Even existing PR didn't work(before my latest commit). fsDataInputStream.g

[GitHub] [hudi] nsivabalan commented on pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem

nsivabalan commented on pull request #2500: URL: https://github.com/apache/hudi/pull/2500#issuecomment-791962579 Finally able to get hold of it. I set up a cluster w/ gcloud dataproc and tested it out. Even existing PR didn't work(before my latest commit). fsDataInputStream.getWrapp

[GitHub] [hudi] codejoyan edited a comment on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes

codejoyan edited a comment on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-791961553 Thanks @bvaradar and @nsivabalan. Please let me know how to improve the performance or if you need any further details to investigate. I used the below configurations (SIMPLE

[GitHub] [hudi] codejoyan edited a comment on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes

codejoyan edited a comment on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-791961553 Thanks @bvaradar and @nsivabalan. Please let me know how to improve the performance or if you need any further details to investigate. I used the below configurations (SIMPLE

[GitHub] [hudi] codejoyan edited a comment on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes

codejoyan edited a comment on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-791961553 Thanks @bvaradar and @nsivabalan. Please let me know how to improve the performance or if you need any further details to investigate. I used the below configurations (SIMPLE

[GitHub] [hudi] codejoyan commented on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes

codejoyan commented on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-791961553 Thanks @bvaradar and @nsivabalan. Please let me know how to improve the performance. I used the below configurations (SIMPLE INDEX and turned off compaction) to speed up the insta

[jira] [Updated] (HUDI-1672) packaging fails w/ scala 12 (due to flink)

[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1672: -- Summary: packaging fails w/ scala 12 (due to flink) (was: mvn compilation fails w/ scal

[jira] [Commented] (HUDI-1672) mvn compilation fails w/ scala 12 (due to flink)

[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296538#comment-17296538 ] sivabalan narayanan commented on HUDI-1672: --- CC [~yanghua] [~leesf] this is in l

[jira] [Updated] (HUDI-1672) mvn compilation fails w/ scala 12 (due to flink)

[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1672: -- Labels: sev:critical user-support-issues (was: user-support-issues) > mvn compilation f

[jira] [Assigned] (HUDI-1672) mvn compilation fails w/ scala 12 (due to flink)

[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1672: - Assignee: vinoyang > mvn compilation fails w/ scala 12 (due to flink) > -

[jira] [Updated] (HUDI-1672) mvn compilation fails w/ scala 12 (due to flink)

[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1672: -- Labels: user-support-issues (was: ) > mvn compilation fails w/ scala 12 (due to flink)

[jira] [Created] (HUDI-1672) mvn compilation fails w/ scala 12 (due to flink)

sivabalan narayanan created HUDI-1672: - Summary: mvn compilation fails w/ scala 12 (due to flink) Key: HUDI-1672 URL: https://issues.apache.org/jira/browse/HUDI-1672 Project: Apache Hudi

[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table

codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr&el=desc) (b8517f2) in

[GitHub] [hudi] WilliamWhispell commented on issue #2515: [HUDI-1615] [SUPPORT] ERROR HoodieTimelineArchiveLog: Failed to archive commits

WilliamWhispell commented on issue #2515: URL: https://github.com/apache/hudi/issues/2515#issuecomment-791912551 Is the current recommended work around to set: 'hoodie.datasource.write.row.writer.enable': 'false', 'hoodie.fail.on.timeline.archiving': 'false' Any negative con

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588857743 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -359,10 +388,21 @@ public abstract O b

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588857743 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -359,10 +388,21 @@ public abstract O b

[GitHub] [hudi] n3nash commented on a change in pull request #2374: [HUDI-845] Added locking capability to allow multiple writers

n3nash commented on a change in pull request #2374: URL: https://github.com/apache/hudi/pull/2374#discussion_r588857493 ## File path: hudi-common/src/main/java/org/apache/hudi/common/lock/LockProvider.java ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation