[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3167 +/- ## = Coverage 46.26% 46.26% Complexity 5362 5362 = Files 920 920 Lines 3982439823-1 Branches 4289 4289 = Hits 1842518425 + Misses1951819517-1 Partials 1881 1881 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (ø)` | | | hudiflink | `61.52% <ø> (+0.01%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.06% <ø> (ø)` | | | hudisync | `54.05% <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/streamer/FlinkStreamerConfig.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9GbGlua1N0cmVhbWVyQ29uZmlnLmphdmE=) | `0.00% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [e99a6b0...47941e9](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `3.81%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## - Coverage 46.26% 42.45% -3.82% + Complexity 5362 4737 -625 Files 920 842 -78 Lines 3982435461-4363 Branches 4289 3619 -670 - Hits 1842515055-3370 + Misses1951819011 -507 + Partials 1881 1395 -486 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (ø)` | | | hudiflink | `61.52% <ø> (+0.01%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/streamer/FlinkStreamerConfig.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9GbGlua1N0cmVhbWVyQ29uZmlnLmphdmE=) | `0.00% <ø> (ø)` | | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...he/hudi/hive/replication/GlobalHiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvcmVwbGljYXRpb24vR2xvYmFsSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-94.74%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.86%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## = - Coverage 46.26% 28.39% -17.87% + Complexity 5362 1255 -4107 = Files 920 375 -545 Lines 3982414304-25520 Branches 4289 1459 -2830 = - Hits 18425 4062-14363 + Misses19518 9946 -9572 + Partials 1881 296 -1585 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] nsivabalan commented on a change in pull request #3142: [HUDI-1483] Support async clustering for deltastreamer and Spark streaming
nsivabalan commented on a change in pull request #3142: URL: https://github.com/apache/hudi/pull/3142#discussion_r659456141 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncClusteringService.java ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hudi.async; + +import org.apache.hudi.client.AbstractClusteringClient; +import org.apache.hudi.client.AbstractHoodieWriteClient; +import org.apache.hudi.common.table.timeline.HoodieInstant; +import org.apache.hudi.common.util.collection.Pair; +import org.apache.hudi.exception.HoodieIOException; + +import org.apache.log4j.LogManager; +import org.apache.log4j.Logger; + +import java.io.IOException; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; +import java.util.stream.IntStream; + +public abstract class AsyncClusteringService extends HoodieAsyncService { Review comment: java docs would be nice. ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncClusteringService.java ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hudi.async; + +import org.apache.hudi.client.AbstractClusteringClient; +import org.apache.hudi.client.AbstractHoodieWriteClient; +import org.apache.hudi.common.table.timeline.HoodieInstant; +import org.apache.hudi.common.util.collection.Pair; +import org.apache.hudi.exception.HoodieIOException; + +import org.apache.log4j.LogManager; +import org.apache.log4j.Logger; + +import java.io.IOException; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; +import java.util.stream.IntStream; + +public abstract class AsyncClusteringService extends HoodieAsyncService { + + private static final Logger LOG = LogManager.getLogger(AsyncClusteringService.class); + + private final int maxConcurrentClustering; + private transient AbstractClusteringClient clusteringClient; + private transient BlockingQueue pendingClustering = new LinkedBlockingQueue<>(); + private transient ReentrantLock queueLock = new ReentrantLock(); + private transient Condition consumed = queueLock.newCondition(); + + public AsyncClusteringService(AbstractHoodieWriteClient writeClient) { +this(writeClient, false); + } + + public AsyncClusteringService(AbstractHoodieWriteClient writeClient, boolean runInDaemonMode) { +super(runInDaemonMode); +this.clusteringClient = createClusteringClient(writeClient); +this.maxConcurrentClustering = 1; + } + + protected abstract AbstractClusteringClient createClusteringClient(AbstractHoodieWriteClient client); + + public void enqueuePendingClustering(HoodieInstant instant) { +LOG.info("Enqueuing new pending clustering instant: " + instant.getTimestamp()); +pendingClustering.add(instant); +
[GitHub] [hudi] hudi-bot edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
hudi-bot edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862905306 ## CI report: * ac71607f82fb615ca9e47e6ac00ace68b2e96a99 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=479) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
codecov-commenter edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862906759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
codecov-commenter edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862906759 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3100](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ac71607) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.89%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3100/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3100 +/- ## = - Coverage 46.26% 28.36% -17.90% + Complexity 5362 1254 -4108 = Files 920 376 -544 Lines 3982414315-25509 Branches 4289 1461 -2828 = - Hits 18425 4061-14364 + Misses19518 9957 -9561 + Partials 1881 297 -1584 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.27% <0.00%> (-8.19%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../strategy/ExplicitPartitionCompactionStrategy.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L3N0cmF0ZWd5L0V4cGxpY2l0UGFydGl0aW9uQ29tcGFjdGlvblN0cmF0ZWd5LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.86%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## = - Coverage 46.26% 28.39% -17.87% + Complexity 5362 1255 -4107 = Files 920 375 -545 Lines 3982414304-25520 Branches 4289 1459 -2830 = - Hits 18425 4062-14363 + Misses19518 9946 -9572 + Partials 1881 296 -1585 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3145: [Docs] Improving Hudi Configurations docs
zhangyue19921010 edited a comment on pull request #3145: URL: https://github.com/apache/hudi/pull/3145#issuecomment-869353685 Here is pic before this PR that Contents and tables will cover each other https://user-images.githubusercontent.com/69956021/123582366-4003cf80-d810-11eb-904b-8867271e69cd.png;> Here is the pic after this pr that tables are scrollable. https://user-images.githubusercontent.com/69956021/123582426-5c077100-d810-11eb-8bda-b87338178efe.png;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3145: [Docs] Improving Hudi Configurations docs
zhangyue19921010 commented on pull request #3145: URL: https://github.com/apache/hudi/pull/3145#issuecomment-869353685 Here is pic before this PR that Contents and tables will not cover each other https://user-images.githubusercontent.com/69956021/123582366-4003cf80-d810-11eb-904b-8867271e69cd.png;> Here is the pic after this pr that tables are scrollable. https://user-images.githubusercontent.com/69956021/123582426-5c077100-d810-11eb-8bda-b87338178efe.png;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f14ffb1f08820146e5d26616aa9b956ff99ec604 UNKNOWN * 131fd8fbe3f7e87e6b9939325e446564b230f7c6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=478) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3158: [WIP][HUDI-1860] Add INSERT_OVERWRITE support to DeltaStreamer
codope commented on a change in pull request #3158: URL: https://github.com/apache/hudi/pull/3158#discussion_r659470200 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java ## @@ -1242,11 +1242,11 @@ private void verifyInsertOverwritePartitionHandling(int batch1RecordsCount, int List insertsAndUpdates2 = new ArrayList<>(); insertsAndUpdates2.addAll(inserts2); JavaRDD insertAndUpdatesRDD2 = jsc.parallelize(insertsAndUpdates2, 2); -HoodieWriteResult writeResult = client.insertOverwrite(insertAndUpdatesRDD2, commitTime2); -statuses = writeResult.getWriteStatuses().collect(); +JavaRDD writeStatusJavaRDD = client.insertOverwrite(insertAndUpdatesRDD2, commitTime2); +statuses = writeStatusJavaRDD.collect(); assertNoWriteErrors(statuses); - -assertEquals(batch1Buckets, new HashSet<>(writeResult.getPartitionToReplaceFileIds().get(testPartitionPath))); +// todo fix +// assertEquals(batch1Buckets, new HashSet<>(writeResult.getPartitionToReplaceFileIds().get(testPartitionPath))); Review comment: I assume it will be fixed in this PR itself. ## File path: hudi-client/hudi-java-client/src/main/java/org/apache/hudi/client/HoodieJavaWriteClient.java ## @@ -140,6 +140,19 @@ public boolean commit(String instantTime, return postWrite(result, instantTime, table); } + @Override + public List insertOverwrite(List> records, String instantTime) { Review comment: Why do we need this operation support in `HoodieJavaWriteClient`? Shouldn't the `SparkRDDWriteClient` support be enough for deltastreamer? ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java ## @@ -260,7 +260,8 @@ private boolean onDeltaSyncShutdown(boolean error) { public long sourceLimit = Long.MAX_VALUE; @Parameter(names = {"--op"}, description = "Takes one of these values : UPSERT (default), INSERT (use when input " -+ "is purely new data/inserts to gain speed)", converter = OperationConverter.class) ++ "is purely new data/inserts to gain speed) INSERT_OVERWRITE (use when input record can overwrite existing " Review comment: We should also look at `DeltaSync::writeToSink` to trigger the insert_overwrite operation. ## File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java ## @@ -173,9 +173,7 @@ public static HoodieWriteResult doWriteOperation(SparkRDDWriteClient client, Jav case UPSERT: return new HoodieWriteResult(client.upsert(hoodieRecords, instantTime)); case INSERT_OVERWRITE: -return client.insertOverwrite(hoodieRecords, instantTime); - case INSERT_OVERWRITE_TABLE: Review comment: Why are we removing `INSERT_OVERWRITE_TABLE` case here? ## File path: hudi-client/hudi-java-client/src/main/java/org/apache/hudi/client/HoodieJavaWriteClient.java ## @@ -140,6 +140,19 @@ public boolean commit(String instantTime, return postWrite(result, instantTime, table); } + @Override + public List insertOverwrite(List> records, String instantTime) { +HoodieTable>, List, List> table = +getTableAndInitCtx(WriteOperationType.INSERT_OVERWRITE, instantTime); +table.validateUpsertSchema(); Review comment: We should validate insert schema here right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
codecov-commenter edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862906759 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3100](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ac71607) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `30.12%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3100/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3100 +/- ## = - Coverage 46.26% 16.14% -30.13% + Complexity 5362 466 -4896 = Files 920 280 -640 Lines 3982410864-28960 Branches 4289 1109 -3180 = - Hits 18425 1754-16671 + Misses19518 8950-10568 + Partials 1881 160 -1721 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../strategy/ExplicitPartitionCompactionStrategy.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L3N0cmF0ZWd5L0V4cGxpY2l0UGFydGl0aW9uQ29tcGFjdGlvblN0cmF0ZWd5LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.87%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## = - Coverage 46.26% 28.39% -17.88% + Complexity 5362 1254 -4108 = Files 920 375 -545 Lines 3982414304-25520 Branches 4289 1459 -2830 = - Hits 18425 4061-14364 + Misses19518 9946 -9572 + Partials 1881 297 -1584 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
hudi-bot edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869325847 ## CI report: * 47941e913528dbacd05f222c0d98f6ab2a34db31 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=477) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
wangxianghu commented on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869345067 reopen to trigger the ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu closed pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
wangxianghu closed pull request #3167: URL: https://github.com/apache/hudi/pull/3167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (131fd8f) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **increase** coverage by `2.00%`. > The diff coverage is `86.88%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## + Coverage 44.28% 46.29% +2.00% - Complexity 4573 5367 +794 Files 824 920 +96 Lines 3639139832+3441 Branches 3939 4290 +351 + Hits 1611718439+2322 - Misses1852819517 +989 - Partials 1746 1876 +130 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (+14.00%)` | :arrow_up: | | hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `61.50% <ø> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `66.96% <55.55%> (-0.11%)` | :arrow_down: | | hudisync | `55.24% <92.30%> (+1.19%)` | :arrow_up: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.26% <ø> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `73.91% <ø> (ø)` | | | [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrU3FsV3JpdGVyLnNjYWxh) | `70.21% <40.00%> (-0.61%)` | :arrow_down: | | [...src/main/scala/org/apache/hudi/DefaultSource.scala](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0RlZmF1bHRTb3VyY2Uuc2NhbGE=) | `74.77% <75.00%> (-0.46%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `77.84% <91.48%> (+5.49%)` | :arrow_up: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `98.03% <100.00%> (+0.21%)` | :arrow_up: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (131fd8f) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **increase** coverage by `0.05%`. > The diff coverage is `86.88%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## + Coverage 44.28% 44.34% +0.05% - Complexity 4573 4579 +6 Files 824 824 Lines 3639136381 -10 Branches 3939 3938 -1 + Hits 1611716132 +15 + Misses1852818510 -18 + Partials 1746 1739 -7 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `16.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `61.50% <ø> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `66.96% <55.55%> (-0.11%)` | :arrow_down: | | hudisync | `55.24% <92.30%> (+1.19%)` | :arrow_up: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.26% <ø> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `73.91% <ø> (ø)` | | | [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrU3FsV3JpdGVyLnNjYWxh) | `70.21% <40.00%> (-0.61%)` | :arrow_down: | | [...src/main/scala/org/apache/hudi/DefaultSource.scala](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0RlZmF1bHRTb3VyY2Uuc2NhbGE=) | `74.77% <75.00%> (-0.46%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `77.84% <91.48%> (+5.49%)` | :arrow_up: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `98.03% <100.00%> (+0.21%)` | :arrow_up: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (131fd8f) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `5.19%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## - Coverage 44.28% 39.09% -5.20% + Complexity 4573 3678 -895 Files 824 710 -114 Lines 3639130271-6120 Branches 3939 3030 -909 - Hits 1611711835-4282 + Misses1852817278-1250 + Partials 1746 1158 -588 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `16.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `61.50% <ø> (+0.25%)` | :arrow_up: | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] zhangyue19921010 commented on pull request #3145: [Docs] Improving Hudi Configurations docs
zhangyue19921010 commented on pull request #3145: URL: https://github.com/apache/hudi/pull/3145#issuecomment-869343192 Hi @leesf Thanks for your review! I generate the site in my local and here is the pic : ![0 0 0 0_4000_docs_configurations html](https://user-images.githubusercontent.com/69956021/123579939-66733c00-d80b-11eb-9425-1f13a35d4236.png) Some tables are relatively large so that I adopt `scroll` to fit in; Contents and tables will not cover each other. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.87%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## = - Coverage 46.26% 28.39% -17.88% + Complexity 5362 1254 -4108 = Files 920 375 -545 Lines 3982414304-25520 Branches 4289 1459 -2830 = - Hits 18425 4061-14364 + Misses19518 9946 -9572 + Partials 1881 297 -1584 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
codecov-commenter edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862906759 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3100](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ac71607) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `43.19%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3100/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3100 +/- ## - Coverage 46.26% 3.07% -43.20% + Complexity 5362 82 -5280 Files 920 280 -640 Lines 39824 10864-28960 Branches 42891109 -3180 - Hits 18425 334-18091 + Misses19518 10504 -9014 + Partials 1881 26 -1855 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.74%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3100?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../strategy/ExplicitPartitionCompactionStrategy.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L3N0cmF0ZWd5L0V4cGxpY2l0UGFydGl0aW9uQ29tcGFjdGlvblN0cmF0ZWd5LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3100/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
hudi-bot edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865079044 ## CI report: * 548390e6bd363be3ec299be8f2a2a3703f36566a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=476) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
hudi-bot edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862905306 ## CI report: * e2423c075b07740a81cf4c275348a04ca58de9b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=349) * ac71607f82fb615ca9e47e6ac00ace68b2e96a99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=479) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
hudi-bot edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862905306 ## CI report: * e2423c075b07740a81cf4c275348a04ca58de9b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=349) * ac71607f82fb615ca9e47e6ac00ace68b2e96a99 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `30.10%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3167 +/- ## = - Coverage 46.26% 16.16% -30.11% + Complexity 5362 466 -4896 = Files 920 279 -641 Lines 3982410853-28971 Branches 4289 1107 -3182 = - Hits 18425 1754-16671 + Misses19518 8939-10579 + Partials 1881 160 -1721 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (131fd8f) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `28.17%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## = - Coverage 44.28% 16.11% -28.18% + Complexity 4573 467 -4106 = Files 824 279 -545 Lines 3639110893-25498 Branches 3939 1115 -2824 = - Hits 16117 1755-14362 + Misses18528 8979 -9549 + Partials 1746 159 -1587 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=474) * f14ffb1f08820146e5d26616aa9b956ff99ec604 UNKNOWN * 131fd8fbe3f7e87e6b9939325e446564b230f7c6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=478) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
hudi-bot edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867438049 ## CI report: * a373b98620420ec6e728f422c42b2a74d11b3d5c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=475) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
codecov-commenter commented on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869327388 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3167](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (47941e9) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `43.18%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3167/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3167 +/- ## - Coverage 46.26% 3.07% -43.19% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39824 10853-28971 Branches 42891107 -3182 - Hits 18425 334-18091 + Misses19518 10493 -9025 + Partials 1881 26 -1855 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.74%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3167?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3167/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (1c05b7f) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `17.86%`. > The diff coverage is `0.00%`. > :exclamation: Current head 1c05b7f differs from pull request most recent head a373b98. Consider uploading reports for the commit a373b98 to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3146/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## = - Coverage 46.24% 28.37% -17.87% + Complexity 5362 1254 -4108 = Files 920 375 -545 Lines 3984214310-25532 Branches 4291 1460 -2831 = - Hits 18425 4061-14364 + Misses19535 9952 -9583 + Partials 1882 297 -1585 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.07%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (131fd8f) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `41.22%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3120 +/- ## - Coverage 44.28% 3.06% -41.23% + Complexity 4573 82 -4491 Files 824 279 -545 Lines 36391 10893-25498 Branches 39391115 -2824 - Hits 16117 334-15783 + Misses18528 10533 -7995 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.73%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
hudi-bot edited a comment on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869325847 ## CI report: * 47941e913528dbacd05f222c0d98f6ab2a34db31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=477) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
hudi-bot edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865079044 ## CI report: * cd545b199fbe8c68ebd0b0be5ce91722a76c349a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=473) * 548390e6bd363be3ec299be8f2a2a3703f36566a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=476) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
hudi-bot commented on pull request #3167: URL: https://github.com/apache/hudi/pull/3167#issuecomment-869325847 ## CI report: * 47941e913528dbacd05f222c0d98f6ab2a34db31 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=474) * f14ffb1f08820146e5d26616aa9b956ff99ec604 UNKNOWN * 131fd8fbe3f7e87e6b9939325e446564b230f7c6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu opened a new pull request #3167: [MINOR] Drop duplicate keygenerator class configuration setting
wangxianghu opened a new pull request #3167: URL: https://github.com/apache/hudi/pull/3167 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
codecov-commenter edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865177596 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
codecov-commenter edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865177596 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3126](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6741d83) into [master](https://codecov.io/gh/apache/hudi/commit/429e9fb5fe78e96e565dd6b2ea0f1f9e56a43c6c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (429e9fb) will **decrease** coverage by `17.62%`. > The diff coverage is `n/a`. > :exclamation: Current head 6741d83 differs from pull request most recent head 548390e. Consider uploading reports for the commit 548390e to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3126/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3126 +/- ## = - Coverage 46.02% 28.39% -17.63% + Complexity 5307 1255 -4052 = Files 911 375 -536 Lines 3947614304-25172 Branches 4254 1459 -2795 = - Hits 18167 4062-14105 + Misses19456 9946 -9510 + Partials 1853 296 -1557 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.15%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (-0.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] nsivabalan commented on a change in pull request #3147: [HUDI-1983] [DOC]Add key gen type config to Configurations page in hudi we…
nsivabalan commented on a change in pull request #3147: URL: https://github.com/apache/hudi/pull/3147#discussion_r659453008 ## File path: docs/_docs/2_4_configurations.md ## @@ -83,8 +83,12 @@ Actual value ontained by invoking .toString() KEYGENERATOR_CLASS_OPT_KEY {#KEYGENERATOR_CLASS_OPT_KEY} Property: `hoodie.datasource.write.keygenerator.class`, Default: `org.apache.hudi.keygen.SimpleKeyGenerator` - Key generator class, that implements will extract the key out of incoming `Row` object - + Key generator class, that implements will extract the key out of incoming `Row` object. This config option is used for user-defined KeyGenerator Review comment: Can you also add a line there, that this has higher precedence over keygen type. ## File path: docs/_docs/2_4_configurations.md ## @@ -83,8 +83,12 @@ Actual value ontained by invoking .toString() KEYGENERATOR_CLASS_OPT_KEY {#KEYGENERATOR_CLASS_OPT_KEY} Property: `hoodie.datasource.write.keygenerator.class`, Default: `org.apache.hudi.keygen.SimpleKeyGenerator` - Key generator class, that implements will extract the key out of incoming `Row` object - + Key generator class, that implements will extract the key out of incoming `Row` object. This config option is used for user-defined KeyGenerator Review comment: also, there is no more default for hoodie.datasource.write.keygenerator.class if I understand correctly. can we fix that as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f14ffb1) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `41.22%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3120 +/- ## - Coverage 44.28% 3.06% -41.23% + Complexity 4573 82 -4491 Files 824 279 -545 Lines 36391 10893-25498 Branches 39391115 -2824 - Hits 16117 334-15783 + Misses18528 10533 -7995 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.73%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (1c05b7f) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `30.09%`. > The diff coverage is `0.00%`. > :exclamation: Current head 1c05b7f differs from pull request most recent head a373b98. Consider uploading reports for the commit a373b98 to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3146/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## = - Coverage 46.24% 16.15% -30.10% + Complexity 5362 466 -4896 = Files 920 279 -641 Lines 3984210859-28983 Branches 4291 1108 -3183 = - Hits 18425 1754-16671 + Misses19535 8945-10590 + Partials 1882 160 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.07%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=474) * f14ffb1f08820146e5d26616aa9b956ff99ec604 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=474) * f14ffb1f08820146e5d26616aa9b956ff99ec604 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
nsivabalan commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r659451616 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java ## @@ -80,22 +84,38 @@ private final String baseFilePath; public ExternalSpillableMap(Long maxInMemorySizeInBytes, String baseFilePath, SizeEstimator keySizeEstimator, - SizeEstimator valueSizeEstimator) throws IOException { + SizeEstimator valueSizeEstimator) throws IOException { +this(maxInMemorySizeInBytes, baseFilePath, keySizeEstimator, +valueSizeEstimator, DiskMapType.DISK_MAP); + } + + public ExternalSpillableMap(Long maxInMemorySizeInBytes, String baseFilePath, SizeEstimator keySizeEstimator, + SizeEstimator valueSizeEstimator, DiskMapType diskMapType) throws IOException { this.inMemoryMap = new HashMap<>(); this.baseFilePath = baseFilePath; -this.diskBasedMap = new DiskBasedMap<>(baseFilePath); this.maxInMemorySizeInBytes = (long) Math.floor(maxInMemorySizeInBytes * sizingFactorForInMemoryMap); this.currentInMemoryMapSize = 0L; this.keySizeEstimator = keySizeEstimator; this.valueSizeEstimator = valueSizeEstimator; +this.diskMapType = diskMapType; } - private DiskBasedMap getDiskBasedMap() { + private SpillableDiskMap getDiskBasedMap() { if (null == diskBasedMap) { synchronized (this) { if (null == diskBasedMap) { try { -diskBasedMap = new DiskBasedMap<>(baseFilePath); +switch (diskMapType) { + case ROCK_DB: +diskBasedMap = new SpillableRocksDBBasedMap<>(baseFilePath); +break; + case COMPRESSED_DISK_MAP: Review comment: compression is just a config right. is there a necessity to introduce a new enum? ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { Review comment: I assume these are not required to be thread safe ? can you confirm this. ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { Review comment: are these required to be public ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f1ea8f1) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `28.18%`. > The diff coverage is `0.00%`. > :exclamation: Current head f1ea8f1 differs from pull request most recent head f14ffb1. Consider uploading reports for the commit f14ffb1 to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## = - Coverage 44.28% 16.10% -28.19% + Complexity 4573 466 -4107 = Files 824 279 -545 Lines 3639110893-25498 Branches 3939 1115 -2824 = - Hits 16117 1754-14363 + Misses18528 8979 -9549 + Partials 1746 160 -1586 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.23% <ø> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
codecov-commenter edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865177596 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3126](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6741d83) into [master](https://codecov.io/gh/apache/hudi/commit/429e9fb5fe78e96e565dd6b2ea0f1f9e56a43c6c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (429e9fb) will **decrease** coverage by `29.84%`. > The diff coverage is `n/a`. > :exclamation: Current head 6741d83 differs from pull request most recent head 548390e. Consider uploading reports for the commit 548390e to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3126/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3126 +/- ## = - Coverage 46.02% 16.17% -29.85% + Complexity 5307 467 -4840 = Files 911 279 -632 Lines 3947610853-28623 Branches 4254 1107 -3147 = - Hits 18167 1755-16412 + Misses19456 8939-10517 + Partials 1853 159 -1694 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (-0.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (1c05b7f) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `43.16%`. > The diff coverage is `0.00%`. > :exclamation: Current head 1c05b7f differs from pull request most recent head a373b98. Consider uploading reports for the commit a373b98 to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3146/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3146 +/- ## - Coverage 46.24% 3.07% -43.17% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39842 10859-28983 Branches 42911108 -3183 - Hits 18425 334-18091 + Misses19535 10499 -9036 + Partials 1882 26 -1856 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.76%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
hudi-bot edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867438049 ## CI report: * 9f29a8afc1a6e049984a789001042921d917f180 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=431) * a373b98620420ec6e728f422c42b2a74d11b3d5c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=475) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
hudi-bot edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867438049 ## CI report: * 9f29a8afc1a6e049984a789001042921d917f180 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=431) * a373b98620420ec6e728f422c42b2a74d11b3d5c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
pengzhiwei2018 commented on a change in pull request #3146: URL: https://github.com/apache/hudi/pull/3146#discussion_r659436687 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala ## @@ -355,6 +355,8 @@ object DataSourceWriteOptions { val HIVE_TABLE_PROPERTIES = "hoodie.datasource.hive_sync.table_properties" val HIVE_TABLE_SERDE_PROPERTIES = "hoodie.datasource.hive_sync.serde_properties" val HIVE_SYNC_AS_DATA_SOURCE_TABLE = "hoodie.datasource.hive_sync.sync_as_datasource" + val HIVE_CREATE_MANAGED_TABLE = "hoodie.datasource.hive_sync_create_managed_table" Review comment: done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f1ea8f1) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `41.22%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3120 +/- ## - Coverage 44.28% 3.06% -41.23% + Complexity 4573 82 -4491 Files 824 279 -545 Lines 36391 10893-25498 Branches 39391115 -2824 - Hits 16117 334-15783 + Misses18528 10533 -7995 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.73%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=474) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3140: [HUDI-2063] Add Doc For Spark Sql Integrates With Hudi
pengzhiwei2018 commented on a change in pull request #3140: URL: https://github.com/apache/hudi/pull/3140#discussion_r659434155 ## File path: docs/_docs/1_1_spark_quick_start_guide.md ## @@ -300,6 +300,221 @@ spark. show(100, false) ``` +# Spark-Sql example +## Setup +Hudi support using spark sql to write and read data with the **HoodieSparkSessionExtension** sql extension. +```shell +# spark sql for spark 3 +spark-sql --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.1 \ +--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ +--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' + +# spark-sql for spark 2 with scala 2.11 +spark-sql --packages org.apache.hudi:hudi-spark-bundle_2.11:0.8.0,org.apache.spark:spark-avro_2.11:2.4.4 \ +--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ +--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' + +# spark-sql for spark 2 with scala 2.12 +spark-sql \ + --packages org.apache.hudi:hudi-spark-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:2.4.4 \ + --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ + --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' +``` + +## Sql syntax +### DDL +Hudi support create table using the spark-sql. +**Create Non-Partitioned Table** +```sql +-- create a managed cow table +create table if not exists h0( + id int, + name string, + price double +) using hudi +options ( + type = 'cow', + primaryKey = 'id' +); + +-- creae an exteranl mor table +create table if not exists h1( + id int, + name string, + price double, + ts bigint +) using hudi +location '/tmp/hudi/h0' +options ( + type = 'mor', + primaryKey = 'id,name', + preCombineField = 'ts' +) +; + +-- create a non-primary key table +create table if not exists h2( + id int, + name string, + price double +) using hudi +options ( + type = 'cow' +); +``` +**Create Non-Partitioned Table** +```sql +create table if not exists h_p0 ( +id bigint, +name string, +dt string, +hh string +) using hudi +location '/tmp/hudi/h_p0' +options ( + type = 'cow', + primaryKey = 'id', + preCombineField = 'ts' + ) +partitioned by (dt, hh) +; +``` +**Create Table Options** + +| Parameter Name | Introduction | +||| +| primaryKey | The primary key names of the table, multiple fields separated by commas. | +| type | The table type to create. type = 'cow' means a COPY-ON-WRITE table,while type = 'mor' means a MERGE-ON-READ table. Default value is 'cow' without specified this option.| +| preCombineField | The Pre-Combine field of the table. | + +## DML +### MergeInto +Hudi support merge-into for both spark 2 & spark 3. +**Syntax** +```sql +MERGE INTO tableIdentifier AS target_alias +USING (sub_query | tableIdentifier) AS source_alias +ON +[ WHEN MATCHED [ AND ] THEN ] +[ WHEN MATCHED [ AND ] THEN ] +[ WHEN NOT MATCHED [ AND ] THEN ] + + =A equal bool condition + = + DELETE | + UPDATE SET * | + UPDATE SET column1 = expression1 [, column2 = expression2 ...] + = + INSERT * | + INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...]) +``` +**Case** +```sql +merge into h0 as target +using ( + select id, name, price, flag from s +) source +on target.id = source.id +when matched then update set * +when not matched then insert * +; + +merge into h0 +using ( + select id, name, price, flag from s +) source +on h0.id = source.id +when matched and flag != 'delete' then update set id = source.id, name = source.name, price = source.price * 2 +when matched and flag = 'delete' then delete +when not matched then insert (id,name,price) values(id, name, price) +; +``` + +### Insert +```sql +insert into h0 select 1, 'a1', 20; + +-- insert static partition +insert into h_p0 partition(dt = '2021-01-02') select 1, 'a1'; + +-- insert dynamic partition +insert into h_p0 select 1, 'a1', dt; + +-- insert dynamic partition +insert into h_p1 select 1 as id, 'a1', '2021-01-03' as dt, '19' as hh; + +-- insert overwrite table +insert overwrite table h0 select 1, 'a1', 20; +``` + +### Update +**Syntax** +```sql + UPDATE tableIdentifier SET column = EXPRESSION(,column = EXPRESSION) [ WHERE boolExpression] +``` +**Case** +```sql + update h0 set price = price + 20 where id = 1; + update h0 set price = price *2, name = 'a2' where id = 2; +``` + +### Delete +**Syntax** +```sql + DELETE FROM tableIdentifier [ WHERE BOOL_EXPRESSION] +``` +**Case** +```sql +delete from h0 where id = 1; +``` +### AlterTable +**Syntx** +```sql +-- Alter table name +ALTER TABLE oldTableName RENAME TO newTableName + +-- Alter table add columns +ALTER TABLE tableIdentifier ADD COLUMNS(colAndType (,colAndType)*) + +-- Alter table column type +ALTER TABLE tableIdentifier CHANGE COLUMN colName colName colType +``` +**Case** +```sql +alter table h0 rename to h0_1; +
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN * f1ea8f1d647b6d30e3c8582f1b0b052e25764388 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
hudi-bot edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865079044 ## CI report: * cd545b199fbe8c68ebd0b0be5ce91722a76c349a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=473) * 548390e6bd363be3ec299be8f2a2a3703f36566a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
codecov-commenter edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865177596 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3126](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6741d83) into [master](https://codecov.io/gh/apache/hudi/commit/429e9fb5fe78e96e565dd6b2ea0f1f9e56a43c6c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (429e9fb) will **decrease** coverage by `42.94%`. > The diff coverage is `n/a`. > :exclamation: Current head 6741d83 differs from pull request most recent head 548390e. Consider uploading reports for the commit 548390e to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3126/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3126 +/- ## - Coverage 46.02% 3.07% -42.95% + Complexity 5307 82 -5225 Files 911 279 -632 Lines 39476 10853-28623 Branches 42541107 -3147 - Hits 18167 334-17833 + Misses19456 10493 -8963 + Partials 1853 26 -1827 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.88%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3126?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3126/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
hudi-bot edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865079044 ## CI report: * 6d70a1dd2a20de7b08d859bc9f51a19e35891c93 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=335) * cd545b199fbe8c68ebd0b0be5ce91722a76c349a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=473) * 548390e6bd363be3ec299be8f2a2a3703f36566a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
hudi-bot edited a comment on pull request #3126: URL: https://github.com/apache/hudi/pull/3126#issuecomment-865079044 ## CI report: * 6d70a1dd2a20de7b08d859bc9f51a19e35891c93 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=335) * cd545b199fbe8c68ebd0b0be5ce91722a76c349a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3126: [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql
pengzhiwei2018 commented on a change in pull request #3126: URL: https://github.com/apache/hudi/pull/3126#discussion_r659425547 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala ## @@ -426,7 +426,8 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Runnab throw new IllegalArgumentException(s"Merge Key[${targetKey2SourceExpression.keySet.mkString(",")}] is not" + s" Equal to the defined primary key[${definedPk.mkString(",")}] in table $targetTableName") } - +// Enable the hive sync by default if spark have enable the hive metastore. +val enableHive = "hive" == sparkSession.sessionState.conf.getConf(StaticSQLConf.CATALOG_IMPLEMENTATION) Review comment: done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (504a677) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `41.22%`. > The diff coverage is `0.00%`. > :exclamation: Current head 504a677 differs from pull request most recent head 75aadbc. Consider uploading reports for the commit 75aadbc to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3120 +/- ## - Coverage 44.28% 3.06% -41.23% + Complexity 4573 82 -4491 Files 824 279 -545 Lines 36391 10893-25498 Branches 39391115 -2824 - Hits 16117 334-15783 + Misses18528 10533 -7995 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.73%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] nsivabalan commented on a change in pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
nsivabalan commented on a change in pull request #3117: URL: https://github.com/apache/hudi/pull/3117#discussion_r659419442 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/SpillableRocksDBBasedMap.java ## @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.common.util.collection; + +import org.apache.hudi.exception.HoodieException; +import org.apache.hudi.exception.HoodieNotSupportedException; + +import org.apache.log4j.LogManager; +import org.apache.log4j.Logger; +import org.jetbrains.annotations.NotNull; + +import java.io.IOException; +import java.io.Serializable; +import java.util.AbstractMap; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.Map; +import java.util.Set; +import java.util.stream.Stream; + +/** + * This class provides a disk spillable only map implementation that is based on RocksDB. + */ +public final class SpillableRocksDBBasedMap implements SpillableDiskMap { + // ColumnFamily allows partitioning data within RockDB, which allows + // independent configuration and faster deletes across partitions + // https://github.com/facebook/rocksdb/wiki/Column-Families + // For this use case, we use a single static column family/ partition + // + private static final String COLUMN_FAMILY_NAME = "spill_map"; + + private static final Logger LOG = LogManager.getLogger(SpillableRocksDBBasedMap.class); + // Stores the key and corresponding value's latest metadata spilled to disk + private final Set keySet; + private final String rocksDbStoragePath; + private RocksDBDAO rocksDb; + + public SpillableRocksDBBasedMap(String rocksDbStoragePath) throws IOException { +this.keySet = new HashSet<>(); +this.rocksDbStoragePath = rocksDbStoragePath; + } + + @Override + public int size() { +return keySet.size(); + } + + @Override + public boolean isEmpty() { +return keySet.isEmpty(); + } + + @Override + public boolean containsKey(Object key) { +return keySet.contains((T) key); + } + + @Override + public boolean containsValue(Object value) { +throw new HoodieNotSupportedException("unable to compare values in map"); + } + + @Override + public R get(Object key) { +if (!containsKey(key)) { + return null; +} +return getRocksDb().get(COLUMN_FAMILY_NAME, (T) key); + } + + @Override + public R put(T key, R value) { +getRocksDb().put(COLUMN_FAMILY_NAME, key, value); +keySet.add(key); +return value; + } + + @Override + public R remove(Object key) { +R value = get(key); +if (value != null) { + keySet.remove((T) key); + getRocksDb().delete(COLUMN_FAMILY_NAME, (T) key); +} +return value; + } + + @Override + public void putAll(Map keyValues) { +getRocksDb().writeBatch(batch -> keyValues.forEach((key, value) -> getRocksDb().putInBatch(batch, COLUMN_FAMILY_NAME, key, value))); +keySet.addAll(keyValues.keySet()); + } + + @Override + public void clear() { +close(); + } + + @Override + public @NotNull Set keySet() { +return keySet; + } + + @Override + public @NotNull Collection values() { +throw new HoodieException("Unsupported Operation Exception"); + } + + @Override + public @NotNull Set> entrySet() { +Set> entrySet = new HashSet<>(); +for (T key : keySet) { + entrySet.add(new AbstractMap.SimpleEntry<>(key, get(key))); +} +return entrySet; + } + + /** + * Custom iterator to iterate over values written to disk. + */ + @Override + public @NotNull Iterator iterator() { +return getRocksDb().iterator(COLUMN_FAMILY_NAME); + } + + @Override + public Stream valueStream() { +return keySet.stream().sorted().sequential().map(valueMetaData -> (R) get(valueMetaData)); + } + + @Override + public long sizeOfFileOnDiskInBytes() { +return getRocksDb().getTotalBytesWritten(); + } + + @Override + public void close() { +keySet.clear(); +if (null != rocksDb) { + rocksDb.close(); +} +rocksDb = null; + } + + private RocksDBDAO getRocksDb() { +if (null == rocksDb) { Review comment: do we need to think about thread
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (02acd11) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `41.22%`. > The diff coverage is `0.00%`. > :exclamation: Current head 02acd11 differs from pull request most recent head 75aadbc. Consider uploading reports for the commit 75aadbc to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3120/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3120 +/- ## - Coverage 44.28% 3.06% -41.23% + Complexity 4573 82 -4491 Files 824 279 -545 Lines 36391 10893-25498 Branches 39391115 -2824 - Hits 16117 334-15783 + Misses18528 10533 -7995 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.73%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN * 75aadbc834d6606527764468dd3dbcb1e802b171 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN * 504a6770be5d4cd3a78d61129be5b1aaadd515df UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) * 02acd1127b72470f6d7adffb787179f0cddfa954 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
pengzhiwei2018 commented on a change in pull request #3154: URL: https://github.com/apache/hudi/pull/3154#discussion_r659420453 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala ## @@ -164,7 +165,47 @@ case class HoodieResolveReferences(sparkSession: SparkSession) extends Rule[Logi case UpdateAction(condition, assignments) => val (resolvedCondition, resolvedAssignments) = resolveConditionAssignments(condition, assignments) - UpdateAction(resolvedCondition, resolvedAssignments) + + // Get the target table type and pre-combine field. + val targetTableId = getMergeIntoTargetTableId(mergeInto) + val targetTable = +sparkSession.sessionState.catalog.getTableMetadata(targetTableId) + val targetTableType = HoodieOptionConfig.getTableType(targetTable.storage.properties) + val preCombineField = HoodieOptionConfig.getPreCombineField(targetTable.storage.properties) + + // Get the map of target attribute to value of the update assignments. + val target2Values = resolvedAssignments.map { + case Assignment(attr: AttributeReference, value) => +attr.name -> value + case o => throw new IllegalArgumentException(s"Assignment key must be an attribute, current is: ${o.key}") + }.toMap + + // Validate if there are incorrect target attributes. + val unKnowTargets = target2Values.keys +.filterNot(removeMetaFields(target.output).map(_.name).contains(_)) + if (unKnowTargets.nonEmpty) { +throw new AnalysisException(s"Cannot find target attributes: ${unKnowTargets.mkString(",")}.") + } + + // Fill the missing target attribute in the update action for COW table to support partial update. + // e.g. If the update action missing 'id' attribute, we fill a "id = target.id" to the update action. + val newAssignments = removeMetaFields(target.output) +.map(attr => { + // TODO support partial update for MOR. Review comment: No, we can support partial update for cow table by the `ExpressionPayload` added in the sql support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1211) Test failures w/ some index tests (TestHoodieIndex)
[ https://issues.apache.org/jira/browse/HUDI-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370374#comment-17370374 ] sivabalan narayanan commented on HUDI-1211: --- don't think this is an issue right now. we can close this out. > Test failures w/ some index tests (TestHoodieIndex) > --- > > Key: HUDI-1211 > URL: https://issues.apache.org/jira/browse/HUDI-1211 > Project: Apache Hudi > Issue Type: Bug > Components: Testing >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available, sev:high > > When I was running tests for all of index package, tests fails with > connection issue following which all tests start to fail. Once the connection > issue starts, I see below errors for successive tests. > > Command : "mvn '-Dtest=org.apache.hudi.index.**' -DfailIfNoTests=false test" > <<< ERROR! > org.apache.spark.SparkException: > Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore > this error, set spark.driver.allowMultipleContexts = true. The currently > running SparkContext was created at: > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) > org.apache.hudi.testutils.FunctionalTestHarness.runBeforeEach(FunctionalTestHarness.java:132) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > Here is the full stack trace: > [https://gist.github.com/nsivabalan/0643202bf1af2d85cdc0d35dd6a68d36] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1211) Test failures w/ some index tests (TestHoodieIndex)
[ https://issues.apache.org/jira/browse/HUDI-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1211: -- Status: In Progress (was: Open) > Test failures w/ some index tests (TestHoodieIndex) > --- > > Key: HUDI-1211 > URL: https://issues.apache.org/jira/browse/HUDI-1211 > Project: Apache Hudi > Issue Type: Bug > Components: Testing >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available, sev:high > > When I was running tests for all of index package, tests fails with > connection issue following which all tests start to fail. Once the connection > issue starts, I see below errors for successive tests. > > Command : "mvn '-Dtest=org.apache.hudi.index.**' -DfailIfNoTests=false test" > <<< ERROR! > org.apache.spark.SparkException: > Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore > this error, set spark.driver.allowMultipleContexts = true. The currently > running SparkContext was created at: > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) > org.apache.hudi.testutils.FunctionalTestHarness.runBeforeEach(FunctionalTestHarness.java:132) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > Here is the full stack trace: > [https://gist.github.com/nsivabalan/0643202bf1af2d85cdc0d35dd6a68d36] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1211) Test failures w/ some index tests (TestHoodieIndex)
[ https://issues.apache.org/jira/browse/HUDI-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-1211. - Resolution: Invalid > Test failures w/ some index tests (TestHoodieIndex) > --- > > Key: HUDI-1211 > URL: https://issues.apache.org/jira/browse/HUDI-1211 > Project: Apache Hudi > Issue Type: Bug > Components: Testing >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available, sev:high > > When I was running tests for all of index package, tests fails with > connection issue following which all tests start to fail. Once the connection > issue starts, I see below errors for successive tests. > > Command : "mvn '-Dtest=org.apache.hudi.index.**' -DfailIfNoTests=false test" > <<< ERROR! > org.apache.spark.SparkException: > Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore > this error, set spark.driver.allowMultipleContexts = true. The currently > running SparkContext was created at: > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) > org.apache.hudi.testutils.FunctionalTestHarness.runBeforeEach(FunctionalTestHarness.java:132) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > Here is the full stack trace: > [https://gist.github.com/nsivabalan/0643202bf1af2d85cdc0d35dd6a68d36] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3071: [WIP] [HUDI-1976] Resolve vulnerability
codecov-commenter edited a comment on pull request #3071: URL: https://github.com/apache/hudi/pull/3071#issuecomment-869203433 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3071: [WIP] [HUDI-1976] Resolve vulnerability
hudi-bot edited a comment on pull request #3071: URL: https://github.com/apache/hudi/pull/3071#issuecomment-869222972 ## CI report: * 9640a41dff18eb2387acd7565015fb97db98f476 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=472) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #3003: [HUDI-1939] Replace joda-time api with java8 new time api
xushiyan commented on pull request #3003: URL: https://github.com/apache/hudi/pull/3003#issuecomment-869225643 - [ ] fix UT failures -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
hudi-bot edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869219682 ## CI report: * 85a7049f582bc1ccbb70e2ce97cd72a3c96758b5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=471) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
codecov-commenter edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869220231 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3166](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (85a7049) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3166/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3166 +/- ## = Coverage 46.26% 46.26% Complexity 5362 5362 = Files 920 920 Lines 3982439824 Branches 4289 4289 = Hits 1842518425 Misses1951819518 Partials 1881 1881 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (ø)` | | | hudiflink | `61.50% <ø> (ø)` | | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.06% <ø> (ø)` | | | hudisync | `54.05% <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [e99a6b0...85a7049](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
codecov-commenter edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869220231 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3166](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (85a7049) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `1.58%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3166/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3166 +/- ## - Coverage 46.26% 44.68% -1.59% + Complexity 5362 5117 -245 Files 920 910 -10 Lines 3982438996 -828 Branches 4289 4236 -53 - Hits 1842517425-1000 - Misses1951819783 +265 + Partials 1881 1788 -93 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (ø)` | | | hudiflink | `61.50% <ø> (ø)` | | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.06% <ø> (ø)` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...he/hudi/hive/replication/GlobalHiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvcmVwbGljYXRpb24vR2xvYmFsSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-94.74%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
codecov-commenter edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869220231 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3166](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (85a7049) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `17.86%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3166/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3166 +/- ## = - Coverage 46.26% 28.39% -17.87% + Complexity 5362 1255 -4107 = Files 920 375 -545 Lines 3982414304-25520 Branches 4289 1459 -2830 = - Hits 18425 4062-14363 + Misses19518 9946 -9572 + Partials 1881 296 -1585 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3071: [WIP] [HUDI-1976] Resolve vulnerability
hudi-bot edited a comment on pull request #3071: URL: https://github.com/apache/hudi/pull/3071#issuecomment-869222972 ## CI report: * 9640a41dff18eb2387acd7565015fb97db98f476 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=472) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3071: [WIP] [HUDI-1976] Resolve vulnerability
hudi-bot commented on pull request #3071: URL: https://github.com/apache/hudi/pull/3071#issuecomment-869222972 ## CI report: * 9640a41dff18eb2387acd7565015fb97db98f476 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
codecov-commenter edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869220231 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3166](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (85a7049) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `30.09%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3166/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3166 +/- ## = - Coverage 46.26% 16.17% -30.10% + Complexity 5362 467 -4895 = Files 920 279 -641 Lines 3982410853-28971 Branches 4289 1107 -3182 = - Hits 18425 1755-16670 + Misses19518 8939-10579 + Partials 1881 159 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter commented on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
codecov-commenter commented on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869220231 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3166](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (85a7049) into [master](https://codecov.io/gh/apache/hudi/commit/e99a6b031bf4f2e3037d4cb5307d443cda2d2002?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e99a6b0) will **decrease** coverage by `43.18%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3166/graphs/tree.svg?width=650=150=pr=VTTXabwbs2_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3166 +/- ## - Coverage 46.26% 3.07% -43.19% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39824 10853-28971 Branches 42891107 -3182 - Hits 18425 334-18091 + Misses19518 10493 -9025 + Partials 1881 26 -1855 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.53% <ø> (-48.74%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3166?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3166/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
hudi-bot edited a comment on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869219682 ## CI report: * 85a7049f582bc1ccbb70e2ce97cd72a3c96758b5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=471) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
hudi-bot commented on pull request #3166: URL: https://github.com/apache/hudi/pull/3166#issuecomment-869219682 ## CI report: * 85a7049f582bc1ccbb70e2ce97cd72a3c96758b5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2081) Move TestHiveSyncTool to functional
[ https://issues.apache.org/jira/browse/HUDI-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2081: - Labels: pull-request-available (was: ) > Move TestHiveSyncTool to functional > --- > > Key: HUDI-2081 > URL: https://issues.apache.org/jira/browse/HUDI-2081 > Project: Apache Hudi > Issue Type: Sub-task > Components: Testing >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available > > Make it similar to TestHiveMetastoreBasedLockProvider, ran by > HiveSyncFunctionalTestSuite > Also clean up HiveTestUtil. > This is a follow-up task to https://github.com/apache/hudi/pull/3043 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] xushiyan opened a new pull request #3166: [HUDI-2081] Move schema util tests out from TestHiveSyncTool
xushiyan opened a new pull request #3166: URL: https://github.com/apache/hudi/pull/3166 ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2081) Move TestHiveSyncTool to functional
[ https://issues.apache.org/jira/browse/HUDI-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2081: - Status: In Progress (was: Open) > Move TestHiveSyncTool to functional > --- > > Key: HUDI-2081 > URL: https://issues.apache.org/jira/browse/HUDI-2081 > Project: Apache Hudi > Issue Type: Sub-task > Components: Testing >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Major > > Make it similar to TestHiveMetastoreBasedLockProvider, ran by > HiveSyncFunctionalTestSuite > Also clean up HiveTestUtil. > This is a follow-up task to https://github.com/apache/hudi/pull/3043 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] vinothchandar commented on pull request #3003: [HUDI-1939][WIP] Replace joda-time api with java8 new time api
vinothchandar commented on pull request #3003: URL: https://github.com/apache/hudi/pull/3003#issuecomment-869217062 Is the WIP in the title still needed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-2083) Hudi CLI does not with S3
Vinay created HUDI-2083: --- Summary: Hudi CLI does not with S3 Key: HUDI-2083 URL: https://issues.apache.org/jira/browse/HUDI-2083 Project: Apache Hudi Issue Type: Task Components: CLI Reporter: Vinay Assignee: Vinay Hudi CLI gives exception when trying to connect to s3 path {code:java} create --path s3://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 --tableType MERGE_ON_READ Failed to get instance of org.apache.hadoop.fs.FileSystem org.apache.hudi.exception.HoodieIOException: Failed to get instance of org.apache.hadoop.fs.FileSystem at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:98) = create --path s3a://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 --tableType MERGE_ON_READ Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) {code} This could be because target/lib folder does not contain hadoop-aws or was-s3 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2083) Hudi CLI does not work with S3
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HUDI-2083: Summary: Hudi CLI does not work with S3 (was: Hudi CLI does not with S3) > Hudi CLI does not work with S3 > -- > > Key: HUDI-2083 > URL: https://issues.apache.org/jira/browse/HUDI-2083 > Project: Apache Hudi > Issue Type: Task > Components: CLI >Reporter: Vinay >Assignee: Vinay >Priority: Major > > Hudi CLI gives exception when trying to connect to s3 path > {code:java} > create --path s3://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Failed to get instance of org.apache.hadoop.fs.FileSystem > org.apache.hudi.exception.HoodieIOException: Failed to get instance of > org.apache.hadoop.fs.FileSystem > at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:98) > = > create --path s3a://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) > {code} > This could be because target/lib folder does not contain hadoop-aws or was-s3 > dependency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution
hudi-bot edited a comment on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-869215376 ## CI report: * f7e53eb22bf7a1e109ba258c7ac21d983d807738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=470) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query
hudi-bot edited a comment on pull request #2378: URL: https://github.com/apache/hudi/pull/2378#issuecomment-869215174 ## CI report: * 295290ebdd0a01a8e403e9799a7bfdb75364b435 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=469) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query
hudi-bot edited a comment on pull request #2378: URL: https://github.com/apache/hudi/pull/2378#issuecomment-869215174 ## CI report: * 295290ebdd0a01a8e403e9799a7bfdb75364b435 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=469) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution
hudi-bot commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-869215376 ## CI report: * f7e53eb22bf7a1e109ba258c7ac21d983d807738 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution
vinothchandar commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-869215210 @nsivabalan whats the next step for this pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query
hudi-bot commented on pull request #2378: URL: https://github.com/apache/hudi/pull/2378#issuecomment-869215174 ## CI report: * 295290ebdd0a01a8e403e9799a7bfdb75364b435 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HUDI-1211) Test failures w/ some index tests (TestHoodieIndex)
[ https://issues.apache.org/jira/browse/HUDI-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370315#comment-17370315 ] Raymond Xu edited comment on HUDI-1211 at 6/27/21, 7:40 PM: [~shivnarayan] is this still an issue? I couldn't repro this in my local {code:java} ➜ mvn -Punit-tests -pl hudi-common,hudi-client/hudi-spark-client test ➜ mvn -Punit-tests -pl hudi-client/hudi-spark-client '-Dtest=org.apache.hudi.index.**' -DfailIfNoTests=false test{code} these 2 worked for me. was (Author: xushiyan): [~shivnarayan] is this still an issue? I couldn't repro this in my local > Test failures w/ some index tests (TestHoodieIndex) > --- > > Key: HUDI-1211 > URL: https://issues.apache.org/jira/browse/HUDI-1211 > Project: Apache Hudi > Issue Type: Bug > Components: Testing >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available, sev:high > > When I was running tests for all of index package, tests fails with > connection issue following which all tests start to fail. Once the connection > issue starts, I see below errors for successive tests. > > Command : "mvn '-Dtest=org.apache.hudi.index.**' -DfailIfNoTests=false test" > <<< ERROR! > org.apache.spark.SparkException: > Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore > this error, set spark.driver.allowMultipleContexts = true. The currently > running SparkContext was created at: > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) > org.apache.hudi.testutils.FunctionalTestHarness.runBeforeEach(FunctionalTestHarness.java:132) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > Here is the full stack trace: > [https://gist.github.com/nsivabalan/0643202bf1af2d85cdc0d35dd6a68d36] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
n3nash commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-869213421 @guanziyue Thanks for the detailed explanation. @tandonraghav During compaction, `combineAndGetUpdateValue` is never called. See this to get an understanding of how compaction works -> https://github.com/apache/hudi/blob/e99a6b031bf4f2e3037d4cb5307d443cda2d2002/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java#L132. The issue is you want a common implementation across both `preCombine` and `combinAndGetUpdateValue`. The way I would recommend doing this is as follows : class YourPayloadImplementation { Map allTheCommonValuesNeededToDetermineHowToMerge2Records `private Boolean commonMergeLogic(allTheCommonValuesNeededToDetermineHowToMerge2Records old, allTheCommonValuesNeededToDetermineHowToMerge2Records new) { }` `preCombine(YourPayloadImplementation that) Use allTheCommonValuesNeededToDetermineHowToMerge2Records to merge 2 different records boolean pickNew commonMergeLogic(this.allTheCommonValuesNeededToDetermineHowToMerge2Records, that.allTheCommonValuesNeededToDetermineHowToMerge2Records); if (pickNew) { .. } .. }` `combinAndGetUpdateValue(GenericRecord old, Schema schema) { allTheCommonValuesNeededToDetermineHowToMerge2RecordsOld = extractValuesfromOld(old); allTheCommonValuesNeededToDetermineHowToMerge2RecordsNew = extractValuesfromNew(getInsertValue(schema)); boolean pickNew = commonMergeLogic(allTheCommonValuesNeededToDetermineHowToMerge2RecordsOld, allTheCommonValuesNeededToDetermineHowToMerge2RecordsNew); if (pickNew) { .. } .. }` Another way is to pass the Schema through the properties file and convert everything to GenericRecord and then there is no need for allTheCommonValuesNeededToDetermineHowToMerge2Records. You method would look as follows : class YourPayloadImplementation { `private Boolean commonMergeLogic(GenericRecord old, GenericRecord new) { }` `preCombine(YourPayloadImplementation that, Properties p) Schema schema = SchemaParse.newSchema(p.getString("schema")) boolean pickNew commonMergeLogic(getInsertValue(schema), that.getData().getInsertValue(schema); if (pickNew) { .. } .. }` `combinAndGetUpdateValue(GenericRecord old, Schema schema) { boolean pickNew = commonMergeLogic(old, getInsertValue(schema)); if (pickNew) { .. } .. }` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-869211898 @guanziyue Well, in my testing combinAndGetUpdateValue in not getting called at all. I am not worried about how compaction is happening for delta records. Do you have a reference implementation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org