[GitHub] [hudi] hudi-bot commented on pull request #5080: [HUDI-3670] fix temp view leak in deltastreamer's Sql transformers
hudi-bot commented on pull request #5080: URL: https://github.com/apache/hudi/pull/5080#issuecomment-1073505612 ## CI report: * 89adec7344e74889a7d28c1aab92c57d22f9f93f UNKNOWN * c64539708a9b54672d60ffd71145e0e71d300e56 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5080: [HUDI-3670] fix temp view leak in deltastreamer's Sql transformers
hudi-bot removed a comment on pull request #5080: URL: https://github.com/apache/hudi/pull/5080#issuecomment-1073504837 ## CI report: * 89adec7344e74889a7d28c1aab92c57d22f9f93f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5080: [HUDI-3670] fix temp view leak in deltastreamer's Sql transformers
hudi-bot commented on pull request #5080: URL: https://github.com/apache/hudi/pull/5080#issuecomment-1073504837 ## CI report: * 89adec7344e74889a7d28c1aab92c57d22f9f93f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-1073504409 @pratyakshsharma -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-1073504190 The purpose of introducing timestamps: Mainly when users want to consume from a certain location, deltastreamer can only specify checkpoint sites in the past. For example, kafka may have 50+ partitions, and users need to manually configure the checkpoint string. Introducing this simplifies this operation Regarding your example: I think you are right and agree with your idea. Partition 2 should not be populated with this value. At that time, the main consideration of this PR was to solve the problem of complex user configuration. It can simplify consumption data as much as possible. This example of partition 2 makes sense for some businesses. Maybe your current scenario may be a bit contradictory, and I feel like we can improve it and make it better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3670) SqlQueryBasedTransformer leaks temp views in continuous mode
[ https://issues.apache.org/jira/browse/HUDI-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3670: - Labels: pull-request-available (was: ) > SqlQueryBasedTransformer leaks temp views in continuous mode > > > Key: HUDI-3670 > URL: https://issues.apache.org/jira/browse/HUDI-3670 > Project: Apache Hudi > Issue Type: Bug > Components: deltastreamer >Reporter: Ji Qi >Priority: Minor > Labels: pull-request-available > > In the Sql transformers, a new temp view with a random name is created for > each incoming batch, but the temp view is never dropped. > This causes a resource leak in Spark SessionState's Catalog. For long-running > deltastreamer jobs, the temp views created causes HiveSessionCatalog to take > up a lot of memory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] qjqqyy opened a new pull request #5080: [HUDI-3670] fix temp view leak in deltastreamer's Sql transformers
qjqqyy opened a new pull request #5080:
URL: https://github.com/apache/hudi/pull/5080
## What is the purpose of the pull request
Transformers such as `SqlQueryBasedTransformer` creates a temp view with a
random name for every batch of incoming data, but they are never freed.
This causes a memory leak in SparkSession's HiveCatalog.
## Brief change log
- In `Sql{File,Query}BasedTransformer` and `FlatteningTransformer`, drop
the temp view before returning the transformed dataframe
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test
coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot removed a comment on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073501604 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) * f5a13c08cc1d2c59a56f85702c455cd1d409fe79 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot commented on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073502423 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) * f5a13c08cc1d2c59a56f85702c455cd1d409fe79 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7119) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot removed a comment on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073297587 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot commented on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073501604 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) * f5a13c08cc1d2c59a56f85702c455cd1d409fe79 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3671) Fix logic for deltastreamer consuming message based on kafka timestamp
Pratyaksh Sharma created HUDI-3671: -- Summary: Fix logic for deltastreamer consuming message based on kafka timestamp Key: HUDI-3671 URL: https://issues.apache.org/jira/browse/HUDI-3671 Project: Apache Hudi Issue Type: Bug Reporter: Pratyaksh Sharma Assignee: Pratyaksh Sharma https://github.com/apache/hudi/pull/2438#discussion_r830456199 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
pratyakshsharma commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r830768428
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -283,6 +323,41 @@ private Long delayOffsetCalculation(Option
lastCheckpointStr, Set getOffsetsByTimestamp(KafkaConsumer consumer,
List partitionInfoList, Set topicPartitions,
+ String topicName, Long
timestamp) {
+
+Map topicPartitionsTimestamp =
partitionInfoList.stream()
+.map(x -> new
TopicPartition(x.topic(), x.partition()))
+
.collect(Collectors.toMap(Function.identity(), x -> timestamp));
+
+Map earliestOffsets =
consumer.beginningOffsets(topicPartitions);
+Map offsetAndTimestamp =
consumer.offsetsForTimes(topicPartitionsTimestamp);
+
+StringBuilder sb = new StringBuilder();
+sb.append(topicName + ",");
+for (Map.Entry map :
offsetAndTimestamp.entrySet()) {
+ if (map.getValue() != null) {
+
sb.append(map.getKey().partition()).append(":").append(map.getValue().offset()).append(",");
+ } else {
+
sb.append(map.getKey().partition()).append(":").append(earliestOffsets.get(map.getKey())).append(",");
Review comment:
created a jira for this - https://issues.apache.org/jira/browse/HUDI-3671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HUDI-3670) SqlQueryBasedTransformer leaks temp views in continuous mode
Ji Qi created HUDI-3670: --- Summary: SqlQueryBasedTransformer leaks temp views in continuous mode Key: HUDI-3670 URL: https://issues.apache.org/jira/browse/HUDI-3670 Project: Apache Hudi Issue Type: Bug Components: deltastreamer Reporter: Ji Qi In the Sql transformers, a new temp view with a random name is created for each incoming batch, but the temp view is never dropped. This causes a resource leak in Spark SessionState's Catalog. For long-running deltastreamer jobs, the temp views created causes HiveSessionCatalog to take up a lot of memory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
[
https://issues.apache.org/jira/browse/HUDI-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509607#comment-17509607
]
HunterHunter commented on HUDI-3669:
[~x1q1j1] I have confirmed that the firewall is off and there is no network
agent. can you assign this issue to me. thank you
> HoodieRemoteException when task communicates with jm's timeline server: IP :
> port failed to respond
> ---
>
> Key: HUDI-3669
> URL: https://issues.apache.org/jira/browse/HUDI-3669
> Project: Apache Hudi
> Issue Type: Bug
> Components: flink
>Affects Versions: 0.10.1, 0.11.0
> Environment: Flink Standalone 1.13.3
>Reporter: HunterHunter
>Priority: Blocker
> Attachments: WechatIMG14.jpeg
>
>
> Flink streaming write to hudi , The task runs fine about an hour after it
> starts, and the communication is normal,When the task runs for a period of
> time,An error is reported when BucketAssignFunction communicates with JM's
> Timeline server, This error will occur under certain circumstances.
> When the task restarts after an error is reported, it happens again after
> running for a period of time,eventually cause the task to fail
> in addition：I have modified NetworkUtils' method of getting ip as suggested
> {code:java}
> org.apache.hudi.exception.HoodieRemoteException: 10.18x.xx.xx:34805 failed to
> respond
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.refresh(RemoteHoodieTableFileSystemView.java:420)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.sync(RemoteHoodieTableFileSystemView.java:484)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.PriorityBasedFileSystemView.sync(PriorityBasedFileSystemView.java:257)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.profile.WriteProfile.reload(WriteProfile.java:252)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssigner.reload(BucketAssigner.java:211)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssignFunction.notifyCheckpointComplete(BucketAssignFunction.java:234)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
>
[hudi] branch master updated (799c78e -> a118d56)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 799c78e [HUDI-3665] Support flink multiple versions (#5072) add a118d56 [MINOR] Fixing sparkUpdateNode for record generation (#5079) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/integ/testsuite/dag/nodes/SparkInsertNode.scala | 6 +- .../org/apache/hudi/integ/testsuite/dag/nodes/SparkUpsertNode.scala | 5 + 2 files changed, 10 insertions(+), 1 deletion(-)
[GitHub] [hudi] nsivabalan merged pull request #5079: [MINOR] Fixing sparkUpdateNode for record generation
nsivabalan merged pull request #5079: URL: https://github.com/apache/hudi/pull/5079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5079: [MINOR] Fixing sparkUpdateNode for record generation
hudi-bot commented on pull request #5079: URL: https://github.com/apache/hudi/pull/5079#issuecomment-1073478422 ## CI report: * 3abe4abed9cd81efd54fc9b6d19458f3cfdfc4d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7117) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5079: [MINOR] Fixing sparkUpdateNode for record generation
hudi-bot removed a comment on pull request #5079: URL: https://github.com/apache/hudi/pull/5079#issuecomment-1073477257 ## CI report: * 3abe4abed9cd81efd54fc9b6d19458f3cfdfc4d5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5079: [MINOR] Fixing sparkUpdateNode for record generation
hudi-bot commented on pull request #5079: URL: https://github.com/apache/hudi/pull/5079#issuecomment-1073477257 ## CI report: * 3abe4abed9cd81efd54fc9b6d19458f3cfdfc4d5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
[
https://issues.apache.org/jira/browse/HUDI-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509598#comment-17509598
]
Forward Xu commented on HUDI-3669:
--
hi [~HunterHunter] About this problem, check the network to see if there is a
firewall or network agent or the like?
> HoodieRemoteException when task communicates with jm's timeline server: IP :
> port failed to respond
> ---
>
> Key: HUDI-3669
> URL: https://issues.apache.org/jira/browse/HUDI-3669
> Project: Apache Hudi
> Issue Type: Bug
> Components: flink
>Affects Versions: 0.10.1, 0.11.0
> Environment: Flink Standalone 1.13.3
>Reporter: HunterHunter
>Priority: Blocker
> Attachments: WechatIMG14.jpeg
>
>
> Flink streaming write to hudi , The task runs fine about an hour after it
> starts, and the communication is normal,When the task runs for a period of
> time,An error is reported when BucketAssignFunction communicates with JM's
> Timeline server, This error will occur under certain circumstances.
> When the task restarts after an error is reported, it happens again after
> running for a period of time,eventually cause the task to fail
> in addition：I have modified NetworkUtils' method of getting ip as suggested
> {code:java}
> org.apache.hudi.exception.HoodieRemoteException: 10.18x.xx.xx:34805 failed to
> respond
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.refresh(RemoteHoodieTableFileSystemView.java:420)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.sync(RemoteHoodieTableFileSystemView.java:484)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.PriorityBasedFileSystemView.sync(PriorityBasedFileSystemView.java:257)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.profile.WriteProfile.reload(WriteProfile.java:252)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssigner.reload(BucketAssigner.java:211)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssignFunction.notifyCheckpointComplete(BucketAssignFunction.java:234)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
>
[GitHub] [hudi] nsivabalan opened a new pull request #5079: [MINOR] Fixing sparkUpdateNode for record generation
nsivabalan opened a new pull request #5079: URL: https://github.com/apache/hudi/pull/5079 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
[
https://issues.apache.org/jira/browse/HUDI-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HunterHunter updated HUDI-3669:
---
Priority: Blocker (was: Major)
> HoodieRemoteException when task communicates with jm's timeline server: IP :
> port failed to respond
> ---
>
> Key: HUDI-3669
> URL: https://issues.apache.org/jira/browse/HUDI-3669
> Project: Apache Hudi
> Issue Type: Bug
> Components: flink
>Affects Versions: 0.10.1, 0.11.0
> Environment: Flink Standalone 1.13.3
>Reporter: HunterHunter
>Priority: Blocker
> Attachments: WechatIMG14.jpeg
>
>
> Flink streaming write to hudi , The task runs fine about an hour after it
> starts, and the communication is normal,When the task runs for a period of
> time,An error is reported when BucketAssignFunction communicates with JM's
> Timeline server, This error will occur under certain circumstances.
> When the task restarts after an error is reported, it happens again after
> running for a period of time,eventually cause the task to fail
> in addition：I have modified NetworkUtils' method of getting ip as suggested
> {code:java}
> org.apache.hudi.exception.HoodieRemoteException: 10.18x.xx.xx:34805 failed to
> respond
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.refresh(RemoteHoodieTableFileSystemView.java:420)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.sync(RemoteHoodieTableFileSystemView.java:484)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.common.table.view.PriorityBasedFileSystemView.sync(PriorityBasedFileSystemView.java:257)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.profile.WriteProfile.reload(WriteProfile.java:252)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssigner.reload(BucketAssigner.java:211)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.hudi.sink.partitioner.BucketAssignFunction.notifyCheckpointComplete(BucketAssignFunction.java:234)
> ~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
> ~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
> at
>
[jira] [Updated] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
[
https://issues.apache.org/jira/browse/HUDI-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HunterHunter updated HUDI-3669:
---
Description:
Flink streaming write to hudi , The task runs fine about an hour after it
starts, and the communication is normal,When the task runs for a period of
time,An error is reported when BucketAssignFunction communicates with JM's
Timeline server, This error will occur under certain circumstances.
When the task restarts after an error is reported, it happens again after
running for a period of time,eventually cause the task to fail
in addition：I have modified NetworkUtils' method of getting ip as suggested
{code:java}
org.apache.hudi.exception.HoodieRemoteException: 10.18x.xx.xx:34805 failed to
respond
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.refresh(RemoteHoodieTableFileSystemView.java:420)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.sync(RemoteHoodieTableFileSystemView.java:484)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.sync(PriorityBasedFileSystemView.java:257)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.sink.partitioner.profile.WriteProfile.reload(WriteProfile.java:252)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.sink.partitioner.BucketAssigner.reload(BucketAssigner.java:211)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.sink.partitioner.BucketAssignFunction.notifyCheckpointComplete(BucketAssignFunction.java:234)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]
Caused by: org.apache.http.NoHttpResponseException: 10.18x.xx.xx:34805 failed
to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
[jira] [Updated] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
[
https://issues.apache.org/jira/browse/HUDI-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HunterHunter updated HUDI-3669:
---
Description:
Flink streaming write to hudi , The task runs fine about an hour after it
starts, and the communication is normal,When the task runs for a period of
time,An error is reported when BucketAssignFunction communicates with JM's
Timeline server, This error will occur under certain circumstances.
When the task restarts after an error is reported, it happens again after
running for a period of time,eventually cause the task to fail
in addition：I have modified NetworkUtils' method of getting ip as suggested
{code:java}
org.apache.hudi.exception.HoodieRemoteException: 10.18x.xx.xx:34805 failed to
respondat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.refresh(RemoteHoodieTableFileSystemView.java:420)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.sync(RemoteHoodieTableFileSystemView.java:484)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.sync(PriorityBasedFileSystemView.java:257)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.sink.partitioner.profile.WriteProfile.reload(WriteProfile.java:252)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.hudi.sink.partitioner.BucketAssigner.reload(BucketAssigner.java:211)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]at
org.apache.hudi.sink.partitioner.BucketAssignFunction.notifyCheckpointComplete(BucketAssignFunction.java:234)
~[hudi-flink-bundle_2.11-0.10-vipshop-SNAPSHOT.jar:0.10-vipshop-SNAPSHOT]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
~[flink-dist_2.11-1.13.3-SNAPSHOT.jar:1.13.3-SNAPSHOT]at
java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]Caused by:
org.apache.http.NoHttpResponseException: 10.18x.xx.xx:34805 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
[jira] [Created] (HUDI-3669) HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond
HunterHunter created HUDI-3669: -- Summary: HoodieRemoteException when task communicates with jm's timeline server: IP : port failed to respond Key: HUDI-3669 URL: https://issues.apache.org/jira/browse/HUDI-3669 Project: Apache Hudi Issue Type: Bug Components: flink Affects Versions: 0.10.1, 0.11.0 Environment: Flink Standalone 1.13.3 Reporter: HunterHunter Attachments: WechatIMG14.jpeg Flink streaming write to hudi , The task runs fine about an hour after it starts, and the communication is normal,When the task runs for a period of time,An error is reported when BucketAssignFunction communicates with JM's Timeline server, This error will occur under certain circumstances. When the task restarts after an error is reported, it happens again after running for a period of time,eventually cause the task to fail in addition：I have modified NetworkUtils' method of getting ip as suggested -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] XuQianJin-Stars commented on pull request #5072: [HUDI-3665] Support flink multiple versions
XuQianJin-Stars commented on pull request #5072: URL: https://github.com/apache/hudi/pull/5072#issuecomment-1073452161 > > hi @danny0405 `bot.yml` add the flink multi version to build? > > Yes, can you help with that ? I tried to add it but find that there would be too many builds if i add in another dimension, there are already 2 dimensions now: spark and scala Well, I'll take time to think about it. There are other things at hand. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073451236 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 08d86460efaf9ab648ecdd50a1cf3c8087ac1786 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073415813 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) * 08d86460efaf9ab648ecdd50a1cf3c8087ac1786 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rkkalluri commented on issue #4635: [SUPPORT] Bulk write failing due to hudi timeline archive exception
rkkalluri commented on issue #4635:
URL: https://github.com/apache/hudi/issues/4635#issuecomment-1073440851
I am able to reproduce this locally on 0.11.0-SNAPSHOT
22/03/20 21:59:22 INFO HoodieActiveTimeline: Loaded instants upto :
Option{val=[==>20220320215909174__commit__INFLIGHT]}
22/03/20 21:59:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient
from file:///tmp/hudi_4635
22/03/20 21:59:22 INFO HoodieTableConfig: Loading table properties from
file:/tmp/hudi_4635/.hoodie/hoodie.properties
22/03/20 21:59:22 INFO HoodieTableMetaClient: Finished Loading Table of type
COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:///tmp/hudi_4635
22/03/20 21:59:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient
from file:///tmp/hudi_4635/.hoodie/metadata
22/03/20 21:59:22 INFO HoodieTableConfig: Loading table properties from
file:/tmp/hudi_4635/.hoodie/metadata/.hoodie/hoodie.properties
22/03/20 21:59:22 INFO HoodieTableMetaClient: Finished Loading Table of type
MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
file:///tmp/hudi_4635/.hoodie/metadata
22/03/20 21:59:22 INFO FileSystemViewManager: Creating View Manager with
storage type :REMOTE_FIRST
22/03/20 21:59:22 INFO FileSystemViewManager: Creating remote first table
view
22/03/20 21:59:22 INFO TransactionUtils: Successfully resolved conflicts, if
any
22/03/20 21:59:22 INFO BaseHoodieWriteClient: Committing 20220320215909174
action commit
22/03/20 21:59:22 INFO SparkContext: Starting job: collect at
HoodieSparkEngineContext.java:134
22/03/20 21:59:22 INFO DAGScheduler: Got job 680 (collect at
HoodieSparkEngineContext.java:134) with 1 output partitions
22/03/20 21:59:22 INFO DAGScheduler: Final stage: ResultStage 984 (collect
at HoodieSparkEngineContext.java:134)
22/03/20 21:59:22 INFO DAGScheduler: Parents of final stage: List()
22/03/20 21:59:22 INFO DAGScheduler: Missing parents: List()
22/03/20 21:59:22 INFO DAGScheduler: Submitting ResultStage 984
(MapPartitionsRDD[2117] at flatMap at HoodieSparkEngineContext.java:134), which
has no missing parents
22/03/20 21:59:22 INFO MemoryStore: Block broadcast_848 stored as values in
memory (estimated size 99.5 KiB, free 357.5 MiB)
22/03/20 21:59:22 INFO MemoryStore: Block broadcast_848_piece0 stored as
bytes in memory (estimated size 35.1 KiB, free 357.5 MiB)
22/03/20 21:59:22 INFO BlockManagerInfo: Added broadcast_848_piece0 in
memory on rkalluri.attlocal.net:63252 (size: 35.1 KiB, free: 364.0 MiB)
22/03/20 21:59:22 INFO SparkContext: Created broadcast 848 from broadcast at
DAGScheduler.scala:1478
22/03/20 21:59:22 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 984 (MapPartitionsRDD[2117] at flatMap at
HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/03/20 21:59:22 INFO TaskSchedulerImpl: Adding task set 984.0 with 1 tasks
resource profile 0
22/03/20 21:59:22 INFO TaskSetManager: Starting task 0.0 in stage 984.0 (TID
2266) (rkalluri.attlocal.net, executor driver, partition 0, PROCESS_LOCAL, 4387
bytes) taskResourceAssignments Map()
22/03/20 21:59:22 INFO Executor: Running task 0.0 in stage 984.0 (TID 2266)
22/03/20 21:59:22 INFO Executor: Finished task 0.0 in stage 984.0 (TID
2266). 888 bytes result sent to driver
22/03/20 21:59:22 INFO TaskSetManager: Finished task 0.0 in stage 984.0 (TID
2266) in 22 ms on rkalluri.attlocal.net (executor driver) (1/1)
22/03/20 21:59:22 INFO TaskSchedulerImpl: Removed TaskSet 984.0, whose tasks
have all completed, from pool
22/03/20 21:59:22 INFO DAGScheduler: ResultStage 984 (collect at
HoodieSparkEngineContext.java:134) finished in 0.041 s
22/03/20 21:59:22 INFO DAGScheduler: Job 680 is finished. Cancelling
potential speculative or zombie tasks for this job
22/03/20 21:59:22 INFO TaskSchedulerImpl: Killing all running tasks in stage
984: Stage finished
22/03/20 21:59:22 INFO DAGScheduler: Job 680 finished: collect at
HoodieSparkEngineContext.java:134, took 0.042314 s
22/03/20 21:59:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient
from file:///tmp/hudi_4635
22/03/20 21:59:22 INFO HoodieTableConfig: Loading table properties from
file:/tmp/hudi_4635/.hoodie/hoodie.properties
22/03/20 21:59:22 INFO HoodieTableMetaClient: Finished Loading Table of type
COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:///tmp/hudi_4635
22/03/20 21:59:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient
from file:///tmp/hudi_4635/.hoodie/metadata
22/03/20 21:59:22 INFO HoodieTableConfig: Loading table properties from
file:/tmp/hudi_4635/.hoodie/metadata/.hoodie/hoodie.properties
22/03/20 21:59:22 INFO HoodieTableMetaClient: Finished Loading Table of type
MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
file:///tmp/hudi_4635/.hoodie/metadata
22/03/20 21:59:22 INFO HoodieActiveTimeline: Loaded instants upto :
[GitHub] [hudi] hudi-bot commented on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot commented on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1073439418 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * ec26a6b6d14f16de6db11dad782fa9c0002dcd04 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot removed a comment on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1073404509 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * 14fff4af616c371f61db8d28e7d66e2dd4e24a5d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7071) * ec26a6b6d14f16de6db11dad782fa9c0002dcd04 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2751) To avoid the duplicates for streaming read MOR table
[
https://issues.apache.org/jira/browse/HUDI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2751:
--
Priority: Critical (was: Blocker)
> To avoid the duplicates for streaming read MOR table
>
>
> Key: HUDI-2751
> URL: https://issues.apache.org/jira/browse/HUDI-2751
> Project: Apache Hudi
> Issue Type: Task
> Components: Common Core
>Reporter: Danny Chen
>Assignee: sivabalan narayanan
>Priority: Critical
> Fix For: 0.12.0
>
>
> Imagine there are commits on the timeline:
> {noformat}
> -delta-99 - commit 100(include 99 delta data
> set) - delta-101 - delta-102 -
> first read ->| second read ->
> – range 1 ---| --range 2
> ---|
> {noformat}
> instant 99, 101, 102 are successful non-compaction delta commits;
> instant 100 is successful compaction instant.
> The first inc read consumes to instant 99 and the second read consumes from
> instant 100 to instant 102, the second read would consumes the commit files
> of instant 100 which has already been consumed before.
> The duplicate reading happens when this condition triggers: a compaction
> instant schedules then completes in *one* consume range.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HUDI-2751) To avoid the duplicates for streaming read MOR table
[
https://issues.apache.org/jira/browse/HUDI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2751:
--
Fix Version/s: 0.12.0
(was: 0.11.0)
> To avoid the duplicates for streaming read MOR table
>
>
> Key: HUDI-2751
> URL: https://issues.apache.org/jira/browse/HUDI-2751
> Project: Apache Hudi
> Issue Type: Task
> Components: Common Core
>Reporter: Danny Chen
>Assignee: sivabalan narayanan
>Priority: Blocker
> Fix For: 0.12.0
>
>
> Imagine there are commits on the timeline:
> {noformat}
> -delta-99 - commit 100(include 99 delta data
> set) - delta-101 - delta-102 -
> first read ->| second read ->
> – range 1 ---| --range 2
> ---|
> {noformat}
> instant 99, 101, 102 are successful non-compaction delta commits;
> instant 100 is successful compaction instant.
> The first inc read consumes to instant 99 and the second read consumes from
> instant 100 to instant 102, the second read would consumes the commit files
> of instant 100 which has already been consumed before.
> The duplicate reading happens when this condition triggers: a compaction
> instant schedules then completes in *one* consume range.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HUDI-2751) To avoid the duplicates for streaming read MOR table
[
https://issues.apache.org/jira/browse/HUDI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509568#comment-17509568
]
sivabalan narayanan commented on HUDI-2751:
---
Synced up via direct chat w/ Danny. here is the gist.
Streaming read in both spark and flink will watch for new timeline files and
serve them to the caller. Even though we do filter for commit times in records,
this ticket is about optimization where in we can avoid the filtering if
possible.
but this spans multiple areas and not just MOR compaction.
For eg, COW merge, MOR compaction and clustering as well.
So, this needs to holistic thought. Myself and Danny will tackle this for 0.12.
> To avoid the duplicates for streaming read MOR table
>
>
> Key: HUDI-2751
> URL: https://issues.apache.org/jira/browse/HUDI-2751
> Project: Apache Hudi
> Issue Type: Task
> Components: Common Core
>Reporter: Danny Chen
>Assignee: sivabalan narayanan
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Imagine there are commits on the timeline:
> {noformat}
> -delta-99 - commit 100(include 99 delta data
> set) - delta-101 - delta-102 -
> first read ->| second read ->
> – range 1 ---| --range 2
> ---|
> {noformat}
> instant 99, 101, 102 are successful non-compaction delta commits;
> instant 100 is successful compaction instant.
> The first inc read consumes to instant 99 and the second read consumes from
> instant 100 to instant 102, the second read would consumes the commit files
> of instant 100 which has already been consumed before.
> The duplicate reading happens when this condition triggers: a compaction
> instant schedules then completes in *one* consume range.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot commented on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073432711 ## CI report: * 8ebfdf950e37b4a68877801fbd1f71f795aac7df Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7113) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot removed a comment on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073403315 ## CI report: * 9f05eb92e3d78dedc8b84dbf5aadb20c926b6229 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7099) * 8ebfdf950e37b4a68877801fbd1f71f795aac7df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7113) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-3453) Metadata table throws NPE when scheduling compaction plan
[
https://issues.apache.org/jira/browse/HUDI-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509564#comment-17509564
]
Yue Zhang commented on HUDI-3453:
-
Hi [~danny0405] Looks like there is a concurrency issue here. Could you please
offer more logs?
For example : LOG.info("Compaction looking for files to compact in " +
partitionPaths + " partitions");
> Metadata table throws NPE when scheduling compaction plan
> -
>
> Key: HUDI-3453
> URL: https://issues.apache.org/jira/browse/HUDI-3453
> Project: Apache Hudi
> Issue Type: Bug
> Components: core
>Reporter: Danny Chen
>Assignee: Yue Zhang
>Priority: Blocker
> Fix For: 0.11.0
>
>
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieException: Error occurs when
> executing flatMap
> at
> org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:50)
> at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
> at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
> at java.util.stream.AbstractTask.compute(AbstractTask.java:327)
> at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1870)
> at
> java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:2467)
> at
> java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:324)
> at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
> at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at
> org.apache.hudi.client.common.HoodieFlinkEngineContext.flatMap(HoodieFlinkEngineContext.java:136)
> at
> org.apache.hudi.table.action.compact.HoodieCompactor.generateCompactionPlan(HoodieCompactor.java:263)
> at
> org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.scheduleCompaction(ScheduleCompactionActionExecutor.java:122)
> at
> org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.execute(ScheduleCompactionActionExecutor.java:92)
> at
> org.apache.hudi.table.HoodieFlinkMergeOnReadTable.scheduleCompaction(HoodieFlinkMergeOnReadTable.java:109)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableServiceInternal(AbstractHoodieWriteClient.java:1100)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableService(AbstractHoodieWriteClient.java:1083)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompactionAtInstant(AbstractHoodieWriteClient.java:850)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompaction(AbstractHoodieWriteClient.java:841)
> at
> org.apache.hudi.util.CompactionUtil.scheduleCompaction(CompactionUtil.java:64)
> at
> org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$2(StreamWriteOperatorCoordinator.java:229)
> at
> org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
> ... 3 more
> Caused by: org.apache.hudi.exception.HoodieMetadataException: Failed to
> retrieve files in partition
> oss://datalake-huifu/hudi/poc/ods/pnrweb_prod/trans_log/20220216 from metadata
> at
> org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:124)
> at
> org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
> at
> org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:304)
> at
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
> at
> org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:295)
> at
> org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestFileSlices(AbstractTableFileSystemView.java:578)
> at
> org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:80)
> at
>
[GitHub] [hudi] YannByron commented on issue #4678: [SUPPORT] spark.read.format("hudi").schema(userSpecifiedSchema) doesn't work in version 0.10.0 ,but does work in 0.5.3
YannByron commented on issue #4678: URL: https://github.com/apache/hudi/issues/4678#issuecomment-1073427339 @nsivabalan In my private opinion, I think the usage of `schema` that can not convert datatype in branch-0.10.0 is reasonable. If want to convert data, should use `select` or `withColumnn` after `load` data. So I don't treat it as a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-3665) Support flink multiple versions
[ https://issues.apache.org/jira/browse/HUDI-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509554#comment-17509554 ] Danny Chen commented on HUDI-3665: -- Fixed via master branch: 799c78e6888ef6a375c6779c3cfe7067756d4be9 > Support flink multiple versions > --- > > Key: HUDI-3665 > URL: https://issues.apache.org/jira/browse/HUDI-3665 > Project: Apache Hudi > Issue Type: Improvement > Components: flink >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3665) Support flink multiple versions
[ https://issues.apache.org/jira/browse/HUDI-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned HUDI-3665: Assignee: Danny Chen > Support flink multiple versions > --- > > Key: HUDI-3665 > URL: https://issues.apache.org/jira/browse/HUDI-3665 > Project: Apache Hudi > Issue Type: Improvement > Components: flink >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HUDI-3665) Support flink multiple versions
[ https://issues.apache.org/jira/browse/HUDI-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-3665. -- > Support flink multiple versions > --- > > Key: HUDI-3665 > URL: https://issues.apache.org/jira/browse/HUDI-3665 > Project: Apache Hudi > Issue Type: Improvement > Components: flink >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[hudi] branch master updated (15d1c18 -> 799c78e)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 15d1c18 [MINOR] Remove flaky assert in TestInLineFileSystem (#5069) add 799c78e [HUDI-3665] Support flink multiple versions (#5072) No new revisions were added by this update. Summary of changes: azure-pipelines.yml| 8 +- hudi-client/hudi-flink-client/pom.xml | 4 +- hudi-flink-datasource/hudi-flink/pom.xml | 364 + .../apache/hudi/configuration/FlinkOptions.java| 4 +- .../apache/hudi/configuration/OptionsResolver.java | 0 .../hudi/schema/FilebasedSchemaProvider.java | 0 .../org/apache/hudi/schema/SchemaProvider.java | 0 .../apache/hudi/schema/SchemaRegistryProvider.java | 0 .../hudi/sink/BucketStreamWriteFunction.java | 13 +- .../hudi/sink/BucketStreamWriteOperator.java | 3 +- .../java/org/apache/hudi/sink/CleanFunction.java | 0 .../org/apache/hudi/sink/StreamWriteFunction.java | 0 .../org/apache/hudi/sink/StreamWriteOperator.java | 0 .../hudi/sink/StreamWriteOperatorCoordinator.java | 0 .../hudi/sink/append/AppendWriteFunction.java | 0 .../hudi/sink/append/AppendWriteOperator.java | 0 .../hudi/sink/bootstrap/BootstrapOperator.java | 6 +- .../apache/hudi/sink/bootstrap/IndexRecord.java| 0 .../bootstrap/aggregate/BootstrapAccumulator.java | 0 .../bootstrap/aggregate/BootstrapAggFunction.java | 0 .../bootstrap/batch/BatchBootstrapOperator.java| 0 .../hudi/sink/bulk/BulkInsertWriteFunction.java| 0 .../hudi/sink/bulk/BulkInsertWriteOperator.java| 0 .../hudi/sink/bulk/BulkInsertWriterHelper.java | 0 .../org/apache/hudi/sink/bulk/RowDataKeyGen.java | 0 .../apache/hudi/sink/bulk/sort/SortOperator.java | 0 .../hudi/sink/bulk/sort/SortOperatorGen.java | 0 .../sink/common/AbstractStreamWriteFunction.java | 0 .../hudi/sink/common/AbstractWriteFunction.java| 1 + .../hudi/sink/common/AbstractWriteOperator.java| 0 .../hudi/sink/common/WriteOperatorFactory.java | 0 .../apache/hudi/sink/compact/CompactFunction.java | 0 .../hudi/sink/compact/CompactionCommitEvent.java | 0 .../hudi/sink/compact/CompactionCommitSink.java| 0 .../hudi/sink/compact/CompactionPlanEvent.java | 0 .../hudi/sink/compact/CompactionPlanOperator.java | 0 .../sink/compact/CompactionPlanSourceFunction.java | 0 .../hudi/sink/compact/FlinkCompactionConfig.java | 0 .../hudi/sink/compact/HoodieFlinkCompactor.java| 0 .../org/apache/hudi/sink/event/CommitAckEvent.java | 0 .../apache/hudi/sink/event/WriteMetadataEvent.java | 0 .../java/org/apache/hudi/sink/meta/CkpMessage.java | 0 .../org/apache/hudi/sink/meta/CkpMetadata.java | 3 +- .../sink/partitioner/BucketAssignFunction.java | 0 .../hudi/sink/partitioner/BucketAssigner.java | 0 .../hudi/sink/partitioner/BucketAssigners.java | 0 .../sink/partitioner/BucketIndexPartitioner.java | 3 +- .../partitioner/profile/DeltaWriteProfile.java | 0 .../partitioner/profile/EmptyWriteProfile.java | 0 .../sink/partitioner/profile/WriteProfile.java | 0 .../sink/partitioner/profile/WriteProfiles.java| 11 +- .../hudi/sink/transform/ChainedTransformer.java| 0 .../sink/transform/RowDataToHoodieFunction.java| 0 .../RowDataToHoodieFunctionWithRateLimit.java | 6 +- .../sink/transform/RowDataToHoodieFunctions.java | 0 .../apache/hudi/sink/transform/Transformer.java| 0 .../apache/hudi/sink/utils/HiveSyncContext.java| 0 .../apache/hudi/sink/utils/NonThrownExecutor.java | 0 .../apache/hudi/sink/utils/PayloadCreation.java| 0 .../java/org/apache/hudi/sink/utils/Pipelines.java | 50 +-- .../java/org/apache/hudi/sink/utils/TimeWait.java | 0 .../java/org/apache/hudi/source/FileIndex.java | 0 .../apache/hudi/source/IncrementalInputSplits.java | 0 .../hudi/source/StreamReadMonitoringFunction.java | 0 .../org/apache/hudi/source/StreamReadOperator.java | 38 +-- .../apache/hudi/streamer/FlinkStreamerConfig.java | 0 .../apache/hudi/streamer/HoodieFlinkStreamer.java | 16 +- .../org/apache/hudi/table/HoodieTableFactory.java | 0 .../org/apache/hudi/table/HoodieTableSink.java | 0 .../org/apache/hudi/table/HoodieTableSource.java | 0 .../apache/hudi/table/catalog/CatalogOptions.java | 0 .../apache/hudi/table/catalog/HoodieCatalog.java | 0 .../hudi/table/catalog/HoodieCatalogFactory.java | 0 .../hudi/table/catalog/TableOptionProperties.java | 0 .../apache/hudi/table/format/FilePathUtils.java| 0 .../org/apache/hudi/table/format/FormatUtils.java | 2 +- .../table/format/cow/CopyOnWriteInputFormat.java | 0 .../table/format/cow/ParquetSplitReaderUtil.java | 2 +-
[GitHub] [hudi] danny0405 merged pull request #5072: [HUDI-3665] Support flink multiple versions
danny0405 merged pull request #5072: URL: https://github.com/apache/hudi/pull/5072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5072: [HUDI-3665] Support flink multiple versions
danny0405 commented on pull request #5072: URL: https://github.com/apache/hudi/pull/5072#issuecomment-1073426858 > hi @danny0405 `bot.yml` add the flink multi version to build? Yes, can you help with that ? I tried to add it but find that there would be too many builds if i add in another dimension, there are already 2 dimensions now: spark and scala -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot commented on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1073425570 ## CI report: * cc09100c92ff215f6cfad471e0d06bdc0b79b1dc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot removed a comment on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1073400378 ## CI report: * dcb127430ce66003aa396f96d5ee9847cb4d2528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6888) * cc09100c92ff215f6cfad471e0d06bdc0b79b1dc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] LaurenceGA commented on pull request #4286: [WIP][HUDI-2955] Upgrade Hadoop to 3.3.1, Hive to 3.1.2, HBase to 2.4.8
LaurenceGA commented on pull request #4286: URL: https://github.com/apache/hudi/pull/4286#issuecomment-1073424942 Would you be able to upgrade Hadoop to 3.3.2? It was just released this month. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073415813 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) * 08d86460efaf9ab648ecdd50a1cf3c8087ac1786 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073400527 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) * 08d86460efaf9ab648ecdd50a1cf3c8087ac1786 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] BruceKellan commented on issue #4993: [SUPPORT] Flink Streaming read about dynamic day partition
BruceKellan commented on issue #4993: URL: https://github.com/apache/hudi/issues/4993#issuecomment-1073413701 Can someone help me, Can I create partitions in advance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot removed a comment on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1073400171 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * 14fff4af616c371f61db8d28e7d66e2dd4e24a5d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7071) * ec26a6b6d14f16de6db11dad782fa9c0002dcd04 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot commented on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1073404509 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * 14fff4af616c371f61db8d28e7d66e2dd4e24a5d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7071) * ec26a6b6d14f16de6db11dad782fa9c0002dcd04 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot commented on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073403315 ## CI report: * 9f05eb92e3d78dedc8b84dbf5aadb20c926b6229 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7099) * 8ebfdf950e37b4a68877801fbd1f71f795aac7df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7113) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot removed a comment on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073401827 ## CI report: * 9f05eb92e3d78dedc8b84dbf5aadb20c926b6229 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7099) * 8ebfdf950e37b4a68877801fbd1f71f795aac7df UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot removed a comment on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073176113 ## CI report: * 9f05eb92e3d78dedc8b84dbf5aadb20c926b6229 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7099) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5004: [HUDI-1180] Upgrade HBase to 2.4.9
hudi-bot commented on pull request #5004: URL: https://github.com/apache/hudi/pull/5004#issuecomment-1073401827 ## CI report: * 9f05eb92e3d78dedc8b84dbf5aadb20c926b6229 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7099) * 8ebfdf950e37b4a68877801fbd1f71f795aac7df UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073400527 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) * 08d86460efaf9ab648ecdd50a1cf3c8087ac1786 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073399394 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot commented on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1073400378 ## CI report: * dcb127430ce66003aa396f96d5ee9847cb4d2528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6888) * cc09100c92ff215f6cfad471e0d06bdc0b79b1dc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot removed a comment on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1073399273 ## CI report: * dcb127430ce66003aa396f96d5ee9847cb4d2528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6888) * cc09100c92ff215f6cfad471e0d06bdc0b79b1dc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot commented on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1073400171 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * 14fff4af616c371f61db8d28e7d66e2dd4e24a5d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7071) * ec26a6b6d14f16de6db11dad782fa9c0002dcd04 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4309: [HUDI-3016][RFC-43] Proposal to implement Compaction/Clustering Servi…
hudi-bot removed a comment on pull request #4309: URL: https://github.com/apache/hudi/pull/4309#issuecomment-1072282189 ## CI report: * fbe27691b5d9de58128cc58158047a4df2b53750 UNKNOWN * 14fff4af616c371f61db8d28e7d66e2dd4e24a5d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7071) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073399394 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073396145 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot commented on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1073399273 ## CI report: * dcb127430ce66003aa396f96d5ee9847cb4d2528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6888) * cc09100c92ff215f6cfad471e0d06bdc0b79b1dc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables
hudi-bot removed a comment on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1066067506 ## CI report: * dcb127430ce66003aa396f96d5ee9847cb4d2528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6888) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073396145 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7111) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073395262 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073394207 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073395262 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 3d3442f90adbe39157fe55a215b60dacfbb1d2ca UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073387511 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073394207 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3668) Fix failing unit tests in hudi-integ-test
[
https://issues.apache.org/jira/browse/HUDI-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3668:
Fix Version/s: 0.11.0
> Fix failing unit tests in hudi-integ-test
> -
>
> Key: HUDI-3668
> URL: https://issues.apache.org/jira/browse/HUDI-3668
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Ethan Guo
>Priority: Major
> Fix For: 0.11.0
>
>
> org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter#testDFSTwoFilesWriteWithRollover
> {code:java}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> avroFileDeltaInputWriter.canWrite();
> Wanted 2 times:
> -> at
> org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
> But was 3 times:
> -> at
> org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
> -> at
> org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
> -> at
> org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
> at
> org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
>
[jira] [Updated] (HUDI-3668) Fix failing unit tests in hudi-integ-test
[
https://issues.apache.org/jira/browse/HUDI-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3668:
Description:
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter#testDFSTwoFilesWriteWithRollover
{code:java}
org.mockito.exceptions.verification.TooManyActualInvocations:
avroFileDeltaInputWriter.canWrite();
Wanted 2 times:
-> at
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
But was 3 times:
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
at
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at
[jira] [Created] (HUDI-3668) Fix failing unit tests in hudi-integ-test
Ethan Guo created HUDI-3668:
---
Summary: Fix failing unit tests in hudi-integ-test
Key: HUDI-3668
URL: https://issues.apache.org/jira/browse/HUDI-3668
Project: Apache Hudi
Issue Type: Bug
Reporter: Ethan Guo
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter#testDFSTwoFilesWriteWithRollover
{code:java}
org.mockito.exceptions.verification.TooManyActualInvocations:
avroFileDeltaInputWriter.canWrite();
Wanted 2 times:
-> at
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
But was 3 times:
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
-> at
org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50)
at
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073387511 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073374874 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073374874 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot removed a comment on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073374416 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
hudi-bot commented on pull request #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1073374416 ## CI report: * b52f52d1db2656d600026961c52fac6f865743fa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3667) Unit tests in hudi-integ-tests are not executed in CI
[ https://issues.apache.org/jira/browse/HUDI-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3667: - Labels: pull-request-available (was: ) > Unit tests in hudi-integ-tests are not executed in CI > - > > Key: HUDI-3667 > URL: https://issues.apache.org/jira/browse/HUDI-3667 > Project: Apache Hudi > Issue Type: Bug >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] yihua opened a new pull request #5078: [WIP][HUDI-3667] Run unit tests of hudi-integ-tests in CI
yihua opened a new pull request #5078: URL: https://github.com/apache/hudi/pull/5078 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3667) Unit tests in hudi-integ-tests are not executed in CI
[ https://issues.apache.org/jira/browse/HUDI-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3667: Summary: Unit tests in hudi-integ-tests are not executed in CI (was: Unit and functional tests in hudi-integ-tests are not executed in CI) > Unit tests in hudi-integ-tests are not executed in CI > - > > Key: HUDI-3667 > URL: https://issues.apache.org/jira/browse/HUDI-3667 > Project: Apache Hudi > Issue Type: Bug >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3667) Unit and functional tests in hudi-integ-tests are not executed in CI
Ethan Guo created HUDI-3667: --- Summary: Unit and functional tests in hudi-integ-tests are not executed in CI Key: HUDI-3667 URL: https://issues.apache.org/jira/browse/HUDI-3667 Project: Apache Hudi Issue Type: Bug Reporter: Ethan Guo Assignee: Ethan Guo Fix For: 0.11.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[hudi] branch master updated (1b6e201 -> 15d1c18)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 1b6e201 [HUDI-3663] Fixing Column Stats index to properly handle first Data Table commit (#5070) add 15d1c18 [MINOR] Remove flaky assert in TestInLineFileSystem (#5069) No new revisions were added by this update. Summary of changes: .../test/java/org/apache/hudi/common/fs/inline/TestInLineFileSystem.java | 1 - 1 file changed, 1 deletion(-)
[GitHub] [hudi] nsivabalan merged pull request #5069: [MINOR] Remove flaky assert in TestInLineFileSystem
nsivabalan merged pull request #5069: URL: https://github.com/apache/hudi/pull/5069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot commented on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073297587 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot removed a comment on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073282373 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql
nsivabalan commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1073297369 @XuQianJin-Stars : let me know once all feedback has been addressed. I can take another look and get it landed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5076: [HUDI-2757] Implement Hudi AWS Glue sync
hudi-bot removed a comment on pull request #5076: URL: https://github.com/apache/hudi/pull/5076#issuecomment-1073279442 ## CI report: * 587f3374a0b52407940b7dbee8a66a4fdebb13e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7106) * 7bb94c049a0c5800d54146a5b87878c0fc7d7e60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7107) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5076: [HUDI-2757] Implement Hudi AWS Glue sync
hudi-bot commented on pull request #5076: URL: https://github.com/apache/hudi/pull/5076#issuecomment-1073289874 ## CI report: * 7bb94c049a0c5800d54146a5b87878c0fc7d7e60 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7107) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on issue #4891: Clustering not working on large table and partitions
suryaprasanna commented on issue #4891: URL: https://github.com/apache/hudi/issues/4891#issuecomment-1073287297 [FelixKJose](https://github.com/FelixKJose) > 1. Let's say my each partitions (date) are large partitions (eg. 6.5 TB uncompressed data), so having the frequent async clustering is suggested right? I am running on r5.4xlarge (meaning 37GB driver memory), so what will be best clusering frequency? You can start with one spark per partition(so it creates one replacecommit for one sorting operation on a partition) and keep increasing no. of partitions to cluster in a single job, to find out the breaking point. I think with the above driver memory it can easily handle 4 partitions. You need to play around with your data to figure out the amount of parallelism you can give. Although locking, archival or other services will be bottleneck when you run clustering with very high parallelism. **Note:** Make `"hoodie.clustering.async.max.commits"` to `"0"`, that way multiple clustering plans can be generated in parallel. Since the clustering jobs are running on different partitions you should be ok. > 2. What will be the best value for hoodie.clustering.plan.strategy.small.file.limit? Also any other configurations I should be using considering the partition size as mentioned above Since, you are using `"hoodie.clustering.plan.strategy.sort.columns"` config, I am assuming you want to sort the partitions. Sorting operation main objective is to sort the data based on columns and create new set of files with parquet file sizes close to value, that is given under `hoodie.clustering.plan.strategy.target.file.max.bytes`. So, you should not worry about small.file.limit, since sorting operation is anyway going to rewrite entire partition and create larger parquet files. I would suggest to keep the small.file.limit value to be higher that way all the files are included. `hoodie.clustering.plan.strategy.small.file.limit` is mainly used for stitching operation. Where you are not sorting data but stitching small files together so that you can reduce the small file limit. > 3. Which lock provider is advised if I am running on AWS EMR? I do not have much knowledge about AWS stack, default lock provider i.e. ZookeeperBasedLockProvider works just fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot removed a comment on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073281768 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot commented on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073282373 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
hudi-bot commented on pull request #5077: URL: https://github.com/apache/hudi/pull/5077#issuecomment-1073281768 ## CI report: * e268f4ceb0883f1092ab468f466a90baf485d5c3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3664) Column Stats are computed incorrectly right now
[ https://issues.apache.org/jira/browse/HUDI-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3664: - Labels: pull-request-available (was: ) > Column Stats are computed incorrectly right now > --- > > Key: HUDI-3664 > URL: https://issues.apache.org/jira/browse/HUDI-3664 > Project: Apache Hudi > Issue Type: Bug >Reporter: Sagar Sumit >Assignee: Sagar Sumit >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > > Instead of using string comparators, convert Avro schema to native Java type. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] codope opened a new pull request #5077: [HUDI-3664] Handle type conversion for comparison of column range metadata
codope opened a new pull request #5077: URL: https://github.com/apache/hudi/pull/5077 ## What is the purpose of the pull request Fixes an issue with comparison of column range metadata. Instead of using string comparators, we get the type of field from the avro schema and convert to native Java type. ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5076: [HUDI-2757] Implement Hudi AWS Glue sync
hudi-bot removed a comment on pull request #5076: URL: https://github.com/apache/hudi/pull/5076#issuecomment-1073278902 ## CI report: * 587f3374a0b52407940b7dbee8a66a4fdebb13e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7106) * 7bb94c049a0c5800d54146a5b87878c0fc7d7e60 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5076: [HUDI-2757] Implement Hudi AWS Glue sync
hudi-bot commented on pull request #5076: URL: https://github.com/apache/hudi/pull/5076#issuecomment-1073279442 ## CI report: * 587f3374a0b52407940b7dbee8a66a4fdebb13e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7106) * 7bb94c049a0c5800d54146a5b87878c0fc7d7e60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7107) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
