Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
nsivabalan commented on code in PR #11553:
URL: https://github.com/apache/hudi/pull/11553#discussion_r1676772962
##
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##
@@ -70,11 +70,67 @@ public class ClusteringUtils {
public static Stream>
getAllPendingClusteringPlans(
HoodieTableMetaClient metaClient) {
List pendingClusterInstants =
-
metaClient.getActiveTimeline().filterPendingClusterTimeline().getInstants();
+
metaClient.getActiveTimeline().filterPendingReplaceOrClusteringTimeline().getInstants();
return pendingClusterInstants.stream().map(instant ->
getClusteringPlan(metaClient, instant))
.filter(Option::isPresent).map(Option::get);
}
+ /**
+ * Returns the pending clustering instant. This can be older pending replace
commit or a new
+ * clustering inflight commit. After HUDI-7905, all the requested and
inflight clustering instants
+ * use clustering action instead of replacecommit.
+ */
+ public static Option getInflightClusteringInstant(String
timestamp, HoodieActiveTimeline activeTimeline) {
+HoodieTimeline pendingReplaceOrClusterTimeline =
activeTimeline.filterPendingReplaceOrClusteringTimeline();
+HoodieInstant inflightInstant =
HoodieTimeline.getClusteringCommitInflightInstant(timestamp);
+if (pendingReplaceOrClusterTimeline.containsInstant(inflightInstant)) {
+ return Option.of(inflightInstant);
+}
+inflightInstant =
HoodieTimeline.getReplaceCommitInflightInstant(timestamp);
+return
Option.ofNullable(pendingReplaceOrClusterTimeline.containsInstant(inflightInstant)
? inflightInstant : null);
+ }
+
+ /**
+ * Returns the pending clustering instant. This can be older requested
replace commit or a new
+ * clustering requested commit. After HUDI-7905, all the requested and
inflight clustering instants
+ * use clustering action instead of replacecommit.
+ */
+ public static Option getRequestedClusteringInstant(String
timestamp, HoodieActiveTimeline activeTimeline) {
+HoodieTimeline pendingReplaceOrClusterTimeline =
activeTimeline.filterPendingReplaceOrClusteringTimeline();
+HoodieInstant requestedInstant =
HoodieTimeline.getClusteringCommitRequestedInstant(timestamp);
+if (pendingReplaceOrClusterTimeline.containsInstant(requestedInstant)) {
+ return Option.of(requestedInstant);
+}
+requestedInstant =
HoodieTimeline.getReplaceCommitRequestedInstant(timestamp);
+return
Option.ofNullable(pendingReplaceOrClusterTimeline.containsInstant(requestedInstant)
? requestedInstant : null);
+ }
+
+ /**
+ * Transitions the provided clustering instant fron inflight to complete
based on the clustering
+ * action type. After HUDI-7905, the new clustering commits are written with
clustering action.
+ */
+ public static void transitionClusterInflightToComplete(boolean shouldLock,
HoodieInstant clusteringInstant,
Review Comment:
oh 1 thing. if there is a pending replacecommit(just requested) in timeline
just before upgrade, then we need to use replacecommit (and not clustering) for
inflight right?
are we handling this case.
##
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##
@@ -70,11 +70,67 @@ public class ClusteringUtils {
public static Stream>
getAllPendingClusteringPlans(
HoodieTableMetaClient metaClient) {
List pendingClusterInstants =
-
metaClient.getActiveTimeline().filterPendingClusterTimeline().getInstants();
+
metaClient.getActiveTimeline().filterPendingReplaceOrClusteringTimeline().getInstants();
return pendingClusterInstants.stream().map(instant ->
getClusteringPlan(metaClient, instant))
.filter(Option::isPresent).map(Option::get);
}
+ /**
+ * Returns the pending clustering instant. This can be older pending replace
commit or a new
+ * clustering inflight commit. After HUDI-7905, all the requested and
inflight clustering instants
+ * use clustering action instead of replacecommit.
+ */
+ public static Option getInflightClusteringInstant(String
timestamp, HoodieActiveTimeline activeTimeline) {
+HoodieTimeline pendingReplaceOrClusterTimeline =
activeTimeline.filterPendingReplaceOrClusteringTimeline();
+HoodieInstant inflightInstant =
HoodieTimeline.getClusteringCommitInflightInstant(timestamp);
+if (pendingReplaceOrClusterTimeline.containsInstant(inflightInstant)) {
+ return Option.of(inflightInstant);
+}
+inflightInstant =
HoodieTimeline.getReplaceCommitInflightInstant(timestamp);
+return
Option.ofNullable(pendingReplaceOrClusterTimeline.containsInstant(inflightInstant)
? inflightInstant : null);
+ }
+
+ /**
+ * Returns the pending clustering instant. This can be older requested
replace commit or a new
+ * clustering requested commit. After HUDI-7905, all the requested and
inflight clustering instants
+ * use clustering action instead of
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
nsivabalan commented on code in PR #11553:
URL: https://github.com/apache/hudi/pull/11553#discussion_r1676772100
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##
@@ -265,9 +264,9 @@ protected abstract Iterator>
handleUpdate(String partitionPath
protected HoodieWriteMetadata>
executeClustering(HoodieClusteringPlan clusteringPlan) {
context.setJobStatus(this.getClass().getSimpleName(), "Clustering records
for " + config.getTableName());
-HoodieInstant instant =
HoodieTimeline.getClusterCommitRequestedInstant(instantTime);
+HoodieInstant instant =
ClusteringUtils.getRequestedClusteringInstant(instantTime,
table.getActiveTimeline()).get();
// Mark instant as clustering inflight
-table.getActiveTimeline().transitionClusterRequestedToInflight(instant,
Option.empty());
+ClusteringUtils.transitionClusterRequestedToInflight(instant,
Option.empty(), table.getActiveTimeline());
Review Comment:
minor.
ClusteringUtils.transitionClusterRequestedToInflight
->
ClusteringUtils.transitionClusterOrRequestedToInflight
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225948371 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 258b695176f274a406b485c64bbc2a88091e89f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24839) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225775837 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * f65201662e752fae45f48055aa4169723a7dcd7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24838) * 258b695176f274a406b485c64bbc2a88091e89f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24839) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225759347 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * f65201662e752fae45f48055aa4169723a7dcd7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24838) * 258b695176f274a406b485c64bbc2a88091e89f1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225496093 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * f65201662e752fae45f48055aa4169723a7dcd7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24838) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225427336 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 39affbca1f586891c4567ab9aa99713011d2dee4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24826) * f65201662e752fae45f48055aa4169723a7dcd7c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24838) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2225416942 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 39affbca1f586891c4567ab9aa99713011d2dee4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24826) * f65201662e752fae45f48055aa4169723a7dcd7c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-73 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 39affbca1f586891c4567ab9aa99713011d2dee4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24826) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-721280 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * dd1dde0eedca311f2fd0803a4b86f0689a5f528a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24821) * 39affbca1f586891c4567ab9aa99713011d2dee4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-271195 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * dd1dde0eedca311f2fd0803a4b86f0689a5f528a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24821) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-257432 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * ffced867810289bde5009de38ada8a7521fac1ee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24820) * dd1dde0eedca311f2fd0803a4b86f0689a5f528a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-243168 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 0960a9f7454d0b7daae0a78d1993af11b94cea53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24808) * ffced867810289bde5009de38ada8a7521fac1ee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2221181265 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 0960a9f7454d0b7daae0a78d1993af11b94cea53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24808) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2221109881 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 77afe6dfa6dfa6f6180929e052d6ad75d8d3618b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24770) * 0960a9f7454d0b7daae0a78d1993af11b94cea53 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2213233857 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 77afe6dfa6dfa6f6180929e052d6ad75d8d3618b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2213163407 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 452e1fa55f644ca6ffae2b99b1fd8d8873233d32 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24713) * 77afe6dfa6dfa6f6180929e052d6ad75d8d3618b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2213141547 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 452e1fa55f644ca6ffae2b99b1fd8d8873233d32 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24713) * 77afe6dfa6dfa6f6180929e052d6ad75d8d3618b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
lokeshj1703 commented on code in PR #11553:
URL: https://github.com/apache/hudi/pull/11553#discussion_r1668053090
##
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##
@@ -241,6 +241,7 @@ private Comparable[] readCommit(GenericRecord record,
boolean skipMetadata) {
case HoodieTimeline.COMPACTION_ACTION:
return commitDetail(record, "hoodieCompactionMetadata", skipMetadata);
case HoodieTimeline.REPLACE_COMMIT_ACTION:
+ case HoodieTimeline.CLUSTER_ACTION:
Review Comment:
Addressed
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##
@@ -821,16 +821,8 @@ protected void archive(HoodieTable table) {
* @return
*/
private HoodieTimeline
getInflightTimelineExcludeCompactionAndClustering(HoodieTableMetaClient
metaClient) {
-HoodieTimeline inflightTimelineWithReplaceCommit =
metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
-HoodieTimeline inflightTimelineExcludeClusteringCommit =
inflightTimelineWithReplaceCommit.filter(instant -> {
- if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
-Option> instantPlan =
ClusteringUtils.getClusteringPlan(metaClient, instant);
-return !instantPlan.isPresent();
- } else {
-return true;
- }
-});
-return inflightTimelineExcludeClusteringCommit;
+HoodieTimeline inflightTimelineExcludingCompaction =
metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
+return inflightTimelineExcludingCompaction.filter(instant ->
!ClusteringUtils.isClusteringInstant(inflightTimelineExcludingCompaction,
instant));
Review Comment:
> For backwards compatibility purpose, we had to consider pending replace
commits also in addition to pending clustering commits. So, lets ensure to
check both timeline for now.
If we are going to check both timeline, then this would remain as it is?
Similarly for the other comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
nsivabalan commented on code in PR #11553:
URL: https://github.com/apache/hudi/pull/11553#discussion_r1667079266
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##
@@ -880,8 +872,7 @@ protected Map>
getPendingRollbackInfos
String instantToRollback =
rollbackPlan.getInstantToRollback().getCommitTime();
if (ignoreCompactionAndClusteringInstants) {
if (!HoodieTimeline.COMPACTION_ACTION.equals(action)) {
-boolean isClustering =
HoodieTimeline.REPLACE_COMMIT_ACTION.equals(action)
-&& ClusteringUtils.getClusteringPlan(metaClient, new
HoodieInstant(true, action, instantToRollback)).isPresent();
+boolean isClustering =
ClusteringUtils.isClusteringInstant(metaClient.getActiveTimeline(), new
HoodieInstant(true, action, instantToRollback));
Review Comment:
again, if its inflight instant that we are checking, action type would
suffice.
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##
@@ -821,16 +821,8 @@ protected void archive(HoodieTable table) {
* @return
*/
private HoodieTimeline
getInflightTimelineExcludeCompactionAndClustering(HoodieTableMetaClient
metaClient) {
-HoodieTimeline inflightTimelineWithReplaceCommit =
metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
-HoodieTimeline inflightTimelineExcludeClusteringCommit =
inflightTimelineWithReplaceCommit.filter(instant -> {
- if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
-Option> instantPlan =
ClusteringUtils.getClusteringPlan(metaClient, instant);
-return !instantPlan.isPresent();
- } else {
-return true;
- }
-});
-return inflightTimelineExcludeClusteringCommit;
+HoodieTimeline inflightTimelineExcludingCompaction =
metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
+return inflightTimelineExcludingCompaction.filter(instant ->
!ClusteringUtils.isClusteringInstant(inflightTimelineExcludingCompaction,
instant));
Review Comment:
won't this be simplified w/ this patch.
inflightTimelineExcludingCompaction will only include pending instants and
not compaction.
so, we might as well just filter out CLUSTER_ACTION right? we do not need to
deser the requested to find if its clustering.
infact thats one of the major wins w/ this change.
##
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##
@@ -241,6 +241,7 @@ private Comparable[] readCommit(GenericRecord record,
boolean skipMetadata) {
case HoodieTimeline.COMPACTION_ACTION:
return commitDetail(record, "hoodieCompactionMetadata", skipMetadata);
case HoodieTimeline.REPLACE_COMMIT_ACTION:
+ case HoodieTimeline.CLUSTER_ACTION:
Review Comment:
I would go with "CLUSTERING_ACTION", since we named "COMPACTION_ACTION" and
not "COMPACT_ACTION".
also, verb makes sense for timeline apis
getPendingCompactionTimeline vs getPendingCompactTimeline
getPendingClusteringTimeline vs getPendingClusterTimeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208984683 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 452e1fa55f644ca6ffae2b99b1fd8d8873233d32 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24713) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208847570 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * cc375bc8ee00bee501b2937dbb6f2054c0fbe2d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24705) * 452e1fa55f644ca6ffae2b99b1fd8d8873233d32 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24713) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208829897 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * cc375bc8ee00bee501b2937dbb6f2054c0fbe2d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24705) * 452e1fa55f644ca6ffae2b99b1fd8d8873233d32 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208274572 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * cc375bc8ee00bee501b2937dbb6f2054c0fbe2d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24705) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208210061 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * c88af1906b580dd3d1497d48daa072728d6f8127 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24704) * cc375bc8ee00bee501b2937dbb6f2054c0fbe2d4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24705) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208196940 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * c88af1906b580dd3d1497d48daa072728d6f8127 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24704) * cc375bc8ee00bee501b2937dbb6f2054c0fbe2d4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208138089 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * c88af1906b580dd3d1497d48daa072728d6f8127 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24704) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2208131882 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * c88af1906b580dd3d1497d48daa072728d6f8127 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2207962441 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
lokeshj1703 commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2207900106 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2207877507 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2207871843 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
lokeshj1703 commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2207852135 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206931281 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206917373 ## CI report: * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206850835 ## CI report: * a2e722c484f73e8082f36e38b3e25100ebdeebc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24686) * 3e526156f7bf7121008c4965bedeeadd969f798a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24687) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206825286 ## CI report: * a2e722c484f73e8082f36e38b3e25100ebdeebc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24686) * 3e526156f7bf7121008c4965bedeeadd969f798a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206186237 ## CI report: * a2e722c484f73e8082f36e38b3e25100ebdeebc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24686) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2206059518 ## CI report: * 1e2b60af5b816e1b9cd89fb40a47c8550337fe5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24672) * a2e722c484f73e8082f36e38b3e25100ebdeebc8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2203114173 ## CI report: * 1e2b60af5b816e1b9cd89fb40a47c8550337fe5f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2203276376 ## CI report: * 1e2b60af5b816e1b9cd89fb40a47c8550337fe5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24672) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-2203129962 ## CI report: * 1e2b60af5b816e1b9cd89fb40a47c8550337fe5f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24672) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
