Jackie-Jiang commented on code in PR #11740:
URL: https://github.com/apache/pinot/pull/11740#discussion_r1361177004


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java:
##########
@@ -666,6 +678,92 @@ public RebalanceResult rebalance(
     }
   }
 
+  @DELETE
+  @Produces(MediaType.APPLICATION_JSON)
+  @Authenticate(AccessType.UPDATE)
+  @Path("/tables/{tableName}/rebalance")
+  @Authorize(targetType = TargetType.TABLE, paramName = "tableName", action = 
Actions.Table.ABORT_REBALANCE)
+  @ApiOperation(value = "Abort all rebalance jobs for the given table, and 
noop if no rebalance is running", notes =
+      "Abort all rebalance jobs for the given table, and noop if no rebalance 
is running")
+  public List<String> abortRebalance(
+      @ApiParam(value = "Name of the table to rebalance", required = true) 
@PathParam("tableName") String tableName,
+      @ApiParam(value = "OFFLINE|REALTIME", required = true) 
@QueryParam("type") String tableTypeStr) {
+    String tableNameWithType = constructTableNameWithType(tableName, 
tableTypeStr);
+    String zkPath = 
ZKMetadataProvider.constructPropertyStorePathForControllerJob(ControllerJobType.TABLE_REBALANCE);
+    List<String> abortedJobIds = new ArrayList<>();
+    // The rebalance job updates progress status kept in ZK regularly. If it 
sees ABORTED status, it aborts itself.
+    boolean updated = 
_pinotHelixResourceManager.updateAllJobsForTable(tableNameWithType, zkPath, 
jobMetadata -> {
+      String jobId = jobMetadata.get(CommonConstants.ControllerJob.JOB_ID);
+      try {
+        String jobStatsInStr = 
jobMetadata.get(RebalanceJobConstants.JOB_METADATA_KEY_REBALANCE_PROGRESS_STATS);
+        TableRebalanceProgressStats jobStats =
+            JsonUtils.stringToObject(jobStatsInStr, 
TableRebalanceProgressStats.class);
+        if (jobStats.getStatus() != RebalanceResult.Status.IN_PROGRESS) {
+          return;
+        }
+        abortedJobIds.add(jobId);
+        LOGGER.info("Abort rebalance job: {} for table: {}", jobId, 
tableNameWithType);
+        jobStats.setStatus(RebalanceResult.Status.ABORTED);
+        
jobMetadata.put(RebalanceJobConstants.JOB_METADATA_KEY_REBALANCE_PROGRESS_STATS,
+            JsonUtils.objectToString(jobStats));
+
+        // Also cancel following retries for the rebalance job.
+        String jobCtxInStr = 
jobMetadata.get(RebalanceJobConstants.JOB_METADATA_KEY_REBALANCE_ATTEMPT_CONTEXT);
+        if (StringUtils.isEmpty(jobCtxInStr)) {
+          // In case the job is submitted by older code, it may not have job 
ctx field.
+          return;
+        }
+        TableRebalanceAttemptContext jobCtx = 
JsonUtils.stringToObject(jobCtxInStr, TableRebalanceAttemptContext.class);
+        jobCtx.setCancelRetry(true);

Review Comment:
   Do we need this extra flag? IMO we should use the rebalance status to mark 
whether we need retry. If `ABORTED` doesn't always mean no retry, we can 
introduce `CANCELLED`



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2063,28 +2070,76 @@ public boolean addNewReloadAllSegmentsJob(String 
tableNameWithType, String jobId
         
ZKMetadataProvider.constructPropertyStorePathForControllerJob(ControllerJobType.RELOAD_SEGMENT));
   }
 
+  /**
+   * Adds a new job metadata for controller job like table rebalance or reload 
into ZK
+   * @param jobId job's UUID
+   * @param jobMetadata the job metadata
+   * @param jobResourcePath zkPath to add the new job metadata
+   * @return boolean representing success / failure of the ZK write step
+   */
   public boolean addControllerJobToZK(String jobId, Map<String, String> 
jobMetadata, String jobResourcePath) {

Review Comment:
   Not introduced in this PR, but suggest changing the last argument to 
`ControllerJobType`



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceJobConstants.java:
##########
@@ -24,4 +24,6 @@ private RebalanceJobConstants() {
 
   // Progress status of the rebalance operartion
   public static final String JOB_METADATA_KEY_REBALANCE_PROGRESS_STATS = 
"REBALANCE_PROGRESS_STATS";
+  // Configs to retry the rebalance operartion
+  public static final String JOB_METADATA_KEY_REBALANCE_ATTEMPT_CONTEXT = 
"REBALANCE_ATTEMPT_CONTEXT";

Review Comment:
   Just call it `REBALANCE_CONTEXT`?



##########
pinot-core/src/main/java/org/apache/pinot/core/auth/Actions.java:
##########
@@ -89,6 +89,7 @@ public static class Cluster {
 
   // Action names for table
   public static class Table {
+    public static final String ABORT_REBALANCE = "AbortRebalance";

Review Comment:
   We might want to call it `CancelRebalance` (we have `CancelQuery`)



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1992,19 +1994,24 @@ public Map<String, Map<String, String>> 
getAllJobsForTable(String tableNameWithT
     for (ControllerJobType jobType : jobTypes) {
       String jobsResourcePath = 
ZKMetadataProvider.constructPropertyStorePathForControllerJob(jobType);
       try {
+        if (!_propertyStore.exists(jobsResourcePath, -1)) {

Review Comment:
   This check is actually redundant. The null check itself is good enough



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2063,28 +2070,76 @@ public boolean addNewReloadAllSegmentsJob(String 
tableNameWithType, String jobId
         
ZKMetadataProvider.constructPropertyStorePathForControllerJob(ControllerJobType.RELOAD_SEGMENT));
   }
 
+  /**
+   * Adds a new job metadata for controller job like table rebalance or reload 
into ZK
+   * @param jobId job's UUID
+   * @param jobMetadata the job metadata
+   * @param jobResourcePath zkPath to add the new job metadata
+   * @return boolean representing success / failure of the ZK write step
+   */
   public boolean addControllerJobToZK(String jobId, Map<String, String> 
jobMetadata, String jobResourcePath) {
+    return addControllerJobToZK(jobId, jobMetadata, jobResourcePath, prev -> 
true);
+  }
+
+  /**
+   * Adds a new job metadata for controller job like table rebalance or reload 
into ZK
+   * @param jobId job's UUID
+   * @param jobMetadata the job metadata
+   * @param jobResourcePath zkPath to add the new job metadata
+   * @param prevJobMetadataChecker to check the previous job metadata before 
adding new one
+   * @return boolean representing success / failure of the ZK write step
+   */
+  public boolean addControllerJobToZK(String jobId, Map<String, String> 
jobMetadata, String jobResourcePath,

Review Comment:
   Suggest changing `jobResourcePath` to `ControllerJobType`



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1992,19 +1994,24 @@ public Map<String, Map<String, String>> 
getAllJobsForTable(String tableNameWithT
     for (ControllerJobType jobType : jobTypes) {
       String jobsResourcePath = 
ZKMetadataProvider.constructPropertyStorePathForControllerJob(jobType);
       try {
+        if (!_propertyStore.exists(jobsResourcePath, -1)) {

Review Comment:
   (minor) Use `AccessOption.PERSISTENT`, same for `get()`



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceChecker.java:
##########
@@ -0,0 +1,330 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.rebalance;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.controllerjob.ControllerJobType;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.LeadControllerManager;
+import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import 
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Periodic task to check whether a user triggered rebalance job is completed 
or not, and retry if failed. The retry
+ * job is started with the same rebalance configs provided by the user and 
does best effort to stop the other jobs
+ * for the same table. This task can be configured to just check failures and 
report metrics, and not to do retry.
+ */
+public class RebalanceChecker extends ControllerPeriodicTask<Void> {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(RebalanceChecker.class);
+  private static final double RETRY_DELAY_SCALE_FACTOR = 2.0;
+  private final ExecutorService _executorService;
+
+  public RebalanceChecker(PinotHelixResourceManager pinotHelixResourceManager,
+      LeadControllerManager leadControllerManager, ControllerConf config, 
ControllerMetrics controllerMetrics,
+      ExecutorService executorService) {
+    super(RebalanceChecker.class.getSimpleName(), 
config.getRebalanceCheckerFrequencyInSeconds(),
+        config.getRebalanceCheckerInitialDelayInSeconds(), 
pinotHelixResourceManager, leadControllerManager,
+        controllerMetrics);
+    _executorService = executorService;
+  }
+
+  protected void processTables(List<String> tableNamesWithType, Properties 
periodicTaskProperties) {
+    int numTables = tableNamesWithType.size();
+    LOGGER.info("Processing {} tables in task: {}", numTables, _taskName);
+    int numTablesProcessed = 0;
+    for (String tableNameWithType : tableNamesWithType) {
+      if (!isStarted()) {
+        LOGGER.info("Task: {} is stopped, early terminate the task", 
_taskName);
+        break;
+      }
+      try {
+        processTable(tableNameWithType);

Review Comment:
   We are still doing per-table check here, which means if there are 1000 
tables, we will read table config and job metadata 1000 times. Instead, we 
should read job metadata only once, and gather the tables needed to be retried, 
then schedule the retries



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -447,6 +467,9 @@ public RebalanceResult rebalance(TableConfig tableConfig, 
RebalanceConfig rebala
 
       // Check version and update IdealState
       try {
+        if (_tableRebalanceObserver.isAborted()) {
+          throw new RuntimeException(String.format("Rebalance for table: %s 
has been aborted", tableNameWithType));

Review Comment:
   Will this cause `RebalanceResult` to be `UNKNOWN_ERROR`? We should make it 
explicit that it is cancelled



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2063,28 +2070,76 @@ public boolean addNewReloadAllSegmentsJob(String 
tableNameWithType, String jobId
         
ZKMetadataProvider.constructPropertyStorePathForControllerJob(ControllerJobType.RELOAD_SEGMENT));
   }
 
+  /**
+   * Adds a new job metadata for controller job like table rebalance or reload 
into ZK
+   * @param jobId job's UUID
+   * @param jobMetadata the job metadata
+   * @param jobResourcePath zkPath to add the new job metadata
+   * @return boolean representing success / failure of the ZK write step
+   */
   public boolean addControllerJobToZK(String jobId, Map<String, String> 
jobMetadata, String jobResourcePath) {
+    return addControllerJobToZK(jobId, jobMetadata, jobResourcePath, prev -> 
true);
+  }
+
+  /**
+   * Adds a new job metadata for controller job like table rebalance or reload 
into ZK
+   * @param jobId job's UUID
+   * @param jobMetadata the job metadata
+   * @param jobResourcePath zkPath to add the new job metadata
+   * @param prevJobMetadataChecker to check the previous job metadata before 
adding new one
+   * @return boolean representing success / failure of the ZK write step
+   */
+  public boolean addControllerJobToZK(String jobId, Map<String, String> 
jobMetadata, String jobResourcePath,
+      Predicate<Map<String, String>> prevJobMetadataChecker) {
     
Preconditions.checkState(jobMetadata.get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS)
 != null,
         "Submission Time in JobMetadata record not set. Cannot expire these 
records");
     Stat stat = new Stat();
-    ZNRecord tableJobsZnRecord = _propertyStore.get(jobResourcePath, stat, 
AccessOption.PERSISTENT);
-    if (tableJobsZnRecord != null) {
-      Map<String, Map<String, String>> tasks = 
tableJobsZnRecord.getMapFields();
-      tasks.put(jobId, jobMetadata);
-      if (tasks.size() > 
CommonConstants.ControllerJob.MAXIMUM_CONTROLLER_JOBS_IN_ZK) {
-        tasks = tasks.entrySet().stream().sorted((v1, v2) -> Long.compare(
+    ZNRecord jobsZnRecord = _propertyStore.get(jobResourcePath, stat, 
AccessOption.PERSISTENT);
+    if (jobsZnRecord != null) {
+      Map<String, Map<String, String>> jobMetadataMap = 
jobsZnRecord.getMapFields();
+      Map<String, String> prevJobMetadata = jobMetadataMap.get(jobId);
+      if (!prevJobMetadataChecker.test(prevJobMetadata)) {
+        return false;
+      }
+      jobMetadataMap.put(jobId, jobMetadata);
+      if (jobMetadataMap.size() > 
CommonConstants.ControllerJob.MAXIMUM_CONTROLLER_JOBS_IN_ZK) {
+        jobMetadataMap = jobMetadataMap.entrySet().stream().sorted((v1, v2) -> 
Long.compare(
                 
Long.parseLong(v2.getValue().get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS)),
                 
Long.parseLong(v1.getValue().get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS))))
             .collect(Collectors.toList()).subList(0, 
CommonConstants.ControllerJob.MAXIMUM_CONTROLLER_JOBS_IN_ZK)
             .stream().collect(Collectors.toMap(Map.Entry::getKey, 
Map.Entry::getValue));
       }
-      tableJobsZnRecord.setMapFields(tasks);
-      return _propertyStore.set(jobResourcePath, tableJobsZnRecord, 
stat.getVersion(), AccessOption.PERSISTENT);
+      jobsZnRecord.setMapFields(jobMetadataMap);
+      return _propertyStore.set(jobResourcePath, jobsZnRecord, 
stat.getVersion(), AccessOption.PERSISTENT);
     } else {
-      tableJobsZnRecord = new ZNRecord(jobResourcePath);
-      tableJobsZnRecord.setMapField(jobId, jobMetadata);
-      return _propertyStore.set(jobResourcePath, tableJobsZnRecord, 
AccessOption.PERSISTENT);
+      jobsZnRecord = new ZNRecord(jobResourcePath);
+      jobsZnRecord.setMapField(jobId, jobMetadata);
+      return _propertyStore.set(jobResourcePath, jobsZnRecord, 
AccessOption.PERSISTENT);
+    }
+  }
+
+  /**
+   * Update existing job metadata belong to the table
+   * @param tableNameWithType whose job metadata to be updated
+   * @param jobResourcePath zkPath to look for the job metadata for the table
+   * @param updater to modify the job metadata in place
+   * @return boolean representing success / failure of the ZK write step
+   */
+  public boolean updateAllJobsForTable(String tableNameWithType, String 
jobResourcePath,

Review Comment:
   Change `jobResourcePath` to `ControllerJobType`, also suggest making the 
updater on the whole `jobMetadataMap` which is more general. Currently it 
relies on the job id to be encoded in the value
   
   (minor) Suggest renaming the method to `updateJobsForTable` since it only 
touches one job type



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceAttemptContext.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.rebalance;
+
+import com.google.common.annotations.VisibleForTesting;
+
+
+/**
+ * Track the job configs and attempt numbers as part of the job ZK metadata 
for future retry.
+ */
+public class TableRebalanceAttemptContext {

Review Comment:
   Suggest just naming it `TableRebalanceContext`. Retry is just one usage of 
this context, and we may add more usage in the future (e.g. `jobId` is not only 
used in retry)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to