ArafatKhan2198 commented on code in PR #11843:
URL: https://github.com/apache/ozone/pull/11843#discussion_r4616639048


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java:
##########
@@ -1772,6 +1774,49 @@ public List<Long> suppressContainers(List<Long> 
containerIds, boolean suppress)
     return failedContainerIDs;
   }
 
+  @Override
+  public String submitContainerIdExport(ContainerID startContainerId, 
HddsProtos.LifeCycleState lifeCycleState,
+      ContainerHealthState healthState, long maxRows, int batchSize, int 
partSize) throws IOException {
+    getScm().checkAdminAccess(getRemoteUser(), false);
+    final Map<String, String> auditMap = Maps.newHashMap();
+    auditMap.put("startContainerID", String.valueOf(startContainerId));
+    auditMap.put("lifeCycleState", String.valueOf(lifeCycleState));
+    auditMap.put("healthState", String.valueOf(healthState));
+    auditMap.put("maxRows", String.valueOf(maxRows));
+    auditMap.put("batchSize", String.valueOf(batchSize));
+    auditMap.put("partSize", String.valueOf(partSize));
+    try {
+      final String jobId = scm.getContainerExportManager().submitJob(

Review Comment:
   If this SCM stops being the leader just before it queues the job, the client 
gives up. Could we return the usual leader-change error here so the client can 
safely try the new leader?



##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java:
##########
@@ -1357,6 +1361,35 @@ public List<Long> suppressContainers(List<Long> 
containerIds, boolean suppress)
     return response.getFailedContainerIDsList();
   }
 
+  @Override
+  public String submitContainerIdExport(ContainerID startContainerId, 
HddsProtos.LifeCycleState lifeCycleState,
+      ContainerHealthState healthState, long maxRows, int batchSize, int 
partSize) throws IOException {
+    final SubmitContainerIdExportRequestProto.Builder request = 
SubmitContainerIdExportRequestProto.newBuilder()
+        .setMaxRows(maxRows)
+        .setBatchSize(batchSize)
+        .setPartSize(partSize);
+    if (startContainerId != null) {
+      request.setStartContainerID(startContainerId.getProtobuf());
+    }
+    if (lifeCycleState != null) {
+      request.setLifecycleState(lifeCycleState);
+    }
+    if (healthState != null) {
+      request.setHealthState(healthState.name());
+    }
+    return submitRequest(Type.SubmitContainerIdExport, builder -> 
builder.setSubmitContainerIdExportRequest(request))

Review Comment:
   If SCM starts the export but the client loses the reply, the client can 
resend the request. If the first job has already finished, this starts a second 
export. Could we reuse a request ID so SCM can return the original job ID?



##########
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto:
##########
@@ -760,6 +766,43 @@ message SuppressContainerResponseProto {
   repeated int64 failedContainerIDs = 1;
 }
 
+message SubmitContainerIdExportRequestProto {
+  optional ContainerID startContainerID = 1;
+  optional LifeCycleState lifecycleState = 2;
+  optional string healthState = 3;
+  optional int64 maxRows = 4;
+  optional int32 batchSize = 5;
+  optional int32 partSize = 6;

Review Comment:
   With partSize=1 and no row limit, a million matching IDs means a million 
temporary files on the SCM disk. Could we cap the number of parts per export?



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java:
##########
@@ -1772,6 +1774,49 @@ public List<Long> suppressContainers(List<Long> 
containerIds, boolean suppress)
     return failedContainerIDs;
   }
 
+  @Override
+  public String submitContainerIdExport(ContainerID startContainerId, 
HddsProtos.LifeCycleState lifeCycleState,
+      ContainerHealthState healthState, long maxRows, int batchSize, int 
partSize) throws IOException {
+    getScm().checkAdminAccess(getRemoteUser(), false);
+    final Map<String, String> auditMap = Maps.newHashMap();
+    auditMap.put("startContainerID", String.valueOf(startContainerId));
+    auditMap.put("lifeCycleState", String.valueOf(lifeCycleState));
+    auditMap.put("healthState", String.valueOf(healthState));
+    auditMap.put("maxRows", String.valueOf(maxRows));
+    auditMap.put("batchSize", String.valueOf(batchSize));
+    auditMap.put("partSize", String.valueOf(partSize));
+    try {
+      final String jobId = scm.getContainerExportManager().submitJob(
+          startContainerId, lifeCycleState, healthState, maxRows, batchSize, 
partSize).getId().toString();
+      auditMap.put("jobId", jobId);
+      
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(SCMAction.EXPORT_CONTAINER_IDS,
 auditMap));
+      return jobId;
+    } catch (IllegalArgumentException ex) {
+      
AUDIT.logWriteFailure(buildAuditMessageForFailure(SCMAction.EXPORT_CONTAINER_IDS,
 auditMap, ex));
+      throw new NonRetriableException(ex.getMessage());
+    } catch (IOException ex) {
+      
AUDIT.logWriteFailure(buildAuditMessageForFailure(SCMAction.EXPORT_CONTAINER_IDS,
 auditMap, ex));
+      throw ex;
+    }
+  }
+
+  @Override
+  public GetContainerIdExportStatusResponseProto 
getContainerIdExportStatus(String jobId) throws IOException {
+    getScm().checkAdminAccess(getRemoteUser(), true);
+    final Map<String, String> auditMap = Maps.newHashMap();
+    auditMap.put("jobId", jobId);
+    final GetContainerIdExportStatusResponseProto status =
+        scm.getContainerExportManager().getContainerExportStatus(jobId);
+    if (status == null) {
+      final IOException ex = new NonRetriableException("Container ID export 
job not found: " + jobId

Review Comment:
   After a leader change, the new SCM can say `job not found `because the old 
SCM holds the job status. Could we mention this in the error and add a test 
that submits on one leader and polls after failover?



##########
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto:
##########
@@ -760,6 +766,43 @@ message SuppressContainerResponseProto {
   repeated int64 failedContainerIDs = 1;
 }
 
+message SubmitContainerIdExportRequestProto {
+  optional ContainerID startContainerID = 1;
+  optional LifeCycleState lifecycleState = 2;
+  optional string healthState = 3;
+  optional int64 maxRows = 4;
+  optional int32 batchSize = 5;
+  optional int32 partSize = 6;
+}
+
+message SubmitContainerIdExportResponseProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusRequestProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusResponseProto {
+  enum ExecutionState {
+    RUNNING = 1;
+    SUCCEEDED = 2;
+    FAILED = 3;
+  }
+
+  optional string jobId = 1;
+  optional ExecutionState state = 2;
+  optional LifeCycleState lifecycleState = 3;
+  optional string healthState = 4;
+  optional int64 maxRows = 5;
+  optional int32 batchSize = 6;
+  optional int32 partSize = 7;
+  optional int64 idsWritten = 8;

Review Comment:
   idsWritten is only set once the job succeeds. Would it be a good idea to 
also report it while the job is running so the CLI can show whether a long 
export is making progress? If you think its unnecessary then we can leave it



##########
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto:
##########
@@ -760,6 +766,43 @@ message SuppressContainerResponseProto {
   repeated int64 failedContainerIDs = 1;
 }
 
+message SubmitContainerIdExportRequestProto {
+  optional ContainerID startContainerID = 1;
+  optional LifeCycleState lifecycleState = 2;
+  optional string healthState = 3;
+  optional int64 maxRows = 4;
+  optional int32 batchSize = 5;
+  optional int32 partSize = 6;
+}
+
+message SubmitContainerIdExportResponseProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusRequestProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusResponseProto {
+  enum ExecutionState {
+    RUNNING = 1;
+    SUCCEEDED = 2;
+    FAILED = 3;
+  }
+
+  optional string jobId = 1;
+  optional ExecutionState state = 2;
+  optional LifeCycleState lifecycleState = 3;
+  optional string healthState = 4;
+  optional int64 maxRows = 5;
+  optional int32 batchSize = 6;
+  optional int32 partSize = 7;
+  optional int64 idsWritten = 8;
+  optional int64 elapsedMs = 9;
+  optional string tarPath = 10;

Review Comment:
   tarPath tells me where the file is on disk. Could the response also include 
the SCM node ID or hostname so the CLI can tell me which machine to copy it 
from?



##########
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto:
##########
@@ -760,6 +766,43 @@ message SuppressContainerResponseProto {
   repeated int64 failedContainerIDs = 1;
 }
 
+message SubmitContainerIdExportRequestProto {
+  optional ContainerID startContainerID = 1;
+  optional LifeCycleState lifecycleState = 2;
+  optional string healthState = 3;
+  optional int64 maxRows = 4;
+  optional int32 batchSize = 5;
+  optional int32 partSize = 6;
+}
+
+message SubmitContainerIdExportResponseProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusRequestProto {
+  optional string jobId = 1;
+}
+
+message GetContainerIdExportStatusResponseProto {
+  enum ExecutionState {
+    RUNNING = 1;
+    SUCCEEDED = 2;
+    FAILED = 3;
+  }
+
+  optional string jobId = 1;
+  optional ExecutionState state = 2;
+  optional LifeCycleState lifecycleState = 3;

Review Comment:
   The submit request can start at a specific container ID, but status doesn’t 
show that value. Could we return the effective startContainerID so someone 
checking the job knows where the export started?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to