devmadhuu commented on code in PR #9994:
URL: https://github.com/apache/ozone/pull/9994#discussion_r3039011628
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java:
##########
@@ -450,6 +458,65 @@ private Response getUnhealthyContainersFromSchema(
return Response.ok(response).build();
}
+ /**
+ * Export all unhealthy containers into a CSV file by streaming the results
directly
+ * from the database without holding them in the JVM heap.
+ *
+ * @param state The container state to filter by, or null for all.
+ * @param limit The maximum number of records to return, 0 for unlimited.
+ * @return {@link Response} containing the CSV StreamingOutput.
+ */
+ @GET
+ @Path("/unhealthy/export")
+ @Produces("text/csv")
+ public Response exportUnhealthyContainers(
+ @QueryParam("state") String state,
+ @DefaultValue("0") @QueryParam(RECON_QUERY_LIMIT) int limit) {
Review Comment:
There should not be any limit for export. Export option is for full
container export for a given state. During real clusters, we have found that
for a given state, there can be upto 4M containers per state sometimes, so in
such unbalanced cluster, need export all for that state.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/ContainerHealthSchemaManager.java:
##########
@@ -381,6 +382,35 @@ public List<UnhealthyContainerRecord>
getUnhealthyContainers(
}
}
+ /**
+ * Returns a streaming cursor over unhealthy container records.
+ * Caller MUST close the cursor.
+ *
+ * @param state filter by state, or null for all states
+ * @param limit max records to return, 0 = unlimited
+ * @return Cursor returning UnhealthyContainersRecord
+ */
+ public Cursor<UnhealthyContainersRecord> getUnhealthyContainersCursor(
+ UnHealthyContainerStates state, int limit) {
Review Comment:
During test it was found that for 1M to fetch from table, it may take upto
~300 to 400 msec, but for 4M records for same state my take up to little
longer. Time not exactly proportional because indexes are defined in
UNHEALTHY_CONTAINER table, but we still cannot tell the time in deterministic
way as external factors like disk, network also involved. So I would suggest to
always implement the API in async way. User action will trigger the request and
UI will continue to poll till results are available and once available and all
exported, it will be downloaded in compressed format having all shards(parts).
Please check my other comment to mark the export done/completed.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java:
##########
@@ -450,6 +458,65 @@ private Response getUnhealthyContainersFromSchema(
return Response.ok(response).build();
}
+ /**
+ * Export all unhealthy containers into a CSV file by streaming the results
directly
+ * from the database without holding them in the JVM heap.
+ *
+ * @param state The container state to filter by, or null for all.
+ * @param limit The maximum number of records to return, 0 for unlimited.
+ * @return {@link Response} containing the CSV StreamingOutput.
+ */
+ @GET
+ @Path("/unhealthy/export")
+ @Produces("text/csv")
+ public Response exportUnhealthyContainers(
+ @QueryParam("state") String state,
+ @DefaultValue("0") @QueryParam(RECON_QUERY_LIMIT) int limit) {
+
+ ContainerSchemaDefinition.UnHealthyContainerStates internalState = null;
+ if (StringUtils.isNotEmpty(state)) {
+ try {
+ internalState =
ContainerSchemaDefinition.UnHealthyContainerStates.valueOf(state);
+ } catch (IllegalArgumentException e) {
+ throw new WebApplicationException(e, Response.Status.BAD_REQUEST);
+ }
+ }
+
+ final ContainerSchemaDefinition.UnHealthyContainerStates filterState =
internalState;
+
+ StreamingOutput stream = outputStream -> {
+ try (Cursor<UnhealthyContainersRecord> cursor =
+
containerHealthSchemaManager.getUnhealthyContainersCursor(filterState, limit)) {
+
+ PrintWriter writer = new PrintWriter(new
OutputStreamWriter(outputStream, StandardCharsets.UTF_8));
Review Comment:
One single csv may not be sufficient enough in large scale unbalanced
cluster. Our all solutions should be designed by keeping scalability in mind.
Break them into multiple shards (parts) with multiple csv files for a given
state. Let the below export logic work in async way, once all export of all
shards are completed, it should put some flag or indicator to detect the
completion. So that async polling request from UI to API will know and allow to
download the exported files in a compressed format.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java:
##########
@@ -253,6 +253,7 @@ public final class ReconServerConfigKeys {
"ozone.recon.scm.container.id.batch.size";
public static final long OZONE_RECON_SCM_CONTAINER_ID_BATCH_SIZE_DEFAULT =
1_000_000;
+
Review Comment:
This change is unnecessary. Revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
