Jackie-Jiang commented on a change in pull request #5221: Add a new server api
for download of segments.
URL: https://github.com/apache/incubator-pinot/pull/5221#discussion_r408346706
##########
File path:
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java
##########
@@ -175,4 +183,49 @@ public String getCrcMetadataForTable(
}
}
}
+
+ // TODO Add access control similar to
PinotSegmentUploadDownloadRestletResource for segment download.
+ @GET
+ @Produces(MediaType.APPLICATION_OCTET_STREAM)
+ @Path("/segments/{tableNameWithType}/{segmentName}")
+ @ApiOperation(value = "Download a segment", notes = "Download a segment in
zipped tar format")
+ public Response downloadSegment(
+ @ApiParam(value = "Name of the table with type REALTIME OR OFFLINE",
required = true, example = "myTable_OFFLINE") @PathParam("tableNameWithType")
String tableNameWithType,
+ @ApiParam(value = "Name of the segment", required = true)
@PathParam("segmentName") @Encoded String segmentName,
+ @Context HttpHeaders httpHeaders)
+ throws Exception {
+ LOGGER.info("Received a request to download segment {} for table {}",
segmentName, tableNameWithType);
+ TableDataManager tableDataManager =
checkGetTableDataManager(tableNameWithType);
+ SegmentDataManager segmentDataManager =
tableDataManager.acquireSegment(segmentName);
+ if (segmentDataManager == null) {
+ throw new WebApplicationException(
+ String.format("Table %s segment %s does not exist",
tableNameWithType, segmentName),
+ Response.Status.NOT_FOUND);
+ }
+ try {
+ String tableDir = tableDataManager.getTableDataDir().getAbsolutePath();
+ // TODO Limit the number of concurrent downloads of segments because
compression is an expensive operation.
+ // Store the tar.gz segment file in the server's segmentTarDir folder
with a unique file name.
+ // Note that two clients asking the same segment file will result in the
same tar.gz files being created twice.
+ // Will revisit for optimization if performance becomes an issue.
+ String tarFilePath =
TarGzCompressionUtils.createTarGzOfDirectory(tableDir + File.separator +
segmentName,
+ serverInstance.getInstanceDataManager().getSegmentFileDirectory() +
File.separator + segmentName + "-"
Review comment:
Make a separate directory (e.g. `peerDownloadTemp`) under segment file
directory to hold all these files. Also good to keep the tableName for
debugging purpose? You can refer to the name in SegmentFetcherAndLoader line 183
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
