Jackie-Jiang commented on code in PR #20160:
URL: https://github.com/apache/pinot/pull/20160#discussion_r4607819396
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentPushUtils.java:
##########
@@ -712,6 +718,46 @@ public static Map<String, String>
getSegmentUriToTarPathMap(URI outputDirURI, Pu
return segmentUriToTarPathMap;
}
+ /// Segments stored in the deep store, e.g. committed REALTIME segments,
have no archive extension. The compression
+ /// format is detected from the content when the segment is decompressed.
+ private static boolean isSegmentFileWithoutExtension(String path) {
+ String fileName = new File(path).getName();
+ // Skip marker files such as _SUCCESS
+ return fileName.indexOf('.') < 0 && !fileName.startsWith("_");
+ }
+
+ /// Segments without archive extension are usually the segments in the deep
store of a table, which must not be
+ /// deleted after pushing.
+ @VisibleForTesting
+ static void checkCleanUpOutputDir(SegmentGenerationJobSpec spec,
List<String> segmentPaths) {
+ if (spec.isCleanUpOutputDir()) {
+ for (String segmentPath : segmentPaths) {
+
Preconditions.checkArgument(!isSegmentFileWithoutExtension(URI.create(segmentPath).getPath()),
+ "cleanUpOutputDir is not supported for segment without archive
extension: %s", segmentPath);
+ }
+ }
+ }
+
+ /// Lists the files under `outputDirURI` recursively for segment push.
Directories are skipped because segment files
+ /// without archive extension are also selected for push.
+ public static String[] listFiles(PinotFS outputDirFS, URI outputDirURI)
+ throws IOException {
+ try {
+ return outputDirFS.listFilesWithMetadata(outputDirURI, true).stream()
+ .filter(fileMetadata -> !fileMetadata.isDirectory())
+ .map(FileMetadata::getFilePath)
+ .toArray(String[]::new);
+ } catch (UnsupportedOperationException e) {
+ List<String> files = new ArrayList<>();
+ for (String file : outputDirFS.listFiles(outputDirURI, true)) {
+ if (!isSegmentFileWithoutExtension(file) ||
!outputDirFS.isDirectory(URI.create(file))) {
Review Comment:
[P2] Normalize raw local paths before checking directories
When a filesystem does not implement `listFilesWithMetadata()`,
`listFiles()` can return raw absolute local paths, as `LocalPinotFS` does. An
extensionless entry such as `/tmp/my data/segments` then causes
`URI.create(file)` to throw before filename filtering, even if the job only
targets ordinary `.tar.gz` archives. Paths containing `#` or `%` can also be
misinterpreted.
Please apply the same absolute-local-path conversion used in
`getSegmentUriToTarPathMap()` (`new File(file).toURI()` for the `file` scheme)
before calling `isDirectory()`. The fallback test should retain real
`LocalPinotFS.listFiles()` output and include a parent directory containing a
space; its current stub replaces every result with a `file:` URI and masks this
case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]