havefunatcode commented on code in PR #20160:
URL: https://github.com/apache/pinot/pull/20160#discussion_r4608088684
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentPushUtils.java:
##########
@@ -712,6 +718,46 @@ public static Map<String, String>
getSegmentUriToTarPathMap(URI outputDirURI, Pu
return segmentUriToTarPathMap;
}
+ /// Segments stored in the deep store, e.g. committed REALTIME segments,
have no archive extension. The compression
+ /// format is detected from the content when the segment is decompressed.
+ private static boolean isSegmentFileWithoutExtension(String path) {
+ String fileName = new File(path).getName();
+ // Skip marker files such as _SUCCESS
+ return fileName.indexOf('.') < 0 && !fileName.startsWith("_");
+ }
+
+ /// Segments without archive extension are usually the segments in the deep
store of a table, which must not be
+ /// deleted after pushing.
+ @VisibleForTesting
+ static void checkCleanUpOutputDir(SegmentGenerationJobSpec spec,
List<String> segmentPaths) {
+ if (spec.isCleanUpOutputDir()) {
+ for (String segmentPath : segmentPaths) {
+
Preconditions.checkArgument(!isSegmentFileWithoutExtension(URI.create(segmentPath).getPath()),
+ "cleanUpOutputDir is not supported for segment without archive
extension: %s", segmentPath);
+ }
+ }
+ }
+
+ /// Lists the files under `outputDirURI` recursively for segment push.
Directories are skipped because segment files
+ /// without archive extension are also selected for push.
+ public static String[] listFiles(PinotFS outputDirFS, URI outputDirURI)
+ throws IOException {
+ try {
+ return outputDirFS.listFilesWithMetadata(outputDirURI, true).stream()
+ .filter(fileMetadata -> !fileMetadata.isDirectory())
+ .map(FileMetadata::getFilePath)
+ .toArray(String[]::new);
+ } catch (UnsupportedOperationException e) {
+ List<String> files = new ArrayList<>();
+ for (String file : outputDirFS.listFiles(outputDirURI, true)) {
+ if (!isSegmentFileWithoutExtension(file) ||
!outputDirFS.isDirectory(URI.create(file))) {
Review Comment:
Fixed in 8d4d795842. The conversion from `getSegmentUriToTarPathMap()` moved
into `toFileURI()`, which both places use. The fallback test keeps the real
`LocalPinotFS.listFiles()` output and now lists a dir named `my data #1`.
Without the fix it throws `Illegal character in path`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]