havefunatcode commented on PR #20160:
URL: https://github.com/apache/pinot/pull/20160#issuecomment-6895630229

   @Jackie-Jiang Thanks, that's a lot simpler. Changed it in e62baaa6cf.
   
   Files with no dot in the name are now selected along with the archive 
extensions, for all push types, and the flag, the URI-only overloads and the 
tmp file check are removed. Only the file name is checked, so dots in parent 
dirs or the host don't matter.
   
   For directories, all push runners now list the output dir through 
`SegmentPushUtils.listFiles()`, which drops them. If the `PinotFS` doesn't 
implement `listFilesWithMetadata()` it falls back to `listFiles()` plus 
`isDirectory()` for names without a dot. Names starting with `_` like 
`_SUCCESS` are skipped.
   
   `pushSegments()` and `getSegmentsToReplace()` use `getSegmentName()` now, so 
tar push and consistent push work with these names too. Metadata push already 
did, I added a test with gzip, LZ4 and zstd content in an extensionless file.
   
   @xiangfu0's fail fast is still there: with `cleanUpOutputDir`, tar and URI 
push throw before pushing anything if a segment has no archive extension.
   
   Note that a non-segment file without a dot under `outputDirURI` (e.g. 
`README`) is now picked up and fails the push. `pushFileNamePattern` can 
exclude it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to