kkrugler commented on issue #7791: URL: https://github.com/apache/pinot/issues/7791#issuecomment-978403625
@richardstartin - yes, re aborting the download once the two files of interest (`metadata.properties` and `creation.meta`) have been extracted. This will still work even if these files aren't near the beginning of the file, or before the bigger data files. It will just not be as much of an optimization, but even in the worst case it will be faster than the current code (no creation of temp files on local disk). As to what happens with copying of files - it's not really the same as messing with transaction logs. Ops people (like the team for one of our clients) copy around segment files all the time, as they try out our different optimizations for how data is indexed, or manually roll back after there's an issue with the data being used to build new segments. Re where to put the metadata files - If the two metadata files are packaged as as currently expected by the controller, then they'll be `<blah>.tar.gz` files. In that case they really shouldn't be in the same directory as the actual segment tarballs, as segment pushes will try to push everything with that suffix to the controller. If you do a metadata push then it could rely on a naming convention to only process the metadata files. You could leave the files untarred, but now you have a bunch of `<name>-metadata.properties` and `<name>-creation.meta` files littering the segment directory. We have HDFS directories with 1200+ segments, so adding another 2400 of these tiny files doesn't feel clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
