AlexanderKM commented on PR #19337: URL: https://github.com/apache/pinot/pull/19337#issuecomment-5415423277
@jadami10 We have started implementing our own custom Purge/Delete minion tasks that take input and search through segments for rows to be deleted, and the minions then upload the purged segment back to the Controller. At a very high volume, we thought it would be optimal to start using the METADATA push option (minions upload their new purged segment to s3, then tell the controller to s3 copy that segment to its final destination), instead of the minion directly pushing the whole segment file from minion->controller. We found that the direct upload to controller, at a high volume, was actually filling up the controller disk at high rates, just from Jersey writing the file to disk, even temporarily. We also assumed the direct segment upload was unncessary when the METADATA push option exists. For unrelated reasons, our pinot controllers have been OOMing a lot, and our purge/delete tasks have also been failing with errors around "unable to to fetch segment to local disk", meaning the segment is missing in S3. Upon further investigation of why segments were missing in the s3 deep-store, it became pretty clear that the purge/delete task had finished processing in the minion, the minion uploaded its new segment to our staging directory in S3, and the controller had deleted the segment from the deep-store in s3, but failed to copy from the staging dir to the deep-store again to complete the full upload/copy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
