bgl-prahlad commented on issue #7275:
URL: https://github.com/apache/pinot/issues/7275#issuecomment-2451789832

   hi @npawar I would like to use this issue as an opportunity to get started 
with simple contributions. I would like some guidance in getting started. I can 
see that this issue is only solving a small part of the larger problem of 
migrating data-stores, and in that spirit I would like to address the usecase 
of migrating from a local store to a new location supported by PinotFS.
   
   I would like to create a new admin-command "SyncSegmentDeepStoreLocations" 
which would identify all segments in the cluster/controller whose location does 
not match the "controller.data.dir" configuration of the controller, and 
migrate these segments to the correct location - and update the ZK metadata.
   
   My initial thought for the workflow for migration is:
   - Pre-condition: Pinot Controller is operating with local store based 
configuration.
   - Steps: 
     - Update the controller configuration to update the deep-store location 
(controller.data.dir - eg see [Use S3 as Deep Storage for Pinot | Apache Pinot 
Docs](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot))
     -  Run the “SyncSegmentDeepStoreLocations” admin command
   - Post-condition: If the SyncSegmentDeepStoreLocations succeeds, then all 
segments are migrated from their older location to the newly configured 
deep-store location (controller.data.dir)
   
   The "SyncSegmentDeepStoreLocations" command itself would:
   - Get the currently configured deep-store location
   - Iterate through all the tables and all segments (look at how 
ShowClusterInfoCommand does this)
   - Identify segments whose location does not match
   - Figure out the targetURI for the segment (follow logic for the 
CreateSegment command)
   - Instantiate PinotFS instance to copy file to the target location.
   
   Does this make sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to