bgl-prahlad commented on issue #7275:
URL: https://github.com/apache/pinot/issues/7275#issuecomment-2451789832
hi @npawar I would like to use this issue as an opportunity to get started
with simple contributions. I would like some guidance in getting started. I can
see that this issue is only solving a small part of the larger problem of
migrating data-stores, and in that spirit I would like to address the usecase
of migrating from a local store to a new location supported by PinotFS.
I would like to create a new admin-command "SyncSegmentDeepStoreLocations"
which would identify all segments in the cluster/controller whose location does
not match the "controller.data.dir" configuration of the controller, and
migrate these segments to the correct location - and update the ZK metadata.
My initial thought for the workflow for migration is:
- Pre-condition: Pinot Controller is operating with local store based
configuration.
- Steps:
- Update the controller configuration to update the deep-store location
(controller.data.dir - eg see [Use S3 as Deep Storage for Pinot | Apache Pinot
Docs](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot))
- Run the “SyncSegmentDeepStoreLocations” admin command
- Post-condition: If the SyncSegmentDeepStoreLocations succeeds, then all
segments are migrated from their older location to the newly configured
deep-store location (controller.data.dir)
The "SyncSegmentDeepStoreLocations" command itself would:
- Get the currently configured deep-store location
- Iterate through all the tables and all segments (look at how
ShowClusterInfoCommand does this)
- Identify segments whose location does not match
- Figure out the targetURI for the segment (follow logic for the
CreateSegment command)
- Instantiate PinotFS instance to copy file to the target location.
Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]