swaminathanmanish opened a new pull request, #10630:
URL: https://github.com/apache/pinot/pull/10630

   **Problem**
   Segment replacement has 2 phases (start, end). The start phase takes in the 
"**from**" and "**to**" list of segments, to create a lineage record that's 
used in the the end phase. The end phase expects all segments in the "**to**" 
to be available. However it can happen that only a subset of segments get 
created between the start and end phase. 
   **Usecase** - While ingesting batch of files via minion tasks, we use 
segment replacement to do a global atomic switch to avoid inconsistencies 
during ingestion. We start with a list of "**to**" segments, however can end up 
with a subset of the original "to" segments list (eg: files can have empty 
records). 
   
   Today the code has validations when segments in the original "to" list are 
not available, during the end phase of the protocol.
   
   **Solution**
   Allow the endReplaceSegments API take in the list of "**to**" segments that 
actually gets generated/created. 
   Validate that this list is a subset of the original to List.
   
   **Tests** 
   1) Tested for backwards compatibility 
(MergeRollupMinionClusterIntegrationTest)
   2) Unit tests
   3) Used Swagger to verify that the new parameter is taken by the API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to