Jackie-Jiang commented on PR #11943:
URL: https://github.com/apache/pinot/pull/11943#issuecomment-1793362417

   > No, you want to count only one segment in the group that finishes 
together. That is because of the formula we use where we pay most attention to 
the past segments and less attention to the most recent segment size (R_next = 
R_prev * 0.75 + R_current * 0.25).
   > 
   > (It could be 0.9 and 0.1, not sure).
   > 
   > So, the most recent segment counted 8 times (in your example) will soil 
the "past" value to be incorrectly weighted as the present value.
   
   The past weight is 0.9 and the current weight is 0.1
   In regular case the ratio for old and new segment should be very close. My 
point is when data distribution changes, why do we want it to pick up the new 
pattern so slowly? IMO we want to quickly adjust to the recent data 
distribution for better prediction. With current algorithm, I can see it gives 
more stable ratio, but when data distribution changes, it will take ~10 rounds 
to adjust to the new ratio, which can take multiple days. For certain stream 
types such as Kinesis, the smallest partition might be fully consumed, which 
will cause it not be able to update the ratio. Given these drawbacks and extra 
overhead, I feel counting all segments is a better approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to