maytasm commented on pull request #11025: URL: https://github.com/apache/druid/pull/11025#issuecomment-812221416
> Perhaps the mechanism in #10676 could be adapted to be used to just block until segments are loaded before marking them unused? (I haven't looked closely at either of these PRs myself) Currently, the functionality of marking existing segment as unused is done transactionally with publishing segments. This means that the job fails if either marking unused or publishing fail (we don't mark unused if publish fails and don't publish if marking unused fails). I think if we go with https://github.com/apache/druid/pull/10676 we have to make marking segment unused as a best-effort. We would have to publish the new segments first, then wait for the loading of new segment (by mechanism in https://github.com/apache/druid/pull/10676), then only if that succeed, we can mark the existing segments as unused. However, the mechanism in https://github.com/apache/druid/pull/10676 can timeout. This means that we cannot mark existing segments as unused since it is not confirmed if new segments is loaded yet or not. In those cases, task would still succeed and new segments would be punished but marking existing segments unused would fails. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
