leesf commented on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-864494981
> > does seem like a bandaid :/, may be necessary for now. any idea why that specific value works? > > @vinothchandar Not completely sure about my theory because the test is flaky, but here's what I think. If you look at the [`filterPendingCompactionTimeline`](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java#L130) you'll notice that it counts even the completed compactions. It should only count the inflight/requested ones. Now, we have a [maxPendingCompactions](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L283) limit in delta streamer. Delta sync will not happen unless pending compactions is less than that limit. So, during the test it could be possible that some of the remaining compactions are waiting to be run but they are not in pending compaction timeline yet. So, increasing the timeout helped. And why that specific number? Well, I just started with 2x timeout (480s) and kept decreasing exponentially to 360, 300, 270. I found that 360 was the minimum timeout for which all runs passed. @codope please see the comment here https://github.com/apache/hudi/pull/3109/files#r654870791 cc @vinothchandar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
