Asquator commented on issue #45636: URL: https://github.com/apache/airflow/issues/45636#issuecomment-3071054359
Turns out we have to nest the windows to make the computation interleaved (otherwise we'll get counters wrong). I think the SQL job shouldn't be too heavy (despite having to compute a 4-level nested window function) because the order is the same, and the only thing we do is checking predicates. Should be a partial scan of the TI table the regular joins. The limit prevents us from running for too long. Anyway, I think the recommended max_tis value should rise (mb 128) with this approach, because we're targeting higher throughput with (presumably) more time in the critical section. Anyway, benchmarks will show. I'm working on the query -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
