Thanks Jim.
> 1. For the testing, I'd call the tests "execution" tests rather than > "restore" tests. For streaming execution, restore tests have the compiled > plan and intermediate state; the tests verify that those can work together > and continue processing. Agree that we don't need to store and restore the intermediate state. So the most critical part is that the CompiledPlan for batch can be executed. 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's > words). Do we need "change detection tests"? I'm a little unsure if that > is presently happening in an automatic way for streaming operators. We might need to elaborate more on this, but the idea is that we need to make sure that compiled plans created by an older version of SQL Planner are executable on newer runtimes. 3. Can we remove old versions of batch operators eventually? Or do we > need to keep them forever like we would for streaming operators? > We could have deprecation paths for old operator nodes in some cases. It is a matter of the time window: what could be practical the "time distance" between query planner and flink runtime against which the query query can be resubmitted. Note, here we don't have continuous queries, so there is always an option to "re-plan" the original SQL query text into a newer version of the CompiledPlan. With this in mind, a time window of 1yr+ would allow deprecation of older batch exec nodes, though I don't see this as a frequent event. -Alexey On Mon, May 13, 2024 at 1:52 PM Jim Hughes <jhug...@confluent.io.invalid> wrote: > Hi Alexey, > > After some thought, I have a question about deprecations: > > 3. Can we remove old versions of batch operators eventually? Or do we > need to keep them forever like we would for streaming operators? > > Cheers, > > Jim > > On Thu, May 9, 2024 at 11:29 AM Jim Hughes <jhug...@confluent.io> wrote: > > > Hi Alexey, > > > > Overall, the FLIP looks good and makes sense to me. > > > > 1. For the testing, I'd call the tests "execution" tests rather than > > "restore" tests. For streaming execution, restore tests have the > compiled > > plan and intermediate state; the tests verify that those can work > together > > and continue processing. > > > > For batch execution, I think we just want that all existing compiled > plans > > can be executed in future versions. > > > > 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's > > words). Do we need "change detection tests"? I'm a little unsure if > that > > is presently happening in an automatic way for streaming operators. > > > > In RestoreTestBase, generateTestSetupFiles is disabled and has to be run > > manually when tests are being written. > > > > Cheers, > > > > Jim > > > > On Tue, May 7, 2024 at 5:11 AM Paul Lam <paullin3...@gmail.com> wrote: > > > >> Hi Alexey, > >> > >> Thanks a lot for bringing up the discussion. I’m big +1 for the FLIP. > >> > >> I suppose the goal doesn’t involve the interchangeability of json plans > >> between batch mode and streaming mode, right? > >> In other words, a json plan compiled in a batch program can’t be run in > >> streaming mode without a migration (which is not yet supported). > >> > >> Best, > >> Paul Lam > >> > >> > 2024年5月7日 14:38，Alexey Leonov-Vendrovskiy <vendrov...@gmail.com> 写道： > >> > > >> > Hi everyone, > >> > > >> > PTAL at the proposed FLIP-456: CompiledPlan support for Batch > Execution > >> > Mode. It is pretty self-describing. > >> > > >> > Any thoughts are welcome! > >> > > >> > Thanks, > >> > Alexey > >> > > >> > [1] > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode > >> > . > >> > >> >
