Kostas, I'm pleased to see some concrete details in this FLIP.
I wonder if the current proposal goes far enough in the direction of recognizing the need some users may have for "batch" and "bounded streaming" to be treated differently. If I've understood it correctly, the section on scheduling allows me to choose STREAMING scheduling even if I have bounded sources. I like that approach, because it recognizes that even though I have bounded inputs, I don't necessarily want batch processing semantics. I think it makes sense to extend this idea to processing time support as well. My thinking is that sometimes in development and testing it's reasonable to run exactly the same job as in production, except with different sources and sinks. While it might be a reasonable default, I'm not convinced that switching a processing time streaming job to read from a bounded source should always cause it to fail. David On Wed, Aug 12, 2020 at 5:22 PM Kostas Kloudas <kklou...@apache.org> wrote: > Hi all, > > As described in FLIP-131 [1], we are aiming at deprecating the DataSet > API in favour of the DataStream API and the Table API. After this work > is done, the user will be able to write a program using the DataStream > API and this will execute efficiently on both bounded and unbounded > data. But before we reach this point, it is worth discussing and > agreeing on the semantics of some operations as we transition from the > streaming world to the batch one. > > This thread and the associated FLIP [2] aim at discussing these issues > as these topics are pretty important to users and can lead to > unpleasant surprises if we do not pay attention. > > Let's have a healthy discussion here and I will be updating the FLIP > accordingly. > > Cheers, > Kostas > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158871522 >
