Thanks! On Thu, Jan 2, 2020 at 9:45 PM Jingsong Li <jingsongl...@gmail.com> wrote:
> Yes, > > 1.9.2 or Coming soon 1.10 > > Best, > Jingsong Lee > > On Fri, Jan 3, 2020 at 12:43 AM RKandoji <rkand...@gmail.com> wrote: > >> Ok thanks, does it mean version 1.9.2 is what I need to use? >> >> On Wed, Jan 1, 2020 at 10:25 PM Jingsong Li <jingsongl...@gmail.com> >> wrote: >> >>> Blink planner was introduced in 1.9. We recommend use blink planner >>> after 1.9. >>> After some bug fix, I think the latest version of 1.9 is OK. The >>> production environment has also been set up in some places. >>> >>> Best, >>> Jingsong Lee >>> >>> On Wed, Jan 1, 2020 at 3:24 AM RKandoji <rkand...@gmail.com> wrote: >>> >>>> Thanks Jingsong and Kurt for more details. >>>> >>>> Yes, I'm planning to try out DeDuplication when I'm done upgrading to >>>> version 1.9. Hopefully deduplication is done by only one task and reused >>>> everywhere else. >>>> >>>> One more follow-up question, I see "For production use cases, we >>>> recommend the old planner that was present before Flink 1.9 for now." >>>> warning >>>> here https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/ >>>> This is actually the reason why started with version 1.8, could you >>>> please let me know your opinion about this? and do you think there is any >>>> production code running on version 1.9 >>>> >>>> Thanks, >>>> Reva >>>> >>>> >>>> >>>> >>>> On Mon, Dec 30, 2019 at 9:02 PM Kurt Young <ykt...@gmail.com> wrote: >>>> >>>>> BTW, you could also have a more efficient version of deduplicating >>>>> user table by using the topn feature [1]. >>>>> >>>>> Best, >>>>> Kurt >>>>> >>>>> [1] >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sql.html#top-n >>>>> >>>>> >>>>> On Tue, Dec 31, 2019 at 9:24 AM Jingsong Li <jingsongl...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi RKandoji, >>>>>> >>>>>> In theory, you don't need to do something. >>>>>> First, the optimizer will optimize by doing duplicate nodes. >>>>>> Second, after SQL optimization, if the optimized plan still has >>>>>> duplicate nodes, the planner will automatically reuse them. >>>>>> There are config options to control whether we should reuse plan, >>>>>> their default value is true. So you don't need modify them. >>>>>> - table.optimizer.reuse-sub-plan-enabled >>>>>> - table.optimizer.reuse-source-enabled >>>>>> >>>>>> Best, >>>>>> Jingsong Lee >>>>>> >>>>>> On Tue, Dec 31, 2019 at 6:29 AM RKandoji <rkand...@gmail.com> wrote: >>>>>> >>>>>>> Thanks Terry and Jingsong, >>>>>>> >>>>>>> Currently I'm on 1.8 version using Flink planner for stream >>>>>>> proessing, I'll switch to 1.9 version to try out blink planner. >>>>>>> >>>>>>> Could you please point me to any examples (Java preferred) using >>>>>>> SubplanReuser? >>>>>>> >>>>>>> Thanks, >>>>>>> RK >>>>>>> >>>>>>> On Sun, Dec 29, 2019 at 11:32 PM Jingsong Li <jingsongl...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi RKandoji, >>>>>>>> >>>>>>>> FYI: Blink-planner subplan reusing: [1] 1.9 available. >>>>>>>> >>>>>>>> Join Join >>>>>>>> / \ / \ >>>>>>>> Filter1 Filter2 Filter1 Filter2 >>>>>>>> | | => \ / >>>>>>>> Project1 Project2 Project1 >>>>>>>> | | | >>>>>>>> Scan1 Scan2 Scan1 >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/reuse/SubplanReuser.scala >>>>>>>> >>>>>>>> Best, >>>>>>>> Jingsong Lee >>>>>>>> >>>>>>>> On Mon, Dec 30, 2019 at 12:28 PM Terry Wang <zjuwa...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi RKandoji~ >>>>>>>>> >>>>>>>>> Could you provide more info about your poc environment? >>>>>>>>> Stream or batch? Flink planner or blink planner? >>>>>>>>> AFAIK, blink planner has done some optimization to deal such >>>>>>>>> duplicate task for one same query. You can have a try with blink >>>>>>>>> planner : >>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#create-a-tableenvironment >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Terry Wang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2019年12月30日 03:07，RKandoji <rkand...@gmail.com> 写道： >>>>>>>>> >>>>>>>>> Hi Team, >>>>>>>>> >>>>>>>>> I'm doing a POC with flink to understand if it's a good fit for my >>>>>>>>> use case. >>>>>>>>> >>>>>>>>> As part of the process, I need to filter duplicate items and >>>>>>>>> created below query to get only the latest records based on >>>>>>>>> timestamp. For >>>>>>>>> instance, I have "Users" table which may contain multiple messages >>>>>>>>> for the >>>>>>>>> same "userId". So I wrote below query to get only the latest message >>>>>>>>> for a >>>>>>>>> given "userId" >>>>>>>>> >>>>>>>>> Table uniqueUsers = tEnv.sqlQuery("SELECT * FROM Users WHERE >>>>>>>>> (userId, userTimestamp) IN (SELECT userId, MAX(userTimestamp) FROM >>>>>>>>> Users >>>>>>>>> GROUP BY userId)"); >>>>>>>>> >>>>>>>>> The above query works as expected and contains only the latest >>>>>>>>> users based on timestamp. >>>>>>>>> >>>>>>>>> The issue is when I use "uniqueUsers" table multiple times in a >>>>>>>>> JOIN operation, I see multiple tasks in the flink dashboard for the >>>>>>>>> same >>>>>>>>> query that is creating "uniqueUsers" table. It is simply creating as >>>>>>>>> many >>>>>>>>> tasks as many times I'm using the table. >>>>>>>>> >>>>>>>>> Below is the JOIN query. >>>>>>>>> tEnv.registerTable("uniqueUsersTbl", uniqueUsers); >>>>>>>>> Table joinOut = tEnv.sqlQuery("SELECT * FROM Car c >>>>>>>>> LEFT JOIN uniqueUsersTbl aa >>>>>>>>> ON c.userId = aa.userId >>>>>>>>> LEFT JOIN uniqueUsersTbl >>>>>>>>> ab ON c.ownerId = ab.userId >>>>>>>>> LEFT JOIN uniqueUsersTbl >>>>>>>>> ac ON c.sellerId = ac.userId >>>>>>>>> LEFT JOIN uniqueUsersTbl >>>>>>>>> ad ON c.buyerId = ad.userId"); >>>>>>>>> >>>>>>>>> Could someone please help me understand how I can avoid these >>>>>>>>> duplicate tasks? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> R Kandoji >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best, Jingsong Lee >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Best, Jingsong Lee >>>>>> >>>>> >>> >>> -- >>> Best, Jingsong Lee >>> >> > > -- > Best, Jingsong Lee >
