Re: Does StreamingSymmetricHashJoinExec work with watermark? I don't think so

2019-11-14 Thread Jungtaek Lim
Jacek, would you mind if I ask for the query to reproduce? Not sure I get you without having the example of "not working". Thanks, Jungtaek Lim (HeartSaVioR) On Tue, Nov 12, 2019 at 12:04 AM Jacek Laskowski wrote: > Hi, > > I think watermark does not work for StreamingSymmetricHashJoinExec

Does StreamingSymmetricHashJoinExec work with watermark? I don't think so

2019-11-11 Thread Jacek Laskowski
Hi, I think watermark does not work for StreamingSymmetricHashJoinExec because of the following: 1. leftKeys and rightKeys have no spark.watermarkDelayMs metadata entry at planning [1] 2. Since the left and right keys had no watermark delay at planning the code [2] won't find it at execution Is