Re: Could watermark could be took into consideration after the channel become active from idle at once?
We meet the same problem in our company. One stream always has data. The other stream is much smaller and can be idle. Once the smaller one becomes active, its data may be dropped in this case. 张静 [via Apache Flink User Mailing List archive.] < ml+s2336050n43873...@n4.nabble.com> 于2021年5月21日周五 上午10:24写道： > Hi, roman >Thanks for reply very much. >In our case, we see some data was dropped in window operator. We > found the root cause by adding a temporary metric about number of > aligned channels, found an active channel resumed from Idle was not > took into account for some time (not alway btw). We could walk around > by allow lateness on window operator, however I wonder does it make > sense to add a configuration to define different watermark aligned > behavior? > > Regards > JING > > Roman Khachatryan <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=43873&i=0>> 于2021年5月21日周五 > 上午1:05写道： > > > > > Hi, > > > > AFAIK, this behavior is not configurable. > > However, for this to happen the channel must consistently generate > > watermarks smaller than watermarks from ALL aligned channels (and its > > elements must have a smaller timestamp). I'm not sure how likely it > > is. Is it something you see in production? > > > > Regards, > > Roman > > > > On Thu, May 20, 2021 at 4:11 PM 张静 <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=43873&i=1>> wrote: > > > > > > Hi community, > > > Now after a channel become active from idle, the watermark on this > > > channel would not be took into account when align watermarks util it > > > generates a watermark equals to or bigger than last emitted watermark. > > > It makes sense because it could prevent the newly active task resumed > > > from idle dragging down the entire job. > > > > > > However, if the newly active task generate watermarks which are > > > smaller than but very closely to the last emitted one for a long time, > > > it could not be took into consideration in watermark alignment which > > > may lead to its data maybe dropped at the later window operator. > > > > > > Is there any way to solve this problem? Or could we add a > > > configuration to define different watermark aligned behavior？ > > > > > > Any suggestion is appreciated. Thanks a lot. > > > > > > Regards > > > JING > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-watermark-could-be-took-into-consideration-after-the-channel-become-active-from-idle-at-once-tp43851p43873.html > To start a new topic under Apache Flink User Mailing List archive., email > ml+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGl1amlhbmdhbmdwZW5nQGdtYWlsLmNvbXwxfC0xMTYwNzM3MjI=> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >
Re: Could watermark could be took into consideration after the channel become active from idle at once?
Hi, roman Thanks for reply very much. In our case, we see some data was dropped in window operator. We found the root cause by adding a temporary metric about number of aligned channels, found an active channel resumed from Idle was not took into account for some time (not alway btw). We could walk around by allow lateness on window operator, however I wonder does it make sense to add a configuration to define different watermark aligned behavior? Regards JING Roman Khachatryan 于2021年5月21日周五 上午1:05写道： > > Hi, > > AFAIK, this behavior is not configurable. > However, for this to happen the channel must consistently generate > watermarks smaller than watermarks from ALL aligned channels (and its > elements must have a smaller timestamp). I'm not sure how likely it > is. Is it something you see in production? > > Regards, > Roman > > On Thu, May 20, 2021 at 4:11 PM 张静 wrote: > > > > Hi community, > > Now after a channel become active from idle, the watermark on this > > channel would not be took into account when align watermarks util it > > generates a watermark equals to or bigger than last emitted watermark. > > It makes sense because it could prevent the newly active task resumed > > from idle dragging down the entire job. > > > > However, if the newly active task generate watermarks which are > > smaller than but very closely to the last emitted one for a long time, > > it could not be took into consideration in watermark alignment which > > may lead to its data maybe dropped at the later window operator. > > > > Is there any way to solve this problem? Or could we add a > > configuration to define different watermark aligned behavior？ > > > > Any suggestion is appreciated. Thanks a lot. > > > > Regards > > JING
Re: Could watermark could be took into consideration after the channel become active from idle at once?
Hi, AFAIK, this behavior is not configurable. However, for this to happen the channel must consistently generate watermarks smaller than watermarks from ALL aligned channels (and its elements must have a smaller timestamp). I'm not sure how likely it is. Is it something you see in production? Regards, Roman On Thu, May 20, 2021 at 4:11 PM 张静 wrote: > > Hi community, > Now after a channel become active from idle, the watermark on this > channel would not be took into account when align watermarks util it > generates a watermark equals to or bigger than last emitted watermark. > It makes sense because it could prevent the newly active task resumed > from idle dragging down the entire job. > > However, if the newly active task generate watermarks which are > smaller than but very closely to the last emitted one for a long time, > it could not be took into consideration in watermark alignment which > may lead to its data maybe dropped at the later window operator. > > Is there any way to solve this problem? Or could we add a > configuration to define different watermark aligned behavior？ > > Any suggestion is appreciated. Thanks a lot. > > Regards > JING
Could watermark could be took into consideration after the channel become active from idle at once?
Hi community, Now after a channel become active from idle, the watermark on this channel would not be took into account when align watermarks util it generates a watermark equals to or bigger than last emitted watermark. It makes sense because it could prevent the newly active task resumed from idle dragging down the entire job. However, if the newly active task generate watermarks which are smaller than but very closely to the last emitted one for a long time, it could not be took into consideration in watermark alignment which may lead to its data maybe dropped at the later window operator. Is there any way to solve this problem? Or could we add a configuration to define different watermark aligned behavior？ Any suggestion is appreciated. Thanks a lot. Regards JING
