One possible way is to introduce a delay operator. I am not 100% that it
will work, but it is worth trying. Introduce a dummy output port on the
operator that takes too long to setup. Connect it to an input port of a
delay operator. Connect delay operator output port to a dummy input port
of the upstream operator. In the upstream operator do not emit on the
actual output port till a tuple is received on the dummy input port. In
the downstream operator emit a tuple to the dummy output port in the
first beginWindow().
Thank you,
Vlad
On 4/21/17 12:36, Ganelin, Ilya wrote:
Yes, Amol – basically wanted to know how to sensibly handle the case
where there’s a big back-buffer to work through.
- Ilya Ganelin
id:image001.png@01D1F7A4.F3D42980
*From: *Amol Kekre <a...@datatorrent.com>
*Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
*Date: *Thursday, April 20, 2017 at 4:27 PM
*To: *"users@apex.apache.org" <users@apex.apache.org>
*Subject: *Re: Wait until setup completes prior to dag execution
Ilya,
Additionally that particular operator will not process data till setup
is done. As Pramod pointed out, there is no reason to wait. At worse
the buffer-servers will fill up. Are you worried about the operator
code need to ramp up in processing data and not get a deluge of it?
Thks
Amol
E:a...@datatorrent.com <mailto:e%3aa...@datatorrent.com> | M:
510-449-2606 | Twitter: @/amolhkekre/
www.datatorrent.com <http://www.datatorrent.com>
On Thu, Apr 20, 2017 at 4:19 PM, Pramod Immaneni
<pra...@datatorrent.com <mailto:pra...@datatorrent.com>> wrote:
Hi,
I am assuming it is an operator that is not the input operator
that has the lengthy setup function? Any specific reason you don't
want to let data in because things don't get committed (like
kafka offsets) till all operators process the data. If you are
worried about input operators are going to run-away there is back
pressure which will put breaks on when the buffers are full.
Thanks
On Thu, Apr 20, 2017 at 3:36 PM, Ganelin, Ilya
<ilya.gane...@capitalone.com <mailto:ilya.gane...@capitalone.com>>
wrote:
Is there any way to wait for setup to complete prior to
allowing data to flow through the DAG? I have an operator with
a lengthy setup function so I’m just wondering what the best
way to handle this is. My best solution at the moment is to
toggle start with an emission rate of zero and toggle it on
once the DAG is launched.
- Ilya Ganelin
------------------------------------------------------------------------
The information contained in this e-mail is confidential
and/or proprietary to Capital One and/or its affiliates and
may only be used solely in performance of work or services for
Capital One. The information transmitted herewith is intended
only for use by the individual or entity to which it is
addressed. If the reader of this message is not the intended
recipient, you are hereby notified that any review,
retransmission, dissemination, distribution, copying or other
use of, or taking of any action in reliance upon this
information is strictly prohibited. If you have received this
communication in error, please contact the sender and delete
the material from your computer.
------------------------------------------------------------------------
The information contained in this e-mail is confidential and/or
proprietary to Capital One and/or its affiliates and may only be used
solely in performance of work or services for Capital One. The
information transmitted herewith is intended only for use by the
individual or entity to which it is addressed. If the reader of this
message is not the intended recipient, you are hereby notified that
any review, retransmission, dissemination, distribution, copying or
other use of, or taking of any action in reliance upon this
information is strictly prohibited. If you have received this
communication in error, please contact the sender and delete the
material from your computer.
