If the operators are not very heavy, perhaps thread local could work.. ~ Bhupesh
_______________________________________________________ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Sat, Apr 22, 2017 at 11:04 PM, Amol Kekre <a...@datatorrent.com> wrote: > > This could be solved by throttling ramp up. We have cases in production > that do this. If throttling at the input operator is there, then it is a > simple matter of communicating it via software. > > Lets assume the input operator(s) can be throttled from 0% to 100%. They > all they need to know is the throttle number. Since this comes from > downstream, you can figure out ways of communicating back. Some ideas (just > brainstorming) > 1. Look at commit window to know all operators are past setup (? Can an > operator know about commit window?) > 2. Communicate via other mechanism -> Write to a file (setup done, ramp up > now) > 3. Use statListner > 4. Vlad' solution > etc. > > I think #3 may work well. Additionally it can be tuned to also help > resolve in run-time "how much can the input ingestion operator at". Via #3 > this can be dynamic, and additionally the dag remains the same, it just > fine-tunes live. #4 can also do this by passing back meta data on the > feedback port. > > Thks > Amol > > > > E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* > > www.datatorrent.com > > > On Fri, Apr 21, 2017 at 7:15 PM, Vlad Rozov <v.ro...@datatorrent.com> > wrote: > >> One possible way is to introduce a delay operator. I am not 100% that it >> will work, but it is worth trying. Introduce a dummy output port on the >> operator that takes too long to setup. Connect it to an input port of a >> delay operator. Connect delay operator output port to a dummy input port of >> the upstream operator. In the upstream operator do not emit on the actual >> output port till a tuple is received on the dummy input port. In the >> downstream operator emit a tuple to the dummy output port in the first >> beginWindow(). >> >> Thank you, >> >> Vlad >> >> On 4/21/17 12:36, Ganelin, Ilya wrote: >> >> Yes, Amol – basically wanted to know how to sensibly handle the case >> where there’s a big back-buffer to work through. >> >> >> >> - Ilya Ganelin >> >> [image: id:image001.png@01D1F7A4.F3D42980] >> >> >> >> *From: *Amol Kekre <a...@datatorrent.com> <a...@datatorrent.com> >> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org> >> <users@apex.apache.org> <users@apex.apache.org> >> *Date: *Thursday, April 20, 2017 at 4:27 PM >> *To: *"users@apex.apache.org" <users@apex.apache.org> >> <users@apex.apache.org> <users@apex.apache.org> >> *Subject: *Re: Wait until setup completes prior to dag execution >> >> >> >> >> >> Ilya, >> >> Additionally that particular operator will not process data till setup is >> done. As Pramod pointed out, there is no reason to wait. At worse the >> buffer-servers will fill up. Are you worried about the operator code need >> to ramp up in processing data and not get a deluge of it? >> >> >> >> Thks >> >> Amol >> >> >> >> >> >> >> E:a...@datatorrent.com | M: 510-449-2606 <(510)%20449-2606> | Twitter: @ >> *amolhkekre* >> >> www.datatorrent.com >> >> >> >> On Thu, Apr 20, 2017 at 4:19 PM, Pramod Immaneni <pra...@datatorrent.com> >> wrote: >> >> Hi, >> >> >> >> I am assuming it is an operator that is not the input operator that has >> the lengthy setup function? Any specific reason you don't want to let data >> in because things don't get committed (like kafka offsets) till all >> operators process the data. If you are worried about input operators are >> going to run-away there is back pressure which will put breaks on when the >> buffers are full. >> >> >> >> Thanks >> >> >> >> >> >> On Thu, Apr 20, 2017 at 3:36 PM, Ganelin, Ilya < >> ilya.gane...@capitalone.com> wrote: >> >> Is there any way to wait for setup to complete prior to allowing data to >> flow through the DAG? I have an operator with a lengthy setup function so >> I’m just wondering what the best way to handle this is. My best solution at >> the moment is to toggle start with an emission rate of zero and toggle it >> on once the DAG is launched. >> >> >> >> - Ilya Ganelin >> >> >> ------------------------------ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> >> >> >> >> ------------------------------ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> >> >
