Hey Ilya, Can you please remove the duplicate output port from your implementation of NewLineFileInputOperator as well.
Thanks, Chandni On Fri, Mar 11, 2016 at 3:42 PM, Ashwin Chandra Putta < ashwinchand...@gmail.com> wrote: > Why do you want to have thread locality and partitioning together, you will > lose parallel processing. What is the use case? > > Regards, > Ashwin. > > On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya < > ilya.gane...@capitalone.com> > wrote: > > > Now with files: > > https://gist.github.com/ilganeli/7f770374113b40ffa18a > > > > From: "Ganelin, Ilya" <ilya.gane...@capitalone.com<mailto: > > ilya.gane...@capitalone.com>> > > Reply-To: "dev@apex.incubator.apache.org<mailto: > > dev@apex.incubator.apache.org>" <dev@apex.incubator.apache.org<mailto: > > dev@apex.incubator.apache.org>> > > Date: Friday, March 11, 2016 at 3:02 PM > > To: "dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org > >" > > <dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>> > > Subject: Stack overflow errors when launching job > > > > Hi guys – I’m running into a very frustrating issue where certain DAG > > configurations cause the following error log (attached). When this > happens, > > my application even fails to launch. This does not seem to be a YARN > issue > > since this occurs even with a relatively small number of > partitions/memory. > > > > I’ve attached the input and output operators in question. > > I can get this to occur predictable by > > > > 1. Increasing the partition count on my input operator (reads from > > HDFS) - values above 20 cause this error > > 2. Increase the partition count on my output operator (writes to HDFS) > > - values above 20 cause this error > > 3. Set stream locality from the default to either thread local, node > > local, or container_local on the output operator > > > > This behavior is very frustrating as it’s preventing me from partitioning > > my HDFS I/O appropriately, thus allowing me to scale to higher > throughputs. > > > > Do you have any thoughts on what’s going wrong? I would love your > feedback. > > > > > > ________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of any > > action in reliance upon this information is strictly prohibited. If you > > have received this communication in error, please contact the sender and > > delete the material from your computer. > > ________________________________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of any > > action in reliance upon this information is strictly prohibited. If you > > have received this communication in error, please contact the sender and > > delete the material from your computer. > > > > > > -- > > Regards, > Ashwin. >
