Quick remark: the fix has also been merged in master and will be in release 1.1.0.
Pierre 2016-10-28 15:22 GMT+02:00 Gop Krr <gop....@gmail.com>: > Thanks Adam. I will try 0.7.1 and update the community on the outcome. If > it works then I can create a patch for 1.x > Thanks > Rai > > On Thu, Oct 27, 2016 at 7:41 PM, Adam Lamar <adamond...@gmail.com> wrote: > >> Hey All, >> >> I believe OP is running into a bug fixed here: >> https://issues.apache.org/jira/browse/NIFI-2631 >> >> Basically, ListS3 attempts to commit all the files it finds >> (potentially 100k+) at once, rather than in batches. NIFI-2631 >> addresses the issue. Looks like the fix is out in 0.7.1 but not yet in >> a 1.x release. >> >> Cheers, >> Adam >> >> >> On Thu, Oct 27, 2016 at 7:59 PM, Joe Witt <joe.w...@gmail.com> wrote: >> > Looking at this line [1] makes me think the FetchS3 processor is >> > properly streaming the bytes directly to the content repository. >> > >> > Looking at the screenshot showing nothing out of the ListS3 processor >> > makes me think the bucket has so many things in it that the processor >> > or associated library isn't handling that well and is just listing >> > everything with no mechanism of max buffer size. Krish please try >> > with the largest heap you can and let us know what you see. >> > >> > [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/ >> nifi-aws-bundle/nifi-aws-processors/src/main/java/org/ >> apache/nifi/processors/aws/s3/FetchS3Object.java#L107 >> > >> > On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> moving dev to bcc >> >> >> >> Yes I believe the issue here is that FetchS3 doesn't do chunked >> >> transfers and so is loading all into memory. I've not verified this >> >> in the code yet but it seems quite likely. Krish if you can verify >> >> that going with a larger heap gets you in the game can you please file >> >> a JIRA. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende <bbe...@gmail.com> wrote: >> >>> Hello, >> >>> >> >>> Are you running with all of the default settings? >> >>> >> >>> If so you would probably want to try increasing the memory settings in >> >>> conf/bootstrap.conf. >> >>> >> >>> They default to 512mb, you may want to try bumping it up to 1024mb. >> >>> >> >>> -Bryan >> >>> >> >>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr <gop....@gmail.com> wrote: >> >>>> >> >>>> Hi All, >> >>>> >> >>>> I have very simple data flow, where I need to move s3 data from one >> bucket >> >>>> in one account to another bucket under another account. I have >> attached my >> >>>> processor configuration. >> >>>> >> >>>> >> >>>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2] >> >>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow >> Service >> >>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space >> >>>> >> >>>> I am very new to NiFi and trying ot get few of the use cases going. >> I need >> >>>> help from the community. >> >>>> >> >>>> Thanks again >> >>>> >> >>>> Rai >> >>>> >> >>>> >> >>>> >> >>> >> > >
