Re: How does Nifi ingest large files?

Indeed. I went ahead and configured my dev cluster to use RAM-disk for content and flowfile repositories and turned back on FileSystemRepository and WriteAheadFlowFileRepository respectively. As long as the content/provenance archive is off, I'm good wrt compliance. The performance seems great so

Re: How does Nifi ingest large files?

I think Jeremy is using Volatile specifically because he does *not* want that data ever persisted to disk for compliance purposes. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 27, 2016, at 8:59 AM, Joe

Re: nifi is running out of memory

Hey All, I believe OP is running into a bug fixed here: https://issues.apache.org/jira/browse/NIFI-2631 Basically, ListS3 attempts to commit all the files it finds (potentially 100k+) at once, rather than in batches. NIFI-2631 addresses the issue. Looks like the fix is out in 0.7.1 but not yet in

Re: nifi is running out of memory

Looking at this line [1] makes me think the FetchS3 processor is properly streaming the bytes directly to the content repository. Looking at the screenshot showing nothing out of the ListS3 processor makes me think the bucket has so many things in it that the processor or associated library isn't

Re: nifi is running out of memory

moving dev to bcc Yes I believe the issue here is that FetchS3 doesn't do chunked transfers and so is loading all into memory. I've not verified this in the code yet but it seems quite likely. Krish if you can verify that going with a larger heap gets you in the game can you please file a JIRA.

Re: nifi is running out of memory

Hello, Are you running with all of the default settings? If so you would probably want to try increasing the memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb. -Bryan On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr wrote: > Hi All, > > I have

data flow from one s3 bucket to another

Has anyone implemented data copy from one s3 bucket to another. i would greatly appreciate if you can share with me your sample processors configuration. Thanks Rai

Re: IllegalStateException

Conrad, Am looking into this now. Could you please file a JIRA. Given the error it is showing there is without any doubt a coding error. Thanks Joe On Thu, Oct 27, 2016 at 9:35 AM, Conrad Crampton wrote: > Hi, > > I’m getting this exception thrown by most of my processors for a given > pipeli

Re: How does Nifi ingest large files?

I should add that if you're comfortable with that sort of volatile behavior a better path to consider is to setup a RAM-Disk and just run a persistent content repository on that. It will survive process restarts, give better memory/heap behavior (by a lot), but you'll lose data on system restarts.

Re: How does Nifi ingest large files?

That is correct. Thanks Joe On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota wrote: > Bryan, > > If I have the content repo implementation set to > org.apache.nifi.controller.repository.VolatileContentRepository, it will > stream the content in memory, correct? > > On Thu, Oct 27, 2016 at 6:22

Re: How does Nifi ingest large files?

Bryan, If I have the content repo implementation set to org.apache.nifi.controller.repository.VolatileContentRepository, it will stream the content in memory, correct? On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende wrote: > Monica, > > Are you asking what does NiFi do when it picks up a large fil

Re: How does Nifi ingest large files?

I will check, thank you! *Monica Franceschini* Solution Architecture Manager *Big Data Competence Center Engineering Group* Corso Stati Uniti 23/C, 35127 Padova, Italia Tel: +39 049.8283547 Fax: +39 049.8692566 Twitter: @twittmonique www.spagobi.org - www.eng.it

Re: How does Nifi ingest large files?

In the case of a GetFile processor it is managed by a single node since the file being picked up is on the local filesystem of one of the nodes. There are other approaches to parallelize work... If you had a shared network location you can use ListFile + FetchFile in a certain way so that one node

Re: How does Nifi ingest large files?

Thank you Bryan, yes that's what I meant and it makes sense to me. Only a further question: is this stream parallelized if needed on the (hypothetical) Nifi cluster or it is managed by a single node? Cheers Monica

IllegalStateException

Hi, I’m getting this exception thrown by most of my processors for a given pipeline, “ConvertAvroSchema[id=3b590825-5ef3-38d7-8c26-4e99d24f9d69] ConvertAvroSchema[id=3b590825-5ef3-38d7-8c26-4e99d24f9d69] failed to process session due to java.lang.IllegalStateException: Cannot create Provenance E

Re: How does Nifi ingest large files?

Monica, Are you asking what does NiFi do when it picks up a large file from the filesystem using a processor like GetFile? If so, it will stream the content of that file into NiFi's content repository, and create a FlowFile pointing to that content. As far as NiFi is concerned the content is just

Re: Frontend for Nifi support / mailing list

Oh ... I was under the impression that there already was a nabble listing for nifi : http://apache-nifi.1125220.n5.nabble.com/Apache-NiFi-f1.subapps.html On Thu, Oct 27, 2016 at 1:36 PM, Tony Kurc wrote: > List has been created on nabble, it will be empty until messages start > coming in. I do

Error instantiating template on cluster: The specified observer identifier already exists.

Hello, I am running into a problem moving some fairly large flows (40-50 processors) from a NiFi 1.0.0 standalone instance to an 3-node NiFi 1.0.0 cluster. I saved the flows in the standalone instance as templates and uploaded the templates to the cluster instance. When I instantiate the temp

Re: Frontend for Nifi support / mailing list

List has been created on nabble, it will be empty until messages start coming in. I don't yet see a way to backfill with history. I'll get this link on https://nifi.apache.org/mailing_lists.html in a bit. http://apache-nifi-users-list.2361937.n4.nabble.com/ On Thu, Oct 27, 2016 at 6:59 AM, Tony K

Re: Frontend for Nifi support / mailing list

I will look into setting it up today On Oct 27, 2016 5:04 AM, "Davy De Waele" wrote: > Hi, > > Is there a procedure for Nifi users to get nabble posting access to the > Nifi mailing list ? > I can post (obviously) via email, but would prefer to use the nabble web > UI. > > Both myself and a coll

Frontend for Nifi support / mailing list

Hi, Is there a procedure for Nifi users to get nabble posting access to the Nifi mailing list ? I can post (obviously) via email, but would prefer to use the nabble web UI. Both myself and a colleague of mine have requested access via the nabble UI but we fail to get a response. Or are there alt

How does Nifi ingest large files?

Hi, I'm figuring out how does Nifi ingest large files: does it split them into chunks or is it a massive load?Can you please, explain the behavior? Kind regards, Monica -- *Monica Franceschini* Solution Architecture Manager *Big Data Competence Center Engineering Group* Corso Stati Uniti 23/C,