Cluster using Site-to-Site

2017-04-14 Thread Mark Bean
In 0.x, one would point a Remote Process Group (RPG) at the NCM. The NCM would then take care of load balancing to the corresponding Input Port on the Nodes. In 1.x, the same behavior is true so long as the RPG points to the Cluster Coordinator. (I believe this is true at least.) Here's the rub. S

Re: Aw: Re: 4 Processors for NiFi: SplitToAttribute, MergeTemplate, ExecuteRuleEngine, GenerateData

2017-04-14 Thread Joe Witt
No problem. Docs on the components are there but we will went to build a lot of solutions oriented docs soon. On Apr 14, 2017 12:04 PM, "Uwe Geercken" wrote: > > > Joe, > > will do so - sorry for that and thanks for the link to the wiki. > > I am still in a learning phase. Programming helps me

Aw: Re: 4 Processors for NiFi: SplitToAttribute, MergeTemplate, ExecuteRuleEngine, GenerateData

2017-04-14 Thread Uwe Geercken
Joe,   will do so - sorry for that and thanks for the link to the wiki.   I am still in a learning phase. Programming helps me to dig in more details and better understand the things under the hood. I will be enhancing my processors over time and will put words in the documentation that point o

Aw: Re: InputStream and LineIterator

2017-04-14 Thread Uwe Geercken
Joe, actually, this is a good question. My processor needs to run on an individual line to run the business rules against the data. The idea is to then output individual rows/flow files. These flow files contain several attributes indicating the state of the ruleengine (the results). In a nect

Re: InputStream and LineIterator

2017-04-14 Thread Matt Burgess
Seems like the need for multiple SplitTexts is only because we keep all the splits in memory until we can get a count to put in an attribute, maybe we should make that configurable in case the user isn't going to use the fragment.* attributes. If they are going to use them, then multiple Splits

Re: 4 Processors for NiFi: SplitToAttribute, MergeTemplate, ExecuteRuleEngine, GenerateData

2017-04-14 Thread Joe Witt
Uwe, Please avoid cross posting to both dev and user lists. I recognize that you'd like to let folks know these exist and that is fine. We started up this wiki page to help folks just like you let others know about processors they've built that they're happy to have others use: https://cwiki.apa

Re: InputStream and LineIterator

2017-04-14 Thread Joe Witt
Uwe Enabling developers to cleanly handle extremely large objects is precisely why the ProcessSession interface to interact with FlowFile content occurs through Input and OutputStreams. With that model you can load just as much content as you need into memory at any one time and nothing more. Th

4 Processors for NiFi: SplitToAttribute, MergeTemplate, ExecuteRuleEngine, GenerateData

2017-04-14 Thread Uwe Geercken
Hello everybody,   I have released (Apache License) my NiFi processors at:   https://github.com/uwegeercken/nifi_processors[https://deref-web-02.de/mail/client/WDRw2DECNaw/dereferrer/?redirectUrl=https%3A%2F%2Fgithub.com%2Fuwegeercken%2Fnifi_processors]   Further below is a summary for the processo

InputStream and LineIterator

2017-04-14 Thread Uwe Geercken
Hi everybody, in my ExecuteRuleEngine processor I have used a LineIteraotr to loop ofer incomming rows of data (CSV). I was wondering if that is the preferred method for (large) files or if there are better ways to do it. Also, is there a way to influence the number of buffered rows or bytes

Using a NiFi nar in MiniFi

2017-04-14 Thread Koen Rutten
Hello all I was trying to use the ConsumeMQTT processor in MiniFi. If I read the docs correctly for the JVM-based version, I would just have to copy over the "nifi-mqtt-nar-1.1.0.2.1.2.0-10.nar” and restart MiniFi. However the new processor doesn’t seem to be picked up. The MiniFi-ap logs sta