Clarification regarding swap of flow file to storage

2019-12-04 Thread sanjeet rath
Hi Team, As per the nifi document, once the number of flow file in a connection queue will reach the threshold value, then it removed from hash map(Java heap space) and moved to disc. could u please confirm the deault which repository it moves to , it's content repository or to flowfile

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread James McMahon
ForkRecord... I’ve never had the chance to use that processor yet, but will look into it tomorrow too. Thanks to both you guys for the options. -Jim On Wed, Dec 4, 2019 at 9:38 PM Matt Burgess wrote: > That's a good idea, I hadn't thought of that because I didn't know what > Jim's upstream flow

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread Matt Burgess
That's a good idea, I hadn't thought of that because I didn't know what Jim's upstream flow looked like. You could also consider ForkRecord, that might give you what you want if you fork on /FNAME. Regards, Matt On Wed, Dec 4, 2019 at 9:26 PM 노대호Daeho Ro wrote: > Of course. > > There is a

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread James McMahon
Thank you Daeho. I will be back at work in just a few hours and will try this approach. It sounds like it is just what I need. Thanks again. On Wed, Dec 4, 2019 at 9:26 PM 노대호Daeho Ro wrote: > Of course. > > There is a processor, the name is SplitJson. It can split the JSON text by > defined

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread 노대호Daeho Ro
Of course. There is a processor, the name is SplitJson. It can split the JSON text by defined key. For example, if there is a key name is 'fname' and has the value [a, b, c]. Once you split the JSON by that processor, the resulted JSON will have the same key and values for others but 'fname' will

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread James McMahon
I don’t quite follow, Daeho. FNAME is an attribute that results *from* EvaluateJSonPath. Can you explain what you mean by splitting the Jason key before EvaluateJSonPath? Jim On Wed, Dec 4, 2019 at 7:45 PM 노대호Daeho Ro wrote: > I think you can split the json key for FNAME just before the >

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread James McMahon
Our NiFi version is 1.7. Is there an approach you can suggest for that version, Matt? On Wed, Dec 4, 2019 at 6:24 PM Matt Burgess wrote: > Jim, > > As of NiFi 1.8.0 [1], you should be able to do this with > UpdateAttribute -> DuplicateFlowFile -> UpdateAttribute pattern, the > first getting the

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread 노대호Daeho Ro
I think you can split the json key for FNAME just before the EvaluateJsonPath processor. Then, the fragment.* attributes will be automatically created. 2019년 12월 5일 (목) 오전 8:24, Matt Burgess 님이 작성: > Jim, > > As of NiFi 1.8.0 [1], you should be able to do this with > UpdateAttribute ->

Re: Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread Matt Burgess
Jim, As of NiFi 1.8.0 [1], you should be able to do this with UpdateAttribute -> DuplicateFlowFile -> UpdateAttribute pattern, the first getting the number of values in the list via the count() EL function, the second using that (minus 1) to generate duplicates, each with a copy.index attribute

Split a flow file to multiples, related by common key and with split count

2019-12-04 Thread James McMahon
I have a series of attributes that result from an EvaluateJSonPath. One of those attributes, FNAME, appears to be a list of values like so: [“A”,”B”,”C”]. I want to split my flow file into one for each list element. I need my results to have the original content, all the original attributes, and

Re: LookupRecord with RestLookupService

2019-12-04 Thread Etienne Jouvin
Hello all. Found a solution. On the LookupRecord, I put those properties: mime.type : toString('application/json', 'UTF-8') request.body : toString('{...}', 'UTF-8') request.method : toString('POST', 'UTF-8') With a big difficulty, the request.body variable content should not contains any

LookupRecord with RestLookupService

2019-12-04 Thread Etienne Jouvin
Hello all. I am trying a little bit the LookupRecord with RestlookupService. When watching the code for RestLookupService, we can see the following in the lookup function : final String endpoint = determineEndpoint(coordinates); final String mimeType = (String)coordinates.get(MIME_TYPE_KEY);

Nifi Cluster Untrusted Proxy Error

2019-12-04 Thread Dweep Sharma
Hi All , I am trying to setup a NIFI Cluster (2 node) with OpenIDConnect (Google). The cluster setup was done using https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup and openid was done using https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect The

Re: NIFI 1.9.2 stuck in cluster mode

2019-12-04 Thread nayan sharma
Hi Mark, One more thing I have noticed about the putHDFS that only two files has been written and rest of them are in complete. Please refer image. https://i.imgur.com/hu26hQG.png Thank you, Mark On 2019/12/04 07:30:03, nayan sharma wrote: > Hi Mark, > Thanks for your valuable suggestion.