Re: NiFi processor for Redis

Suddep Also, when you say push/retrieve do you mean publish/subscribe or put/get since Redis is both storage and messaging system. As Joe mentioned we don’t have any at the moment but would be nice to start prioritizing. Cheers Oleg > On Jan 13, 2016, at 12:41 AM, Joe Witt wrote: > > Sudeep,

Re: "Processor requires an upstream connection" for FetchS3Object?

Russell, The general contract is for ListXXX to identify files to process without moving content and FetchXXX to pull the content into the data flow. So, FetchS3 is ideally meant to work in conjunction with a ListS3 instance, but your solution of using ExecuteStreamCommand with the S3 CLI tools s

Re: PutDistributedMapCache

Hello Sudeep, I put up a patch on the GetDistributedMapCache ticket[1]. Let me know what you think. The PutDistributedMapCache processor and GetDistributedMapCache work with the data as a byte[] so it should be format agnostic. That being said it will be up to you to know what is in there in

Re: PutDistributedMapCache

Thank you very much Joe. Can you please let me know how I can use the .patch file? I am using the NiFi via the binaries... Do I need to setup the source code and build the same along with the patch? Thanks & Regards, Sudeep On Wed, Jan 13, 2016 at 9:02 PM, Joe Percivall wrote: > Hello Sudeep,

Re: PutDistributedMapCache

You would need to clone the nifi source from github and then apply the patch using git. Here is how to clone a repo: https://help.github.com/articles/cloning-a-repository/ Along with the nifi repo itself: https://github.com/apache/nifi and how to apply a patch: http://makandracards.com/makandr

Re: PutDistributedMapCache

Thanks Joe. I will try out the patch. On Wed, Jan 13, 2016 at 9:31 PM, Joe Percivall wrote: > You would need to clone the nifi source from github and then apply the > patch using git. > > Here is how to clone a repo: > https://help.github.com/articles/cloning-a-repository/ > Along with the nifi

PutSQL question

Hi I want to use the PutSQL processor to execute an insert statement. The use case is straightforward. I need to pull an id attribute from a JSON message and write it to a database. I am using EvaluateJSONPath to grab the id but I’m hung up on how to then pass the SQL statement as the content

Re: PutSQL question

Ralph, Hello. An easy way to do this is to use ReplaceText to set the content of the FlowFile. You can set the Replacement Strategy to "Always Replace" and set the Replacement Value to something like: INSERT INTO MY_TABLE (ID) VALUES ('${id}') You could also use SQL parameters (though it's slig

Re: PutSQL question

Hello Ralph, A common way to replace the contents of a FlowFile with specific text is to use the ReplaceText processor. The default search query will match the entire content and replace it with whatever your replacement value is. So if you set the replacement value (complete with expression lan

Re: PutSQL question

Mark and Joe – Thanks for the quick replay – that did the trick. Worked great. From: Mark Payne mailto:marka...@hotmail.com>> Reply-To: "users@nifi.apache.org" mailto:users@nifi.apache.org>> Date: Wednesday, January 13, 2016 at 12:25 PM To: "users@nifi.apache.org

Re: Data Ingestion forLarge Source Files and Masking

Hi Joe & Others, Thanks for all of your suggestions. Now I am using below code: 1. Buffered reader (I tried to use NLKBufferedReader, but it requires too many libs & Nifi failed to start. I was lost.) 2. Buffered writer 3. Using appending line end instead to concat new line Still no performance

Re: Data Ingestion forLarge Source Files and Masking

Hello Let's narrow in on potential issues. So while this process is running and appears sluggish in nature please run the following on the command line 'jps' This command will tell you the process id of NiFi. You'll want the pid associated with the Java process other than what is called 'jps'

Re: PutDistributedMapCache

Is it possible to build the code for only a particular processor? Just curious if we can build and deploy a particular processor in an existing NiFi environment. On Wed, Jan 13, 2016 at 9:33 PM, sudeep mishra wrote: > Thanks Joe. I will try out the patch. > > On Wed, Jan 13, 2016 at 9:31 PM, Joe

Re: Data Ingestion forLarge Source Files and Masking

Hi Joe, Please find attached jstat & iostat output. So far it seems to me that it is CPU bound. However, your eyes are better tan mine :). -Obaid On Thu, Jan 14, 2016 at 11:51 AM, Joe Witt wrote: > Hello > > Let's narrow in on potential issues. So while this process is running > and appears

Re: Data Ingestion forLarge Source Files and Masking

Obaid, Great so this is helpful info. Iostat output shows both CPU and disk are generally bored and ready for more work. Looking at the gc output though suggests trouble. We see there are 32 samples at 1 second spread each and in that time spent more than 6 seconds of it doing garbage collectio

Re: Data Ingestion forLarge Source Files and Masking

Joe, Last time it was below: java.arg.2=-Xms512m java.arg.3=-Xmx512m Now I made as below: java.arg.2=-Xms5120m java.arg.3=-Xmx10240m latest jstate & iostate output are attached. To me it is still slow, no significant improvements. -Obaid On Thu, Jan 14, 2016 at 12:41 PM, Joe Witt wrote: > Ob

Re: Data Ingestion forLarge Source Files and Masking

Quick observation for now off latest data: - GC looks pretty good though it is surprising there were any full GCs during that short test - cpu has low utilization - disk has low utilization Can you share your sample input data, processor code, flow as a template? Attaching to a JIRA for example c

Re: Data Ingestion forLarge Source Files and Masking

Joe, I am doing some optimizations on my csv processing. Let clear them out then I will share the final version. -Obaid On Thursday, January 14, 2016, Joe Witt wrote: > Quick observation for now off latest data: > - GC looks pretty good though it is surprising there were any full GCs > during t

Re: NIFI connecting to Activemq

Thanks for the update. Do you have suggestion what I should set my processor to? On Jan 9, 2016 8:25 PM, "Joe Witt" wrote: > Chris, > > Thanks for sending the screenshot. The NullPointerException is > trivially reproduced using your settings. I didn't even need a JMS > server to cause it. > > H

Re: NIFI connecting to Activemq

This page provides some good examples to consider for your configuration http://activemq.apache.org/uri-protocols.html On Thu, Jan 14, 2016 at 1:17 AM, Christopher Hamm wrote: > Thanks for the update. Do you have suggestion what I should set my processor > to? > > On Jan 9, 2016 8:25 PM, "Joe Wi

Re: PutDistributedMapCache

Upon building the repository we get different .nar files which can be updated in the lib for my requirement. Thanks for your help. On Thu, Jan 14, 2016 at 9:27 AM, sudeep mishra wrote: > Is it possible to build the code for only a particular processor? Just > curious if we can build and deploy a

Is there a way to configure a processor to run only N times

Hi, Can we configure to run a processor to run only 'N' times. In my data flow I want that some processor should run only once. How can I achieve it? Thanks & Regards, Sudeep