Re: Sqoop Support in NIFI

For Below reasons i have choose Sqoop in NIFI Processor is the best method to move data in Table Structure. If once move the Table from oracle or sql server into HDFS then whole moved data which must be in Table format not in avro or json..etc. For Example:Table Data from Orac

Re: PutHDFS and LZ4 compression ERROR

Sure Joe, I'll get that for you tomorrow morning. Thad +ThadGuidry On Tue, Mar 29, 2016 at 10:00 PM, Joe Witt wrote: > Thad, > > Can you share the full stack trace that should be present in the log > with that? There is clearly a bit of Java code attempti

Re: PutHDFS and LZ4 compression ERROR

Thad, Can you share the full stack trace that should be present in the log with that? There is clearly a bit of Java code attempting to load the native library and unable to find it. Placing the jar file in the classpath which contains the native library may well not be enough because loading th

Re: Developing dataflows in the canvas editor

Dmitry these are great questions and Chris that was in my opinion a pretty excellent response - 'noob' or not. The only thing I'd add Dmitry is that some of what you're saying regarding templates themselves is very true. We can do better and so much more than we are. We have a feature proposal/d

Re: Developing dataflows in the canvas editor

Dimitri, From one noob to another, welcome. All modifications to the canvas are automatically saved. If you want to organize multiple flow instances look to process groups. Drag a process group onto the canvas. Double click the process group to open it. Then drag a template onto the canvas.

Re: OutOfMemory / PermGen space

Dmitry, What version of Java are you running? My guess is you are running Java 7 and in that case I suggest you uncomment out the java related lines in the bootstrap.conf file: java.arg.7=-XX:ReservedCodeCacheSize=256m java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m java.arg.9=-XX:+U

Re: OutOfMemory / PermGen space

Dmitry, This indicates you're using Java 7 most likely. Look here [1] which points to [2]. Specifically you'll want to update conf/bootstrap.properties To have these lines uncommented java.arg.11=-XX:PermSize=128M java.arg.12=-XX:MaxPermSize=128M You can choose different values than 128M as y

OutOfMemory / PermGen space

Hi, Started getting OOM once I've added PutHDFS to a flow. I'm running on a CentOS VM with 2GB of memory allocated. I've munged the nifi startup script to do -Xms1024m -Xmx1024m". Any recommendations on this? Thanks, - Dmitry 2016-03-29 11:25:54,200 ERROR [Flow Service Tasks Thread-2] o.a.nif

PutHDFS and LZ4 compression ERROR

I get an error: 13:04:51 CDT ERROR 765efcb2-5ab0-4a72-a86f-71865dec264d PutHDFS[id=765efcb2-5ab0-4a72-a86f-71865dec264d] Failed to write to HDFS due to java.lang.RuntimeException: native lz4 library not available: java.lang.RuntimeException: native lz4 library not available even though I built s

Developing dataflows in the canvas editor

Hi, These may be just 'noob' impressions from someone who hasn't learned enough NiFi yet (I may be missing something obvious). My first confusion is about dataflows vs. templates. I've developed a couple of templates. Now I want to drop a template into the canvas and treat that as a dataflow or

Re: EvaluateJsonPath and Json Field Name Starting with @ as the First Character

Thanks, Matt. Out version is 0.5.1. We have been using HDF. But we are going to upgrade to 0.6.0 from Apache directly this time. Hong *Hong Li* *Centric Consulting* *In Balance* (888) 781-7567 office (614) 296-7644 mobile www.centricconsulting.com | @Centric

Re: Having on processor block while another one is running

Vincent I do have a suggestion for you but need a bit more time to craft my response. Give me till tonight EST. Cheers Oleg On Mar 29, 2016, at 8:55 AM, Vincent Russell mailto:vincent.russ...@gmail.com>> wrote: Thanks Oleg and Joe, I am not currently convinced that nifi is the solution as wel

Re: Having on processor block while another one is running

Thanks Oleg and Joe, I am not currently convinced that nifi is the solution as well, but it is a nice way for us to manage actions based on the result of a mapreduce job. Our use cases is to have follow on processors that perform actions based on the results of the map reduce jobs. One processor

Re: EvaluateJsonPath and Json Field Name Starting with @ as the First Character

Hong, I was able to use EvaluateJsonPath with eventClass $.event.@class and the attribute had the correct value (see output from LogAttribute below): -- Standard FlowFile Attributes Key: 'entryDate' Value: 'Tue Mar 29 08:41:36 EDT 2016' Key: 'lineag

Re: Sqoop Support in NIFI

Another processor that may be of interest to you is the QueryDatabaseTable processor, which has just been released in 0.6.0. This provides incremental load capabilities similar to sqoop. If you’re looking for the schema type functionality, bear in mind that the ExecuteSQL (and new Query process

EvaluateJsonPath and Json Field Name Starting with @ as the First Character

How can I assign the value of a Json field that has a name starting with @ as the first character of its name to a user-defined-property in EvaluateJsonPath? For example, I have the following sample Json message: { "effectiveTimestamp": "2015-12-03T23:17:29.874Z", "event": { > "@class": "events.

Re: Sqoop Support in NIFI

Hi, Yes, In my case i have created the Custom processor with Sqoop API which accommodates complete functionality of sqoop. As per you concern we have able to move the data only from HDFS to SQl or Vice versa, But sqoop having more functionality which we can achieve it by Sqoop.RunTool() in org.apa

Re: Sqoop Support in NIFI

Hi, If you could explain exactly what you are trying to achieve I.e. What part of the data pipeline you are looking to use NiFi for and where you wish to retain Sqoop I could perhaps have a more informed input (although I have only been using NiFi myself for a few weeks). Sqoop obviously can mov