Re: How to add python modules ?

i made small progess but seeing different execption not sure why i am seeing nil value. error: 22:58:35 EDT ERROR 6f15a6f2-7744-404c-9961-f545d3f29042 ExecuteScript[id=6f15a6f2-7744-404c-9961-f545d3f29042] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: java

Re: How to add python modules ?

Hi Matt, My Python/Jython skills are poor. Can you provide me an example plz? -Madhu On Wed, Mar 30, 2016 at 5:53 PM, Matt Burgess wrote: > Mahdu, > > Since you won't be able to return your dictionary, another approach would > be to create the dictionary from the main script and pass it into t

Re: PutHDFS and LZ4 compression ERROR

Oh gosh your right...lol...to much OS mixing on my fingertips. Let me try the build of the LZ4 jar again. Thanks Matt, Thad +ThadGuidry

Re: PutHDFS and LZ4 compression ERROR

I don't think the .so will help you on Windows, you'd need a .dll instead. > On Mar 30, 2016, at 11:39 AM, Thad Guidry wrote: > > Evidently, on Windows the PATH environment variable should also have the path > to your native libraries, so that java.library.path can find them. > > I added the p

Re: How to add python modules ?

Mahdu, Since you won't be able to return your dictionary, another approach would be to create the dictionary from the main script and pass it into the callback constructor. Then process() can update it, and you can use the populated dictionary after process() returns to set attributes and such.

Re: How to add python modules ?

You are returning self.d from process() which is a void method. Needs to return None. Sent from my iPhone > On Mar 30, 2016, at 5:00 PM, Madhukar Thota wrote: > > Matt, > > I tired the following code but i am getting the following error. Can you help > me where i am doing wrong? > > Error:

Re: How to add python modules ?

Matt, I tired the following code but i am getting the following error. Can you help me where i am doing wrong? Error: 16:56:10 EDT ERROR 6f15a6f2-7744-404c-9961-f545d3f29042 ExecuteScript[id=6f15a6f2-7744-404c-9961-f545d3f29042] Failed to process session due to org.apache.nifi.processor.excepti

Re: String conversion to Int, float double

Thanks Joe. I updated my config and not seeing the issue anymore. On Wed, Mar 30, 2016 at 1:18 PM, Joe Witt wrote: > Hello > > From your screenshot it shows you have both success and failure > looping back to Kafka. Do not loop success and you should be fine. > > Thanks > Joe > > On Wed, Mar 30

Re: String conversion to Int, float double

Hello >From your screenshot it shows you have both success and failure looping back to Kafka. Do not loop success and you should be fine. Thanks Joe On Wed, Mar 30, 2016 at 11:16 AM, Madhukar Thota wrote: > I was able to construct the Json with right data type output from > ExecuteScript and s

Re: String conversion to Int, float double

I was able to construct the Json with right data type output from ExecuteScript and sending to Kafka directly. The problem i am seeing is if i send one record to kafka, kafka processor is writing the message again and again and not ending the loop. How can i send exactly once message? Any help. He

Re: Having on processor block while another one is running

Also to note if your not plugged in to the Data Management industry... Work Flow Orchestration is also sometimes called Process Management, where there are specific tools and frameworks to deal with that scope on multiple levels. You may have heard of a specific Process Management called Business

Re: PutHDFS and LZ4 compression ERROR

Evidently, on Windows the PATH environment variable should also have the path to your native libraries, so that java.library.path can find them. I added the path to my .so file to my PATH environment variableyet I still get the error 2016-03-30 10:38:16,181 ERROR [Timer-Driven Process Thread-

Re: Having on processor block while another one is running

Vincent Sorry for the late reply, but here it is Based on what you have described it appears you have a mix of two problems: Work Flow Orchestration and Data Flow. The main issue is that at the surface it’s not always easy to tell the difference, but I’ll try. Work Flow Orchestration allows on

Re: PutHDFS and LZ4 compression ERROR

Chase, It is a self-driven subscribe and unsubscribe process. Please see here https://nifi.apache.org/mailing_lists.html Thanks Joe On Wed, Mar 30, 2016 at 9:18 AM, Chase Cunningham wrote: > take me off this list... > > unsubscribe > > > On 3/30/16 10:13 AM, Joe Witt wrote: >> >> Did you set t

Re: PutHDFS and LZ4 compression ERROR

take me off this list... unsubscribe On 3/30/16 10:13 AM, Joe Witt wrote: Did you set the LD_LIBRARY_PATH as Burgress mentioned at the end? I am not in a good position to dig in at the moment so my apologies for the half-help here. The loading of native libs as I recall is a pretty specific p

Re: PutHDFS and LZ4 compression ERROR

Yes of course, on Windows 7 in my User environment variables, I set LD_LIBRARY_PATH=C:\Program Files\Java\jdk1.8.0_74\jre\lib\amd64 which as the .so file ENGAGE ! Mr Sulu :)​ Thad +ThadGuidry

Re: PutHDFS and LZ4 compression ERROR

Did you set the LD_LIBRARY_PATH as Burgress mentioned at the end? I am not in a good position to dig in at the moment so my apologies for the half-help here. The loading of native libs as I recall is a pretty specific process. I know a few folks are familiar with it in the community so let's kee

Re: PutHDFS and LZ4 compression ERROR

Ah, that's really helpful. Looks like I did have the native library in the built .jar file \lz4-1.3-SNAPSHOT\net\jpountz\util\win32\amd64\liblz4-java.so but placing that .so file in my C:\Program Files\Java\jdk1.8.0_74\jre\lib\amd64 folder results in the same missing lz4 native NiFi errors Idea

Re: PutHDFS and LZ4 compression ERROR

Thad This thread [1] seems related. Take a look and see if that helps. The basic gist as I understand it is we won't have access to that native library unless it is pointed to somewhere or unless the Java code that calls it knows how to set/find it for you. [1] http://apache-nifi-developer-list

Re: PutHDFS and LZ4 compression ERROR

My badthere is... in the app log... 2016-03-30 09:39:27,709 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@7615666e checkpointed with 8 Records and 0 Swap Files in 69 milliseconds (Stop-the-world time = 6 millisecon

Re: PutHDFS and LZ4 compression ERROR

Joe, There is no more additional output, even when I set to DEBUG level. 09:36:32 CDT ERROR 765efcb2-5ab0-4a72-a86f-71865dec264d PutHDFS[id=765efcb2-5ab0-4a72-a86f-71865dec264d] Failed to write to HDFS due to java.lang.RuntimeException: native lz4 library not available: java.lang.RuntimeExceptio

Having trouble with PutHDFS as Avro, possibly due to codec

Hi, I've got a dataflow which successfully writes Avro into a directory in HDFS. Avro-tools is able to read the Avro files so that seems fine. Now I'm trying to create a table in Hive using the imported files and my schema but the CREATE statement's query is just hanging in the Hive query editor

Re: Developing dataflows in the canvas editor

Thanks, Chris and Joe! The only other thing I'd add is that I kept looking for an Undo capability with Ctrl-Z or some such but that doesn't seem to be supported. - Dmitry On Tue, Mar 29, 2016 at 10:58 PM, Joe Witt wrote: > Dmitry these are great questions and Chris that was in my opinion a > p

Re: Sqoop Support in NIFI

Are you planning to use something like Hive or Spark to query the data? Both will work fine with Avro formatted data under a table. I’m not sure what you mean by “Table Structure” or if you have a particular format in mind, but there is I believe talk of adding processors that will write direct