Re: Query related to ExecuteScript

2016-08-18 Thread Matt Burgess 
Koustav,

Glad to hear it! Let me know if you run into any other issues.

Regards,
Matt

On Thu, Aug 18, 2016 at 6:05 PM, koustav choudhuri 
wrote:

> I found the issue . Uncommented the parameter java.arg.11 in
> bootstrap.conf which resolved the issue. Now the template runs fine.
>
>
> Thanks
> Koustav
>
> On Thu, Aug 18, 2016 at 1:57 PM, koustav choudhuri 
> wrote:
>
>> Hi Matt
>>
>> This is what I did :
>>
>> Removed the NIFI directory and killed all the related processes.
>> Installed Nifi 0.7
>> started the service
>> imported your template JythonJsonToJsonExample and started the job
>>
>> This is when the web server stops responding.
>>
>> Errors from bootstrap.log:
>> 2016-08-18 20:49:44,138 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> 2016-08-18 20:49:44,138 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
>> the UncaughtExceptionHandler in thread "NiFi Web Server-93"
>> 2016-08-18 20:49:44,347 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> 2016-08-18 20:49:44,347 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
>> the UncaughtExceptionHandler in thread "NiFi Web Server-113"
>> 2016-08-18 20:50:21,388 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> 2016-08-18 20:50:21,388 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
>> the UncaughtExceptionHandler in thread "NiFi Web Server-118"
>> 2016-08-18 20:50:21,787 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> 2016-08-18 20:50:21,787 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
>> the UncaughtExceptionHandler in thread "NiFi Web Server-120"
>>
>>
>> Errors from nifi-app.log:
>>
>>
>> 2016-08-18 20:49:41,987 WARN [NiFi Web Server-112]
>> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3@54b057d5 in NiFi Web
>> Server{STARTED,8<=13<=200,i=3,q=0}
>> 2016-08-18 20:49:41,987 ERROR [NiFi Web Server-112] org.apache.nifi.NiFi
>> An Unknown Error Occurred in Thread Thread[NiFi Web Server-112,5,main]:
>> java.lang.OutOfMemoryError: PermGen space
>> 2016-08-18 20:49:42,759 WARN [NiFi Web Server-111]
>> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3@54b057d5 in NiFi Web
>> Server{STARTED,8<=13<=200,i=4,q=0}
>> 2016-08-18 20:49:42,759 ERROR [NiFi Web Server-111] org.apache.nifi.NiFi
>> An Unknown Error Occurred in Thread Thread[NiFi Web Server-111,5,main]:
>> java.lang.OutOfMemoryError: PermGen space
>>
>>
>> free -m results in :
>>
>> [image: Inline image 1]
>>
>> Any pointers on what is going wrong ?
>>
>>
>> Thanks
>> Koustav
>>
>>
>>
>> On Wed, Aug 17, 2016 at 7:08 PM, Matt Burgess 
>> wrote:
>>
>>> Koustav,
>>>
>>> I took your script (with GenerateFlowFile before it) and ran just fine
>>> (meaning the file is created and the processor does not hang). When I
>>> didn't have write permissions for the target folder, I got an
>>> error/bulletin telling me this. When I did have write permissions, it works
>>> fine.
>>>
>>> However I should mention that if you want something to go to the PutFile
>>> (in your flow above), you will not get anything output (to the flow) with
>>> your current script. Instead you need to create/reuse a flow file and then
>>> use the session to transfer it. There is an example of how to do this on my
>>> blog: http://funnifi.blogspot.com/2016/03/executescript-json
>>> -to-json-revisited_14.html
>>>
>>> Regards,
>>> Matt
>>>
>>> On Wed, Aug 17, 2016 at 7:34 PM, koustav choudhuri <
>>> koustav1...@gmail.com> wrote:
>>>
 Matt

 Thanks for Replying , but I am still struggling with ExecuteScript.


 [image: Inline image 1]


 in that ExecuteScript all I have is :

 Script - set to Python
 Script Body :
 from sys import argv

 filename = "nifi_output/nifi_test.txt"
 target = open(filename, 'w')
 target.write("My Name is Bond")
 target.close()

 When I run this , Nifi hangs for some reason. Am I doing something
 grossly wrong ?






 On Wed, Aug 17, 2016 at 3:57 PM, Matt Burgess 
 wrote:

> If you need an input flowfile, you're probably better off with
> ExecuteStreamCommand than ExecuteScript for this use case.
> ExecuteStreamCommand is much like ExecuteProcess but it accepts input flow
> files.
>
> Regards,
> Matt
>
> > On Aug 17, 2016, at 6:49 PM, koustav choudhuri <
> koustav1...@gmail.com> wrote:
> >
> > HI All
> >
> > I am trying to execute 2 different shell scripts ( calling a sqoop
> job and calling a hive script) in a sequence. Can I achieve this through
> NIFI ?

Re: Query related to ExecuteScript

2016-08-18 Thread koustav choudhuri 
I found the issue . Uncommented the parameter java.arg.11 in bootstrap.conf
which resolved the issue. Now the template runs fine.


Thanks
Koustav

On Thu, Aug 18, 2016 at 1:57 PM, koustav choudhuri 
wrote:

> Hi Matt
>
> This is what I did :
>
> Removed the NIFI directory and killed all the related processes.
> Installed Nifi 0.7
> started the service
> imported your template JythonJsonToJsonExample and started the job
>
> This is when the web server stops responding.
>
> Errors from bootstrap.log:
> 2016-08-18 20:49:44,138 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2016-08-18 20:49:44,138 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
> the UncaughtExceptionHandler in thread "NiFi Web Server-93"
> 2016-08-18 20:49:44,347 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2016-08-18 20:49:44,347 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
> the UncaughtExceptionHandler in thread "NiFi Web Server-113"
> 2016-08-18 20:50:21,388 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2016-08-18 20:50:21,388 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
> the UncaughtExceptionHandler in thread "NiFi Web Server-118"
> 2016-08-18 20:50:21,787 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2016-08-18 20:50:21,787 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from
> the UncaughtExceptionHandler in thread "NiFi Web Server-120"
>
>
> Errors from nifi-app.log:
>
>
> 2016-08-18 20:49:41,987 WARN [NiFi Web Server-112] 
> o.e.jetty.util.thread.QueuedThreadPool
> Unexpected thread death: org.eclipse.jetty.util.thread.
> QueuedThreadPool$3@54b057d5 in NiFi Web Server{STARTED,8<=13<=200,i=3,q=0}
> 2016-08-18 20:49:41,987 ERROR [NiFi Web Server-112] org.apache.nifi.NiFi
> An Unknown Error Occurred in Thread Thread[NiFi Web Server-112,5,main]:
> java.lang.OutOfMemoryError: PermGen space
> 2016-08-18 20:49:42,759 WARN [NiFi Web Server-111] 
> o.e.jetty.util.thread.QueuedThreadPool
> Unexpected thread death: org.eclipse.jetty.util.thread.
> QueuedThreadPool$3@54b057d5 in NiFi Web Server{STARTED,8<=13<=200,i=4,q=0}
> 2016-08-18 20:49:42,759 ERROR [NiFi Web Server-111] org.apache.nifi.NiFi
> An Unknown Error Occurred in Thread Thread[NiFi Web Server-111,5,main]:
> java.lang.OutOfMemoryError: PermGen space
>
>
> free -m results in :
>
> [image: Inline image 1]
>
> Any pointers on what is going wrong ?
>
>
> Thanks
> Koustav
>
>
>
> On Wed, Aug 17, 2016 at 7:08 PM, Matt Burgess  wrote:
>
>> Koustav,
>>
>> I took your script (with GenerateFlowFile before it) and ran just fine
>> (meaning the file is created and the processor does not hang). When I
>> didn't have write permissions for the target folder, I got an
>> error/bulletin telling me this. When I did have write permissions, it works
>> fine.
>>
>> However I should mention that if you want something to go to the PutFile
>> (in your flow above), you will not get anything output (to the flow) with
>> your current script. Instead you need to create/reuse a flow file and then
>> use the session to transfer it. There is an example of how to do this on my
>> blog: http://funnifi.blogspot.com/2016/03/executescript-json
>> -to-json-revisited_14.html
>>
>> Regards,
>> Matt
>>
>> On Wed, Aug 17, 2016 at 7:34 PM, koustav choudhuri > > wrote:
>>
>>> Matt
>>>
>>> Thanks for Replying , but I am still struggling with ExecuteScript.
>>>
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> in that ExecuteScript all I have is :
>>>
>>> Script - set to Python
>>> Script Body :
>>> from sys import argv
>>>
>>> filename = "nifi_output/nifi_test.txt"
>>> target = open(filename, 'w')
>>> target.write("My Name is Bond")
>>> target.close()
>>>
>>> When I run this , Nifi hangs for some reason. Am I doing something
>>> grossly wrong ?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 17, 2016 at 3:57 PM, Matt Burgess 
>>> wrote:
>>>
 If you need an input flowfile, you're probably better off with
 ExecuteStreamCommand than ExecuteScript for this use case.
 ExecuteStreamCommand is much like ExecuteProcess but it accepts input flow
 files.

 Regards,
 Matt

 > On Aug 17, 2016, at 6:49 PM, koustav choudhuri 
 wrote:
 >
 > HI All
 >
 > I am trying to execute 2 different shell scripts ( calling a sqoop
 job and calling a hive script) in a sequence. Can I achieve this through
 NIFI ?
 >
 > I tried to do it through ExecuteProcess Processor but the issue is
 Execute Processor does not accept an input.
 >
 > Then I moved to ExecuteScript :
 > Created 2 different python scripts, 1 calling the Sqoop job and the
 other calling the Hive script and arranged them in 2 different

Re: Query related to ExecuteScript

2016-08-17 Thread Jagrut Sharma 
Koustav - Try checking the permissions for the directory you are trying to
write to. Does the user running nifi have appropriate access?
--
Jagrut

On Wed, Aug 17, 2016 at 4:34 PM, koustav choudhuri 
wrote:

> Matt
>
> Thanks for Replying , but I am still struggling with ExecuteScript.
>
>
> [image: Inline image 1]
>
>
> in that ExecuteScript all I have is :
>
> Script - set to Python
> Script Body :
> from sys import argv
>
> filename = "nifi_output/nifi_test.txt"
> target = open(filename, 'w')
> target.write("My Name is Bond")
> target.close()
>
> When I run this , Nifi hangs for some reason. Am I doing something grossly
> wrong ?
>
>
>
>
>
>
> On Wed, Aug 17, 2016 at 3:57 PM, Matt Burgess  wrote:
>
>> If you need an input flowfile, you're probably better off with
>> ExecuteStreamCommand than ExecuteScript for this use case.
>> ExecuteStreamCommand is much like ExecuteProcess but it accepts input flow
>> files.
>>
>> Regards,
>> Matt
>>
>> > On Aug 17, 2016, at 6:49 PM, koustav choudhuri 
>> wrote:
>> >
>> > HI All
>> >
>> > I am trying to execute 2 different shell scripts ( calling a sqoop job
>> and calling a hive script) in a sequence. Can I achieve this through NIFI ?
>> >
>> > I tried to do it through ExecuteProcess Processor but the issue is
>> Execute Processor does not accept an input.
>> >
>> > Then I moved to ExecuteScript :
>> > Created 2 different python scripts, 1 calling the Sqoop job and the
>> other calling the Hive script and arranged them in 2 different
>> ExecuteScript Processors but looks like it doesnt work.
>> >
>> > Can you guys tell me if this is achievable in NIFI? Can I call
>> Sqoop/HIVE/ SPARK jobs residing on other servers in a sequence or flow with
>> NIFI?
>> >
>> >
>> > Thanks
>> > Koustav
>>
>

Re: Query related to ExecuteScript

2016-08-17 Thread koustav choudhuri 
Matt

Thanks for Replying , but I am still struggling with ExecuteScript.


[image: Inline image 1]


in that ExecuteScript all I have is :

Script - set to Python
Script Body :
from sys import argv

filename = "nifi_output/nifi_test.txt"
target = open(filename, 'w')
target.write("My Name is Bond")
target.close()

When I run this , Nifi hangs for some reason. Am I doing something grossly
wrong ?






On Wed, Aug 17, 2016 at 3:57 PM, Matt Burgess  wrote:

> If you need an input flowfile, you're probably better off with
> ExecuteStreamCommand than ExecuteScript for this use case.
> ExecuteStreamCommand is much like ExecuteProcess but it accepts input flow
> files.
>
> Regards,
> Matt
>
> > On Aug 17, 2016, at 6:49 PM, koustav choudhuri 
> wrote:
> >
> > HI All
> >
> > I am trying to execute 2 different shell scripts ( calling a sqoop job
> and calling a hive script) in a sequence. Can I achieve this through NIFI ?
> >
> > I tried to do it through ExecuteProcess Processor but the issue is
> Execute Processor does not accept an input.
> >
> > Then I moved to ExecuteScript :
> > Created 2 different python scripts, 1 calling the Sqoop job and the
> other calling the Hive script and arranged them in 2 different
> ExecuteScript Processors but looks like it doesnt work.
> >
> > Can you guys tell me if this is achievable in NIFI? Can I call
> Sqoop/HIVE/ SPARK jobs residing on other servers in a sequence or flow with
> NIFI?
> >
> >
> > Thanks
> > Koustav
>

Re: Query related to ExecuteScript

2016-08-17 Thread Matt Burgess 
If you need an input flowfile, you're probably better off with 
ExecuteStreamCommand than ExecuteScript for this use case. ExecuteStreamCommand 
is much like ExecuteProcess but it accepts input flow files.

Regards,
Matt

> On Aug 17, 2016, at 6:49 PM, koustav choudhuri  wrote:
> 
> HI All 
> 
> I am trying to execute 2 different shell scripts ( calling a sqoop job and 
> calling a hive script) in a sequence. Can I achieve this through NIFI ?
> 
> I tried to do it through ExecuteProcess Processor but the issue is Execute 
> Processor does not accept an input. 
> 
> Then I moved to ExecuteScript :
> Created 2 different python scripts, 1 calling the Sqoop job and the other 
> calling the Hive script and arranged them in 2 different ExecuteScript 
> Processors but looks like it doesnt work.
> 
> Can you guys tell me if this is achievable in NIFI? Can I call Sqoop/HIVE/ 
> SPARK jobs residing on other servers in a sequence or flow with NIFI?
> 
> 
> Thanks
> Koustav