Re: Groovy script
Hello.
Not sure, but I already did something like this.
But unlike Java, I defined variable with keyword def
flowfile = session.get()
> if(!flowfile) return
> def filePath = flowfile.getAttribute('file_path')
> def file = new File(file_path)
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> } else {
> session.transfer(flowfile, REL_SUCCESS)
> }
Could be something like this.
Le mer. 24 févr. 2021 à 17:47, Tomislav Novosel <
tomislav.novo...@clearpeaks.com> a écrit :
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> }
> else{
> session.transfer(flowfile, REL_SUCCESS)
> }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of
> them go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
>
RE: Groovy script
Hi Mike,
attribute 'file_path' is not pointing to folder only, it has value
/path/to/filename, so it is like /opt/data/folder/filename.txt. The attribute
value is ok, I double checked.
Tom
-Original Message-
From: Mike Thomsen
Sent: 24 February 2021 18:00
To: users@nifi.apache.org
Subject: Re: Groovy script
If file_path is pointing to a folder as you said, it's going to check for the
folder's existence. The fact that it's failing to return true there suggests
that something is wrong with the path in the file_path attribute.
On Wed, Feb 24, 2021 at 11:47 AM Tomislav Novosel
wrote:
>
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE) } else{
> session.transfer(flowfile, REL_SUCCESS) }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of
> them go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
Re: Groovy script
If file_path is pointing to a folder as you said, it's going to check
for the folder's existence. The fact that it's failing to return true
there suggests that something is wrong with the path in the file_path
attribute.
On Wed, Feb 24, 2021 at 11:47 AM Tomislav Novosel
wrote:
>
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> }
> else{
> session.transfer(flowfile, REL_SUCCESS)
> }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of them
> go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
Groovy script
Hi guys,
I want to check if file exists with this groovy script:
flowfile = session.get()
if(!flowfile) return
file_path = flowfile.getAttribute('file_path')
File file = new File(file_path)
if(file.exists()){
session.transfer(flowfile, REL_FAILURE)
}
else{
session.transfer(flowfile, REL_SUCCESS)
}
and to route all files which exist to FAILURE relationship, but all of them go
to SUCCESS, file is for sure in the folder
'file_path', I checked.
What am I doing wrong?
Thanks,
Tom
Re: NIFI Groovy Script - Filter file names and get count
Thanks Mike.
Sent from my iPhone
> On Oct 30, 2020, at 5:10 AM, Mike Thomsen wrote:
>
> You need to use the listFiles() that has a FilenameFilter interface in it:
>
> https://docs.oracle.com/javase/8/docs/api/java/io/File.html#listFiles-java.io.FilenameFilter-
>
>> On Thu, Oct 29, 2020 at 5:12 PM KhajaAsmath Mohammed
>> wrote:
>>
>> Hi,
>>
>> I have a requirement where I need to get the file count from the path using
>> the groovy script.
>>
>> I came up with the below but unable to filter and count only txt files . Any
>> suggestions please?
>>
>> import org.apache.commons.io.IOUtils
>> import java.nio.charset.*;
>> import java.io.*;
>>
>> def flowFile = session.get()
>> if(!flowFile) return
>>
>>
>> def eahpath = flowFile.getAttribute("eahpath")
>> def count = new File(eahpath).listFiles().size();// I need to filter
>> only txt files and get count
>> flowFile=session.putAttribute(flowFile,"eahfilecount",count+"");
>> def fail = false
>> if(fail){
>> session.transfer(flowFile, REL_FAILURE)
>> fail = false
>> } else {
>> session.transfer(flowFile, REL_SUCCESS)
>> }
>>
>> Thanks,
>> Asmath
Re: NIFI Groovy Script - Filter file names and get count
You need to use the listFiles() that has a FilenameFilter interface in it:
https://docs.oracle.com/javase/8/docs/api/java/io/File.html#listFiles-java.io.FilenameFilter-
On Thu, Oct 29, 2020 at 5:12 PM KhajaAsmath Mohammed
wrote:
>
> Hi,
>
> I have a requirement where I need to get the file count from the path using
> the groovy script.
>
> I came up with the below but unable to filter and count only txt files . Any
> suggestions please?
>
> import org.apache.commons.io.IOUtils
> import java.nio.charset.*;
> import java.io.*;
>
> def flowFile = session.get()
> if(!flowFile) return
>
>
> def eahpath = flowFile.getAttribute("eahpath")
> def count = new File(eahpath).listFiles().size();// I need to filter only
> txt files and get count
> flowFile=session.putAttribute(flowFile,"eahfilecount",count+"");
> def fail = false
> if(fail){
> session.transfer(flowFile, REL_FAILURE)
> fail = false
> } else {
> session.transfer(flowFile, REL_SUCCESS)
> }
>
> Thanks,
> Asmath
NIFI Groovy Script - Filter file names and get count
Hi,
I have a requirement where I need to get the file count from the path using
the groovy script.
I came up with the below but unable to filter and count only txt files .
Any suggestions please?
import org.apache.commons.io.IOUtils
import java.nio.charset.*;
import java.io.*;
def flowFile = session.get()
if(!flowFile) return
def eahpath = flowFile.getAttribute("eahpath")
*def count = new File(eahpath).listFiles().size();// I need to filter
only txt files and get count*
flowFile=session.putAttribute(flowFile,"eahfilecount",count+"");
def fail = false
if(fail){
session.transfer(flowFile, REL_FAILURE)
fail = false
} else {
session.transfer(flowFile, REL_SUCCESS)
}
Thanks,
Asmath
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Matt, Chris, et al.
I thought I had replied to this thread already noting that I created NIFI-7447
[1] to look at how to address this at the framework level.
I just wanted to close the loop on this, though. This issue was very easy to
replicate in 1.11.4. I tested this scenario with the fix for NIFI-7447 and I
tested many other scenarios around Record processors, schema registry, scripted
and unscripted record writers, as well as some issues around JMS processors
that had crept up before. The fix for NIFI-7447 has now been merged to master,
and I believe it addresses all of these concerns.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7447
On May 11, 2020, at 9:27 PM, Matt Burgess
mailto:mattyb...@apache.org>> wrote:
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess
mailto:mattyb...@apache.org>> wrote:
Chris,
There's definitely something funky going on there, the script doesn't
get the same classloader chain that the ScriptedRecordSetWriter (that
loads the script) does, instead it gets one with the standard NAR as
the parent instead of the scripting NAR. I'm looking into it now.
BTW for scripted component issues, you might be better off emailing
the dev list, there may be more folks in there familiar with the NiFi
code and scripting languages and such. Having said that, we can
maintain this thread until we get to the bottom of the issue.
Regards,
Matt
On Mon, May 11, 2020 at 3:47 AM Chris Herssens
mailto:chris.herss...@gmail.com>> wrote:
Hello,
I try to implement a groovy script where I'm using jsonOutput.
With nifi 1.5 the script works, but If I'm try to use the same groovy script
with nifi 1.11.4, I get
"unable to load FastStringService"
example code :
class GroovyRecordSetWriter implements RecordSetWriter {
@Override
WriteResult write(Record r) throws IOException {
...
def j = JsonOutput.toJson([
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess wrote:
>
> Chris,
>
> There's definitely something funky going on there, the script doesn't
> get the same classloader chain that the ScriptedRecordSetWriter (that
> loads the script) does, instead it gets one with the standard NAR as
> the parent instead of the scripting NAR. I'm looking into it now.
>
> BTW for scripted component issues, you might be better off emailing
> the dev list, there may be more folks in there familiar with the NiFi
> code and scripting languages and such. Having said that, we can
> maintain this thread until we get to the bottom of the issue.
>
> Regards,
> Matt
>
> On Mon, May 11, 2020 at 3:47 AM Chris Herssens
> wrote:
> >
> > Hello,
> >
> > I try to implement a groovy script where I'm using jsonOutput.
> > With nifi 1.5 the script works, but If I'm try to use the same groovy
> > script with nifi 1.11.4, I get
> > "unable to load FastStringService"
> >
> > example code :
> >
> > class GroovyRecordSetWriter implements RecordSetWriter {
> >
> > @Override
> > WriteResult write(Record r) throws IOException {
> > ...
> > def j = JsonOutput.toJson([name: 'John Doe', age: 42])
> > out.write(j.getBytes())
> >
> > ...
> >
> > Regards,
> > Chris
groovy script in nifi 1.11 : unable to loasd FastStringService
Hello,
I try to implement a groovy script where I'm using jsonOutput.
With nifi 1.5 the script works, but If I'm try to use the same groovy
script with nifi 1.11.4, I get
"unable to load FastStringService"
example code :
class GroovyRecordSetWriter implements RecordSetWriter {
@Override
WriteResult write(Record r) throws IOException {
...
def j = JsonOutput.toJson([name: 'John Doe', age: 42])
out.write(j.getBytes())
...
Regards,
Chris
Re: How to merge two rows data into single row in groovy script/nifi processors?
If your data is always set up such that the first line has a row of
data and the second has additional data, you can set up a variable
outside the eachLine(), then if the line number is even (because it's
zero-based) you store the line and if it is odd you append it to the
previous line and output it:
def myLine = ''
reader.eachLine {line, lineNum ->
if(lineNum % 2 == 0) {
myLine = line
} else {
writer << (myLine + line + '\n')
}
}
Please let me know if I've misunderstood your question.
Regards,
Matt
On Wed, Aug 16, 2017 at 9:06 AM, prabhu Mahendran
<prabhuu161...@gmail.com> wrote:
> my data is in form of unstructured data in which having end of column stored
> in two rows like below.
>
> UID|Name|ID|Mail
> 1|Ester|991|sd
> gmail
> 2|Siva|992|siva
> hotmail
> 3|Hari|993|hi gmail
>
> Some rows in data has been fulfilled but some rows to avoid those two line
> data into single line like below.
>
> UID|Name|ID|Mail
> 1|Ester|991|sd gmail
> 2|Siva|992|siva hotmail
> 3|Hari|993|hi gmail
> I don't know that nifi processors in which helpful for this conversion.
>
> But i have tried following Groovy Script to read lines and not able to find
> way to combine spitted rows into single row.
>
> def flowfile = session.get()
> if(!flowfile)return
> flowfile = session.write(flowfile, {rawIn, rawOut->
> // ## transform streams into reader and writer
> rawIn.withReader("UTF-8"){reader->
> rawOut.withWriter("UTF-8"){writer->
> reader.eachLine{line, lineNum->
> if(!line.isEmpty())
> {// ## let use regular expression to transform each line
> writer << line << '\n'
> }
> }
> }
> }
> } as StreamCallback)
> session.transfer(flowfile, REL_SUCCESS)
>
> Can anyone suggest me idea convert my data into requirement?
Re: Issue with Groovy script
Thanks both - I assumed if I included the root directory that would not
only pick up the http-builder.jar but also the dependencies. Including the
dependencies directory fixed the issue.
Much appreciated,
Mike
On 4 May 2017 at 20:09, Matt Burgess <mattyb...@apache.org> wrote:
> Mike,
>
> To follow up on Andy's question, you will likely need more than just
> the http-builder JAR, I don't believe it is shaded (aka "fat JAR"). I
> have the "http-builder-0.7-all.zip" unzipped to a folder, and it has
> the http-builder-0.7.jar at the root level, but then a "dependencies"
> folder as well. If you have something similar, you will want to add
> the JAR and the dependencies folder to the Module Directory property.
>
> Regards,
> Matt
>
> On Thu, May 4, 2017 at 3:04 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
> > Mike,
> >
> > When you say you’ve “included the http-builder jar as a dependency” do
> you
> > mean you provided the location of the directory containing that JAR as
> the
> > Module Path in the ExecuteScript processor?
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On May 4, 2017, at 1:58 PM, Mike Harding <mikeyhard...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I'm trying to run a simple groovy script in ExecuteScript processor to
> make
> > a HTTP GET request (I understand their are processors get this but I'm
> just
> > exploring Groovy at the minute).
> >
> >> import groovyx.net.http.HTTPBuilder
> >> flowFile = session.get()
> >> def http = new HTTPBuilder('https://google.com')
> >> def html = http.get(path : '/search', query : [q:'waffles'])
> >> log.warn(html)
> >> session.transfer(flowFile, REL_SUCCESS)
> >
> >
> > Ive included the http-builder jar as a dependency but I'm getting the
> error:
> >
> >
> >
> > I'm not new to NiFi but new to using Groovy. I've tried import
> > org.apache.http.* but that doesn't help. I'm assuming that the missing
> class
> > library is a default library in Groovy?
> >
> > Any help much appreciated,
> > Mike
> >
> >
>
Re: Issue with Groovy script
Mike,
To follow up on Andy's question, you will likely need more than just
the http-builder JAR, I don't believe it is shaded (aka "fat JAR"). I
have the "http-builder-0.7-all.zip" unzipped to a folder, and it has
the http-builder-0.7.jar at the root level, but then a "dependencies"
folder as well. If you have something similar, you will want to add
the JAR and the dependencies folder to the Module Directory property.
Regards,
Matt
On Thu, May 4, 2017 at 3:04 PM, Andy LoPresto <alopre...@apache.org> wrote:
> Mike,
>
> When you say you’ve “included the http-builder jar as a dependency” do you
> mean you provided the location of the directory containing that JAR as the
> Module Path in the ExecuteScript processor?
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On May 4, 2017, at 1:58 PM, Mike Harding <mikeyhard...@gmail.com> wrote:
>
> Hi all,
>
> I'm trying to run a simple groovy script in ExecuteScript processor to make
> a HTTP GET request (I understand their are processors get this but I'm just
> exploring Groovy at the minute).
>
>> import groovyx.net.http.HTTPBuilder
>> flowFile = session.get()
>> def http = new HTTPBuilder('https://google.com')
>> def html = http.get(path : '/search', query : [q:'waffles'])
>> log.warn(html)
>> session.transfer(flowFile, REL_SUCCESS)
>
>
> Ive included the http-builder jar as a dependency but I'm getting the error:
>
>
>
> I'm not new to NiFi but new to using Groovy. I've tried import
> org.apache.http.* but that doesn't help. I'm assuming that the missing class
> library is a default library in Groovy?
>
> Any help much appreciated,
> Mike
>
>
Re: Issue with Groovy script
Mike,
When you say you’ve “included the http-builder jar as a dependency” do you mean
you provided the location of the directory containing that JAR as the Module
Path in the ExecuteScript processor?
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On May 4, 2017, at 1:58 PM, Mike Harding <mikeyhard...@gmail.com> wrote:
>
> Hi all,
>
> I'm trying to run a simple groovy script in ExecuteScript processor to make a
> HTTP GET request (I understand their are processors get this but I'm just
> exploring Groovy at the minute).
>
> import groovyx.net.http.HTTPBuilder
> flowFile = session.get()
> def http = new HTTPBuilder('https://google.com <https://google.com/>')
> def html = http.get(path : '/search', query : [q:'waffles'])
> log.warn(html)
> session.transfer(flowFile, REL_SUCCESS)
>
> Ive included the http-builder jar as a dependency but I'm getting the error:
>
>
>
> I'm not new to NiFi but new to using Groovy. I've tried import
> org.apache.http.* but that doesn't help. I'm assuming that the missing class
> library is a default library in Groovy?
>
> Any help much appreciated,
> Mike
signature.asc
Description: Message signed with OpenPGP using GPGMail
Issue with Groovy script
Hi all,
I'm trying to run a simple groovy script in ExecuteScript processor to make
a HTTP GET request (I understand their are processors get this but I'm just
exploring Groovy at the minute).
import groovyx.net.http.HTTPBuilder
> flowFile = session.get()
> def http = new HTTPBuilder('https://google.com')
> def html = http.get(path : '/search', query : [q:'waffles'])
> log.warn(html)
> session.transfer(flowFile, REL_SUCCESS)
Ive included the http-builder jar as a dependency but I'm getting the error:
[image: Inline images 1]
I'm not new to NiFi but new to using Groovy. I've tried import
org.apache.http.* but that doesn't help. I'm assuming that the missing
class library is a default library in Groovy?
Any help much appreciated,
Mike
