Re: Groovy script
Hello.
Not sure, but I already did something like this.
But unlike Java, I defined variable with keyword def
flowfile = session.get()
> if(!flowfile) return
> def filePath = flowfile.getAttribute('file_path')
> def file = new File(file_path)
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> } else {
> session.transfer(flowfile, REL_SUCCESS)
> }
Could be something like this.
Le mer. 24 févr. 2021 à 17:47, Tomislav Novosel <
tomislav.novo...@clearpeaks.com> a écrit :
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> }
> else{
> session.transfer(flowfile, REL_SUCCESS)
> }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of
> them go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
>
RE: Groovy script
Hi Mike,
attribute 'file_path' is not pointing to folder only, it has value
/path/to/filename, so it is like /opt/data/folder/filename.txt. The attribute
value is ok, I double checked.
Tom
-Original Message-
From: Mike Thomsen
Sent: 24 February 2021 18:00
To: users@nifi.apache.org
Subject: Re: Groovy script
If file_path is pointing to a folder as you said, it's going to check for the
folder's existence. The fact that it's failing to return true there suggests
that something is wrong with the path in the file_path attribute.
On Wed, Feb 24, 2021 at 11:47 AM Tomislav Novosel
wrote:
>
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE) } else{
> session.transfer(flowfile, REL_SUCCESS) }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of
> them go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
Re: Groovy script
If file_path is pointing to a folder as you said, it's going to check
for the folder's existence. The fact that it's failing to return true
there suggests that something is wrong with the path in the file_path
attribute.
On Wed, Feb 24, 2021 at 11:47 AM Tomislav Novosel
wrote:
>
> Hi guys,
>
>
>
> I want to check if file exists with this groovy script:
>
>
>
> flowfile = session.get()
> if(!flowfile) return
>
> file_path = flowfile.getAttribute('file_path')
> File file = new File(file_path)
>
> if(file.exists()){
> session.transfer(flowfile, REL_FAILURE)
> }
> else{
> session.transfer(flowfile, REL_SUCCESS)
> }
>
>
>
> and to route all files which exist to FAILURE relationship, but all of them
> go to SUCCESS, file is for sure in the folder
>
> ‘file_path’, I checked.
>
>
>
> What am I doing wrong?
>
>
>
> Thanks,
>
>
>
> Tom
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Matt, Chris, et al.
I thought I had replied to this thread already noting that I created NIFI-7447
[1] to look at how to address this at the framework level.
I just wanted to close the loop on this, though. This issue was very easy to
replicate in 1.11.4. I tested this scenario with the fix for NIFI-7447 and I
tested many other scenarios around Record processors, schema registry, scripted
and unscripted record writers, as well as some issues around JMS processors
that had crept up before. The fix for NIFI-7447 has now been merged to master,
and I believe it addresses all of these concerns.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7447
On May 11, 2020, at 9:27 PM, Matt Burgess
mailto:mattyb...@apache.org>> wrote:
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess
mailto:mattyb...@apache.org>> wrote:
Chris,
There's definitely something funky going on there, the script doesn't
get the same classloader chain that the ScriptedRecordSetWriter (that
loads the script) does, instead it gets one with the standard NAR as
the parent instead of the scripting NAR. I'm looking into it now.
BTW for scripted component issues, you might be better off emailing
the dev list, there may be more folks in there familiar with the NiFi
code and scripting languages and such. Having said that, we can
maintain this thread until we get to the bottom of the issue.
Regards,
Matt
On Mon, May 11, 2020 at 3:47 AM Chris Herssens
mailto:chris.herss...@gmail.com>> wrote:
Hello,
I try to implement a groovy script where I'm using jsonOutput.
With nifi 1.5 the script works, but If I'm try to use the same groovy script
with nifi 1.11.4, I get
"unable to load FastStringService"
example code :
class GroovyRecordSetWriter implements RecordSetWriter {
@Override
WriteResult write(Record r) throws IOException {
...
def j = JsonOutput.toJson([name: 'John Doe', age: 42])
out.write(j.getBytes())
...
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess wrote:
>
> Chris,
>
> There's definitely something funky going on there, the script doesn't
> get the same classloader chain that the ScriptedRecordSetWriter (that
> loads the script) does, instead it gets one with the standard NAR as
> the parent instead of the scripting NAR. I'm looking into it now.
>
> BTW for scripted component issues, you might be better off emailing
> the dev list, there may be more folks in there familiar with the NiFi
> code and scripting languages and such. Having said that, we can
> maintain this thread until we get to the bottom of the issue.
>
> Regards,
> Matt
>
> On Mon, May 11, 2020 at 3:47 AM Chris Herssens
> wrote:
> >
> > Hello,
> >
> > I try to implement a groovy script where I'm using jsonOutput.
> > With nifi 1.5 the script works, but If I'm try to use the same groovy
> > script with nifi 1.11.4, I get
> > "unable to load FastStringService"
> >
> > example code :
> >
> > class GroovyRecordSetWriter implements RecordSetWriter {
> >
> > @Override
> > WriteResult write(Record r) throws IOException {
> > ...
> > def j = JsonOutput.toJson([name: 'John Doe', age: 42])
> > out.write(j.getBytes())
> >
> > ...
> >
> > Regards,
> > Chris
