Re: File causes unexpected stoppage in play
James
This article should help you get started
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
especially since you only have to go through the first part only “Maven
Processor Archetype” (I am assuming you are familiar with maven).
Also, make sure that you pick the right NiFi version when executing 'mvn
archetype. . .’ command. The current website example use 1.0.0. Change it to
0.6.1 (see below).
$> mvn archetype:generate -DarchetypeGroupId=org.apache.nifi
-DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=0.6.1
-DnifiVersion=0.6.1
After that you may want to copy the UnpackContent into some package within he
bundle, rename it, adjust the class path to ensure it compiles and build. Make
sure you add/modify your processor’s name in
src/main/resources/META-INF/services/org.apache.nifi.processor.Processor file.
Iet me know if you run into any issues.
Cheers
Oleg
On Dec 14, 2016, at 3:08 PM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
I've never tried to create and deploy my own custom bundle but would like to
try #3 as you recommend Oleg. If you can offer guidance to that process let me
try that. Thank you. -Jim
On Wed, Dec 14, 2016 at 2:57 PM, Oleg Zhurakousky
mailto:ozhurakou...@hortonworks.com>> wrote:
James
Not sure what kind of liberties you have with your environment, but here is
what you can do outside of full upgrade.
1. You can download and drop the bundle (NAR) from NiFi 0.7 release into 0.6
release. That should work (just replace it in the ‘lib’ directory of NiFi
home). However, this particular component is located in ’nifi-standard-nar’
which means it contains majority of the processors and controller services that
come with NiFi. So while upgrading one component you will essentially upgrade
almost all and I am not sure what it may lead to, but you can definitely try.
2. You can certainly checkout the NiFi 0.6.1 source from the repo and apply
the same patch on UpdateContent, build NAR and replace it as described in the
step above, Even though you’ll still be replacing the entire NAR the changes
are only going to be in that single component.
3. You can create your own custom bundle (NAR), copy the UnpackContent code
from 0.7 branch, rename the actual class to something different (e.g.,
UnzipContent) and then deploy it as your own custom bundle and use it in your
flow.
4. You can also apply patch through fixing the byte code in flight but that
would require you to write javaagent and be familiar with byte code
manipulations frameworks. Not advisable. . .
Honestly, aside from full upgrade the only viable option here is #3 since it
gives you full control and complete isolation from every other component in
NiFi.
Let me know what you think so we can guide you thru the process.
Cheers
Oleg
On Dec 14, 2016, at 2:18 PM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
Oleg, I am bound to NiFi 0.6.1 by enterprise application dependencies. So this
fix will not be in my baseline if I understand you correctly. Let me ask you
this: is there any way I can build this into my code baseline - either through
a code mod and rebuild or as a custom plugin feature specific to my flow?
Thanks very much for your help.
On Wed, Dec 14, 2016 at 10:59 AM, Oleg Zhurakousky
mailto:ozhurakou...@hortonworks.com>> wrote:
James
Could you also let us know what version of NiFi you are using? The issue with
properly handling InvalidPathException was fixed in NiFi 0.7.0 as part of
https://issues.apache.org/jira/browse/NIFI-920
It essentially has this catch block:
} catch (final ProcessException | InvalidPathException e) {
logger.error("Unable to unpack {} due to {}; routing to failure", new
Object[]{flowFile, e});
session.transfer(flowFile, REL_FAILURE);
So I am wondering if you are on the proviso release?
Cheers
Oleg
On Dec 14, 2016, at 10:49 AM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:
James,
Can you please share the full log entry for that failure. It is
possible the processor is not catching the exception properly and
routing the data to failure. It might simply be letting the exception
loose thus the framework detects the issue and rollsback the session
and yields the processor.
Likely an easy thing to fix in the processor but please do share the
full nifi-app.log entry for this.
Thanks
Joe
On Wed, Dec 14, 2016 at 10:47 AM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
Hello. Am wondering if anyone knows how to overcome a challenge with
unmappable file characters? I have used a GetFile processor to bring a large
number of zip files into a NiFi flow. All read in successfully. I try to
then use an UnpackContent processor to unzip the files to their individual
file members. Most work just fine. However there appears to be a file that
throws this error in UnpackContent:
failed to process session due to java.nio.file.InvalidPathException:
Malformed input or input contains unmappabl
Re: File causes unexpected stoppage in play
I've never tried to create and deploy my own custom bundle but would like
to try #3 as you recommend Oleg. If you can offer guidance to that process
let me try that. Thank you. -Jim
On Wed, Dec 14, 2016 at 2:57 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:
> James
>
> Not sure what kind of liberties you have with your environment, but here
> is what you can do outside of full upgrade.
>
> 1. You can download and drop the bundle (NAR) from NiFi 0.7 release into
> 0.6 release. That should work (just replace it in the ‘lib’ directory of
> NiFi home). However, this particular component is located in
> ’nifi-standard-nar’ which means it contains majority of the processors and
> controller services that come with NiFi. So while upgrading one component
> you will essentially upgrade almost all and I am not sure what it may lead
> to, but you can definitely try.
>
> 2. You can certainly checkout the NiFi 0.6.1 source from the repo and
> apply the same patch on UpdateContent, build NAR and replace it as
> described in the step above, Even though you’ll still be replacing the
> entire NAR the changes are only going to be in that single component.
>
> 3. You can create your own custom bundle (NAR), copy the UnpackContent
> code from 0.7 branch, rename the actual class to something different (e.g.,
> UnzipContent) and then deploy it as your own custom bundle and use it in
> your flow.
>
> 4. You can also apply patch through fixing the byte code in flight but
> that would require you to write javaagent and be familiar with byte code
> manipulations frameworks. Not advisable. . .
>
> Honestly, aside from full upgrade the only viable option here is #3 since
> it gives you full control and complete isolation from every other component
> in NiFi.
>
> Let me know what you think so we can guide you thru the process.
>
> Cheers
> Oleg
>
> On Dec 14, 2016, at 2:18 PM, James McMahon wrote:
>
> Oleg, I am bound to NiFi 0.6.1 by enterprise application dependencies. So
> this fix will not be in my baseline if I understand you correctly. Let me
> ask you this: is there any way I can build this into my code baseline -
> either through a code mod and rebuild or as a custom plugin feature
> specific to my flow? Thanks very much for your help.
>
> On Wed, Dec 14, 2016 at 10:59 AM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>> James
>>
>> Could you also let us know what version of NiFi you are using? The issue
>> with properly handling InvalidPathException was fixed in NiFi 0.7.0 as part
>> of https://issues.apache.org/jira/browse/NIFI-920
>> It essentially has this catch block:
>> } catch (final ProcessException | InvalidPathException e) {
>> logger.error("Unable to unpack {} due to {}; routing to failure",
>> new Object[]{flowFile, e});
>> session.transfer(flowFile, REL_FAILURE);
>>
>> So I am wondering if you are on the proviso release?
>> Cheers
>> Oleg
>>
>> On Dec 14, 2016, at 10:49 AM, Joe Witt wrote:
>>
>> James,
>>
>> Can you please share the full log entry for that failure. It is
>> possible the processor is not catching the exception properly and
>> routing the data to failure. It might simply be letting the exception
>> loose thus the framework detects the issue and rollsback the session
>> and yields the processor.
>>
>> Likely an easy thing to fix in the processor but please do share the
>> full nifi-app.log entry for this.
>>
>> Thanks
>> Joe
>>
>> On Wed, Dec 14, 2016 at 10:47 AM, James McMahon
>> wrote:
>>
>> Hello. Am wondering if anyone knows how to overcome a challenge with
>> unmappable file characters? I have used a GetFile processor to bring a
>> large
>> number of zip files into a NiFi flow. All read in successfully. I try to
>> then use an UnpackContent processor to unzip the files to their individual
>> file members. Most work just fine. However there appears to be a file that
>> throws this error in UnpackContent:
>>
>> failed to process session due to java.nio.file.InvalidPathException:
>> Malformed input or input contains unmappable characters
>>
>> My processing stalls. Nothing else flows. What is the proper way to
>> configure the UnpackContent processor step so that it shuttle such files
>> off
>> to the side when it encounters them, and permits the other files waiting
>> in
>> queue to process? I do now have a "failure" path established for my
>> UnpackContent processor, but for some reason it does not send these
>> problem
>> files down that path. I suspect it may be because the zip files does
>> unpack
>> successfully but the underlying file(s) within the zip cause processing to
>> choke.
>>
>> How can I engineer a flow to overcome this challenge? Thanks in advance
>> for
>> your help.
>>
>>
>>
>>
>
>
Re: File causes unexpected stoppage in play
James
Not sure what kind of liberties you have with your environment, but here is
what you can do outside of full upgrade.
1. You can download and drop the bundle (NAR) from NiFi 0.7 release into 0.6
release. That should work (just replace it in the ‘lib’ directory of NiFi
home). However, this particular component is located in ’nifi-standard-nar’
which means it contains majority of the processors and controller services that
come with NiFi. So while upgrading one component you will essentially upgrade
almost all and I am not sure what it may lead to, but you can definitely try.
2. You can certainly checkout the NiFi 0.6.1 source from the repo and apply
the same patch on UpdateContent, build NAR and replace it as described in the
step above, Even though you’ll still be replacing the entire NAR the changes
are only going to be in that single component.
3. You can create your own custom bundle (NAR), copy the UnpackContent code
from 0.7 branch, rename the actual class to something different (e.g.,
UnzipContent) and then deploy it as your own custom bundle and use it in your
flow.
4. You can also apply patch through fixing the byte code in flight but that
would require you to write javaagent and be familiar with byte code
manipulations frameworks. Not advisable. . .
Honestly, aside from full upgrade the only viable option here is #3 since it
gives you full control and complete isolation from every other component in
NiFi.
Let me know what you think so we can guide you thru the process.
Cheers
Oleg
On Dec 14, 2016, at 2:18 PM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
Oleg, I am bound to NiFi 0.6.1 by enterprise application dependencies. So this
fix will not be in my baseline if I understand you correctly. Let me ask you
this: is there any way I can build this into my code baseline - either through
a code mod and rebuild or as a custom plugin feature specific to my flow?
Thanks very much for your help.
On Wed, Dec 14, 2016 at 10:59 AM, Oleg Zhurakousky
mailto:ozhurakou...@hortonworks.com>> wrote:
James
Could you also let us know what version of NiFi you are using? The issue with
properly handling InvalidPathException was fixed in NiFi 0.7.0 as part of
https://issues.apache.org/jira/browse/NIFI-920
It essentially has this catch block:
} catch (final ProcessException | InvalidPathException e) {
logger.error("Unable to unpack {} due to {}; routing to failure", new
Object[]{flowFile, e});
session.transfer(flowFile, REL_FAILURE);
So I am wondering if you are on the proviso release?
Cheers
Oleg
On Dec 14, 2016, at 10:49 AM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:
James,
Can you please share the full log entry for that failure. It is
possible the processor is not catching the exception properly and
routing the data to failure. It might simply be letting the exception
loose thus the framework detects the issue and rollsback the session
and yields the processor.
Likely an easy thing to fix in the processor but please do share the
full nifi-app.log entry for this.
Thanks
Joe
On Wed, Dec 14, 2016 at 10:47 AM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
Hello. Am wondering if anyone knows how to overcome a challenge with
unmappable file characters? I have used a GetFile processor to bring a large
number of zip files into a NiFi flow. All read in successfully. I try to
then use an UnpackContent processor to unzip the files to their individual
file members. Most work just fine. However there appears to be a file that
throws this error in UnpackContent:
failed to process session due to java.nio.file.InvalidPathException:
Malformed input or input contains unmappable characters
My processing stalls. Nothing else flows. What is the proper way to
configure the UnpackContent processor step so that it shuttle such files off
to the side when it encounters them, and permits the other files waiting in
queue to process? I do now have a "failure" path established for my
UnpackContent processor, but for some reason it does not send these problem
files down that path. I suspect it may be because the zip files does unpack
successfully but the underlying file(s) within the zip cause processing to
choke.
How can I engineer a flow to overcome this challenge? Thanks in advance for
your help.
Re: File causes unexpected stoppage in play
Oleg, I am bound to NiFi 0.6.1 by enterprise application dependencies. So
this fix will not be in my baseline if I understand you correctly. Let me
ask you this: is there any way I can build this into my code baseline -
either through a code mod and rebuild or as a custom plugin feature
specific to my flow? Thanks very much for your help.
On Wed, Dec 14, 2016 at 10:59 AM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:
> James
>
> Could you also let us know what version of NiFi you are using? The issue
> with properly handling InvalidPathException was fixed in NiFi 0.7.0 as part
> of https://issues.apache.org/jira/browse/NIFI-920
> It essentially has this catch block:
> } catch (final ProcessException | InvalidPathException e) {
> logger.error("Unable to unpack {} due to {}; routing to failure", new
> Object[]{flowFile, e});
> session.transfer(flowFile, REL_FAILURE);
>
> So I am wondering if you are on the proviso release?
> Cheers
> Oleg
>
> On Dec 14, 2016, at 10:49 AM, Joe Witt wrote:
>
> James,
>
> Can you please share the full log entry for that failure. It is
> possible the processor is not catching the exception properly and
> routing the data to failure. It might simply be letting the exception
> loose thus the framework detects the issue and rollsback the session
> and yields the processor.
>
> Likely an easy thing to fix in the processor but please do share the
> full nifi-app.log entry for this.
>
> Thanks
> Joe
>
> On Wed, Dec 14, 2016 at 10:47 AM, James McMahon
> wrote:
>
> Hello. Am wondering if anyone knows how to overcome a challenge with
> unmappable file characters? I have used a GetFile processor to bring a
> large
> number of zip files into a NiFi flow. All read in successfully. I try to
> then use an UnpackContent processor to unzip the files to their individual
> file members. Most work just fine. However there appears to be a file that
> throws this error in UnpackContent:
>
> failed to process session due to java.nio.file.InvalidPathException:
> Malformed input or input contains unmappable characters
>
> My processing stalls. Nothing else flows. What is the proper way to
> configure the UnpackContent processor step so that it shuttle such files
> off
> to the side when it encounters them, and permits the other files waiting in
> queue to process? I do now have a "failure" path established for my
> UnpackContent processor, but for some reason it does not send these problem
> files down that path. I suspect it may be because the zip files does unpack
> successfully but the underlying file(s) within the zip cause processing to
> choke.
>
> How can I engineer a flow to overcome this challenge? Thanks in advance for
> your help.
>
>
>
>
Re: File causes unexpected stoppage in play
Thank you Joe. While I'm unable to export and share the nifi-app.log file, I can tell you that the processor is throwing the error "...failed to process due to java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /dir/path//_?.jpg " On Wed, Dec 14, 2016 at 10:49 AM, Joe Witt wrote: > James, > > Can you please share the full log entry for that failure. It is > possible the processor is not catching the exception properly and > routing the data to failure. It might simply be letting the exception > loose thus the framework detects the issue and rollsback the session > and yields the processor. > > Likely an easy thing to fix in the processor but please do share the > full nifi-app.log entry for this. > > Thanks > Joe > > On Wed, Dec 14, 2016 at 10:47 AM, James McMahon > wrote: > > Hello. Am wondering if anyone knows how to overcome a challenge with > > unmappable file characters? I have used a GetFile processor to bring a > large > > number of zip files into a NiFi flow. All read in successfully. I try to > > then use an UnpackContent processor to unzip the files to their > individual > > file members. Most work just fine. However there appears to be a file > that > > throws this error in UnpackContent: > > > > failed to process session due to java.nio.file.InvalidPathException: > > Malformed input or input contains unmappable characters > > > > My processing stalls. Nothing else flows. What is the proper way to > > configure the UnpackContent processor step so that it shuttle such files > off > > to the side when it encounters them, and permits the other files waiting > in > > queue to process? I do now have a "failure" path established for my > > UnpackContent processor, but for some reason it does not send these > problem > > files down that path. I suspect it may be because the zip files does > unpack > > successfully but the underlying file(s) within the zip cause processing > to > > choke. > > > > How can I engineer a flow to overcome this challenge? Thanks in advance > for > > your help. >
Re: File causes unexpected stoppage in play
James
Could you also let us know what version of NiFi you are using? The issue with
properly handling InvalidPathException was fixed in NiFi 0.7.0 as part of
https://issues.apache.org/jira/browse/NIFI-920
It essentially has this catch block:
} catch (final ProcessException | InvalidPathException e) {
logger.error("Unable to unpack {} due to {}; routing to failure", new
Object[]{flowFile, e});
session.transfer(flowFile, REL_FAILURE);
So I am wondering if you are on the proviso release?
Cheers
Oleg
On Dec 14, 2016, at 10:49 AM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:
James,
Can you please share the full log entry for that failure. It is
possible the processor is not catching the exception properly and
routing the data to failure. It might simply be letting the exception
loose thus the framework detects the issue and rollsback the session
and yields the processor.
Likely an easy thing to fix in the processor but please do share the
full nifi-app.log entry for this.
Thanks
Joe
On Wed, Dec 14, 2016 at 10:47 AM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
Hello. Am wondering if anyone knows how to overcome a challenge with
unmappable file characters? I have used a GetFile processor to bring a large
number of zip files into a NiFi flow. All read in successfully. I try to
then use an UnpackContent processor to unzip the files to their individual
file members. Most work just fine. However there appears to be a file that
throws this error in UnpackContent:
failed to process session due to java.nio.file.InvalidPathException:
Malformed input or input contains unmappable characters
My processing stalls. Nothing else flows. What is the proper way to
configure the UnpackContent processor step so that it shuttle such files off
to the side when it encounters them, and permits the other files waiting in
queue to process? I do now have a "failure" path established for my
UnpackContent processor, but for some reason it does not send these problem
files down that path. I suspect it may be because the zip files does unpack
successfully but the underlying file(s) within the zip cause processing to
choke.
How can I engineer a flow to overcome this challenge? Thanks in advance for
your help.
Re: File causes unexpected stoppage in play
James, Can you please share the full log entry for that failure. It is possible the processor is not catching the exception properly and routing the data to failure. It might simply be letting the exception loose thus the framework detects the issue and rollsback the session and yields the processor. Likely an easy thing to fix in the processor but please do share the full nifi-app.log entry for this. Thanks Joe On Wed, Dec 14, 2016 at 10:47 AM, James McMahon wrote: > Hello. Am wondering if anyone knows how to overcome a challenge with > unmappable file characters? I have used a GetFile processor to bring a large > number of zip files into a NiFi flow. All read in successfully. I try to > then use an UnpackContent processor to unzip the files to their individual > file members. Most work just fine. However there appears to be a file that > throws this error in UnpackContent: > > failed to process session due to java.nio.file.InvalidPathException: > Malformed input or input contains unmappable characters > > My processing stalls. Nothing else flows. What is the proper way to > configure the UnpackContent processor step so that it shuttle such files off > to the side when it encounters them, and permits the other files waiting in > queue to process? I do now have a "failure" path established for my > UnpackContent processor, but for some reason it does not send these problem > files down that path. I suspect it may be because the zip files does unpack > successfully but the underlying file(s) within the zip cause processing to > choke. > > How can I engineer a flow to overcome this challenge? Thanks in advance for > your help.
