[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576360#comment-14576360 ] Ricky Saltzer commented on NIFI-583: I'm going to make a test to try and catch/fix this as soon as I can Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576355#comment-14576355 ] Ricky Saltzer commented on NIFI-583: [~joewitt] I'm glad you find this useful! Your use case of lock/catch and signaling processors to run is the exact reason for the proposed change :) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576387#comment-14576387 ] Aldrin Piri commented on NIFI-583: -- Patch submitted. I feel like there's some slight difference in functionality between the test runner and the actual framework based on the error reported through how the file was handled. The issue seemed to be based around using the new flow file as the source input (and zero bytes). This was not an error in the testing framework, but feels like it perhaps should be. I think the error is complaining that there is duplicative use of that particular flow file in different callbacks (or in this case, the same call back but in read and write capacities). Anyone have some more pointed thoughts? Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Aldrin Piri Fix For: 0.2.0 Attachments: 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth.patch, NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576404#comment-14576404 ] Joseph Witt commented on NIFI-583: -- i spoke slightly too soon [WARNING] src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java[343] (sizes) LineLength: Line is longer than 200 characters (found 214). The contrib-check was running when I hit enter above. Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Aldrin Piri Fix For: 0.2.0 Attachments: 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth.patch, NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576402#comment-14576402 ] Joseph Witt commented on NIFI-583: -- +1. Code looks good. Build works including -Pcontrib-check Tested on a raspberry pi running iostat. Had 4 different tests of various combinations. All worked as expected now. Thanks Joe Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Aldrin Piri Fix For: 0.2.0 Attachments: 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth.patch, NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576351#comment-14576351 ] Ricky Saltzer commented on NIFI-583: Is the behavior the same when using non-0 B flow files? such as, 1 B? Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576352#comment-14576352 ] Aldrin Piri commented on NIFI-583: -- Yep, I am getting it for 1B files as well. Must have had a gap in my configurations when testing. Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576356#comment-14576356 ] Joseph Witt commented on NIFI-583: -- This is definitely the fun part of open source. Someone builds something they think will be useful. It turns out to be useful for my own purposes only days later. Last night i played 'review all licensing/notice' in light of all the changes: That is NOT! the fun part. Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576354#comment-14576354 ] Ricky Saltzer commented on NIFI-583: good catch, I wonder what the threshold is for file size to where it starts working? Perhaps there is some sort of flush that needs to be called? Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576353#comment-14576353 ] Joseph Witt commented on NIFI-583: -- yeah seems to be a problem with content supplied as well. I'm playing around on a raspberry pi just because it is so stinkin' cool and it turns out this processor change is quite useful. It allows this processor to act as a sort of lock/latch where the presence of a flowfile signals it is ok to call it. It is like passing the token for who gets to run. So I can bounce between taking pictures and taking videos and never worry about overlap. Cool idea Ricky! Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576416#comment-14576416 ] Aldrin Piri commented on NIFI-583: -- Fixed that check style issue and ran contrib check. Looks good. That patch is included. Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Aldrin Piri Fix For: 0.2.0 Attachments: 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth-Checkstyle-Fix.patch, 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth.patch, NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576414#comment-14576414 ] Joseph Witt commented on NIFI-583: -- +1 Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Aldrin Piri Fix For: 0.2.0 Attachments: 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth-Checkstyle-Fix.patch, 0001-NIFI-583-Adjusting-the-callback-to-be-aware-of-wheth.patch, NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574972#comment-14574972 ] Aldrin Piri commented on NIFI-583: -- [~rickysaltzer] Thanks! Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-583) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
[ https://issues.apache.org/jira/browse/NIFI-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574971#comment-14574971 ] ASF subversion and git services commented on NIFI-583: -- Commit 6d1128497bcfcfd105fd3d18305051615b254576 in incubator-nifi's branch refs/heads/develop from [~rickysaltzer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=6d11284 ] NIFI-583: Ignore STDIN for ExecuteStreamCommand - Added the ability (default: false) to ignore STDIN when passing a flowfile to the ExecuteStreamCommand processor. This is useful if the command you are executing cannot take STDIN, or passing STDIN is unnecessary Signed-off-by: Aldrin Piri ald...@apache.org Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile Key: NIFI-583 URL: https://issues.apache.org/jira/browse/NIFI-583 Project: Apache NiFi Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ricky Saltzer Assignee: Ricky Saltzer Fix For: 0.2.0 Attachments: NIFI-583.1.patch, NIFI-583.2.patch In some cases it would be really nice to allow a FlowFile to trigger an OS action. For instance, after a daily dump of data is written to an Impala table in HDFS, I would like to execute a refresh on the table via the shell. As it stands, the ExecuteProcess processor will allow a FlowFile in a connection to trigger execution, but unless your connection has an expiration set, the FlowFile will stay there indefinitely. The main issue here is that it will continue to re-execute your ExecuteProcess processor over and over. As far as I know, there's only two clear ways around this. (1) - you can use the ExecuteStreamCommand, instead, but *only* if that command can properly handle STDIN. (2) - you can set your ExecuteProcess processor to execute on a schedule (e.g. 1 per minute) and expire the FlowFile before it can re-execute (e.g. 10 seconds). It would be useful if the ExecuteProcess processor consumed the FlowFile, and passed it through a passthrough relationship of some kind. A second option would be to make it configurable (false by default) to drop the FlowFile, or to pass it through a second relationship, that way it doesn't break anyone's current pipelines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
