[jira] [Commented] (NIFI-924) Add Camel support in NiFi
[ https://issues.apache.org/jira/browse/NIFI-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731792#comment-14731792 ] Joseph Witt commented on NIFI-924: -- Hello Jean-Baptiste, Thanks for contributing. Can you describe this idea a bit more? What is meant by a 'nifi route' ? I had a great conversation at a meetup recently with Hadrian. Would be cool to hear your view on how you think nifi and camel could compliment eachother. Thanks Joe > Add Camel support in NiFi > - > > Key: NIFI-924 > URL: https://issues.apache.org/jira/browse/NIFI-924 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Jean-Baptiste Onofré > > I'm working on a NiFi Route able to leverage a Camel route (runtime routing), > and another one being able to bootstrap a Camel route starting from Camel > DSLs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731563#comment-14731563 ] Joseph Percivall commented on NIFI-447: --- I created an initial patch. I wasn't sure of all the test cases I should go into since the default is mainly Regex edge cases and many of the new features don't deal with them. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > Attachments: NIFI-447.patch > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-447: -- Attachment: NIFI-447.patch Initial Patch > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > Attachments: NIFI-447.patch > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731397#comment-14731397 ] Ryan Blue commented on NIFI-919: I think it makes sense to support bare record output because other systems use it. We support Flume, for example, and a few Flume sinks use bare Avro record payloads with headers to set the schema fingerprint, url, or literal. I don't see a need to support the JSON encoding, at least as output. It is needlessly expensive and is mostly used so that you can read the encoded payload by viewing the file. Adequate tooling can avoid the need. > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-916) NiFi stop command occasionally results in Socket Timeout exception
[
https://issues.apache.org/jira/browse/NIFI-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731363#comment-14731363
]
Mark Payne commented on NIFI-916:
-
[~bende] - code for this patch looks good. Makes sense to me. Compiles & passes
contrib-check. I am going to rebuild and do some testing to make sure.
> NiFi stop command occasionally results in Socket Timeout exception
> --
>
> Key: NIFI-916
> URL: https://issues.apache.org/jira/browse/NIFI-916
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Bryan Bende
> Fix For: 0.3.0
>
> Attachments: NIFI-916.patch
>
>
> When I issue the bin/nifi.sh stop command, running in CentOS 7 in my VM, I
> occasionally get a timeout exception. If this happens, it will continue to
> happen until I manually kill the process.
> Below is the output that I receive, as well as the results of running netstat
> to verify that NiFi is in fact listening on the port reported.
> {noformat}
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ bin/nifi.sh stop
> Java home: /usr/java/latest
> NiFi home: /opt/nifi-0.3.0-SNAPSHOT
> Bootstrap Config File: /opt/nifi-0.3.0-SNAPSHOT/conf/bootstrap.conf
> 2015-08-31 16:05:38,454 ERROR [main] org.apache.nifi.bootstrap.Command Failed
> to send shutdown command to port 49560 due to
> java.net.SocketTimeoutException: Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_45]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:223)
> ~[na:1.8.0_45]
> at org.apache.nifi.bootstrap.RunNiFi.stop(RunNiFi.java:536)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:208)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ netstat -anp | grep 49560
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp0 0 127.0.0.1:49560 0.0.0.0:* LISTEN
> 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37907
> CLOSE_WAIT 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37905
> CLOSE_WAIT 6873/java
> tcp6 0 0 127.0.0.1:37907 127.0.0.1:49560 FIN_WAIT2
> -
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-927) NiFi Spark Receiver not serializing data appropriately
[ https://issues.apache.org/jira/browse/NIFI-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731342#comment-14731342 ] Bryan Bende commented on NIFI-927: -- +1 code looks good and passed contrib-check > NiFi Spark Receiver not serializing data appropriately > -- > > Key: NIFI-927 > URL: https://issues.apache.org/jira/browse/NIFI-927 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-916) NiFi stop command occasionally results in Socket Timeout exception
[
https://issues.apache.org/jira/browse/NIFI-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731336#comment-14731336
]
Mark Payne commented on NIFI-916:
-
[~bende] - yeah this is okay. It happens sometimes during shutdown. This is
caused because a session commit is updating the FlowFile Repository and the
repo has already been shutdown. This is OK because the session will just be
rolled back, essentially, on restart.
> NiFi stop command occasionally results in Socket Timeout exception
> --
>
> Key: NIFI-916
> URL: https://issues.apache.org/jira/browse/NIFI-916
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Bryan Bende
> Fix For: 0.3.0
>
> Attachments: NIFI-916.patch
>
>
> When I issue the bin/nifi.sh stop command, running in CentOS 7 in my VM, I
> occasionally get a timeout exception. If this happens, it will continue to
> happen until I manually kill the process.
> Below is the output that I receive, as well as the results of running netstat
> to verify that NiFi is in fact listening on the port reported.
> {noformat}
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ bin/nifi.sh stop
> Java home: /usr/java/latest
> NiFi home: /opt/nifi-0.3.0-SNAPSHOT
> Bootstrap Config File: /opt/nifi-0.3.0-SNAPSHOT/conf/bootstrap.conf
> 2015-08-31 16:05:38,454 ERROR [main] org.apache.nifi.bootstrap.Command Failed
> to send shutdown command to port 49560 due to
> java.net.SocketTimeoutException: Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_45]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:223)
> ~[na:1.8.0_45]
> at org.apache.nifi.bootstrap.RunNiFi.stop(RunNiFi.java:536)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:208)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ netstat -anp | grep 49560
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp0 0 127.0.0.1:49560 0.0.0.0:* LISTEN
> 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37907
> CLOSE_WAIT 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37905
> CLOSE_WAIT 6873/java
> tcp6 0 0 127.0.0.1:37907 127.0.0.1:49560 FIN_WAIT2
> -
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731330#comment-14731330 ] Mark Payne commented on NIFI-447: - Agreed. That's the route that I would take. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731310#comment-14731310 ] Joseph Percivall commented on NIFI-447: --- In the case where we are appending and a line or text ends with an endline character. For line by line I'm thinking we'll keep the endline character as the last character and "append" the replacement value before the endline character. It doesn't really make sense to put more information on a line after an endline character. For entire text though I'm a little torn. I think I'm leaning to ignoring what ever the text ends with and just appending it at the very end. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731304#comment-14731304 ] Mark Payne commented on NIFI-810: - [~aldrin]: thoughts on using an annotation like "@InputRequirement" that takes an enumeration instead of @IgnoresInput and @RequestInput? Then you could use one of: @InputRequirement(INPUT_REQUIRED) @InputRequirement(INPUT_OPTIONAL) @InputRequirement(INPUT_FORBIDDEN) If the annotation is not present, it would be the same as using INPUT_OPTIONAL, which is how this behaves now. Not sure which way is cleaner: to have explicit annotations for each or to have an annotation that accepts input? > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731287#comment-14731287 ] Bryan Bende edited comment on NIFI-919 at 9/4/15 7:26 PM: -- Thanks for reviewing Ryan. So moving from the splitting side of things then, to the output side... did we decide if it makes sense to support bare-record output here? and also, do we want to allow choosing of output encoding (binary vs json)? It would keep the first version of this fairly simple if we did binary datafile in, split by records, and binary datafiles out. However, want to make sure we are helping solve the most common real world scenarios that people have. was (Author: bende): Thanks for reviewing Ryan. So moving from the splitting side of things thenm to the output side... did we decide if it makes sense to support bare-record output here? and also, do we want to allow choosing of output encoding (binary vs json)? It would keep the first version of this fairly simple if we did binary datafile in, split by records, and binary datafiles out. However, want to make sure we are helping solve the most common real world scenarios that people have. > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731287#comment-14731287 ] Bryan Bende commented on NIFI-919: -- Thanks for reviewing Ryan. So moving from the splitting side of things thenm to the output side... did we decide if it makes sense to support bare-record output here? and also, do we want to allow choosing of output encoding (binary vs json)? It would keep the first version of this fairly simple if we did binary datafile in, split by records, and binary datafiles out. However, want to make sure we are helping solve the most common real world scenarios that people have. > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731248#comment-14731248 ] Joseph Percivall commented on NIFI-447: --- Yup, that's exactly what I was thinking > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731240#comment-14731240 ] Mark Payne commented on NIFI-447: - I can't argue that we shouldn't. Though if we are going to allow it for each option.. perhaps that should be separated into a new property? Similar to the way that it is currently configured... Line-by-Line or Entire Text, and then Append, Prepend, Regex, Literal Replacement > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731233#comment-14731233 ] Joseph Percivall commented on NIFI-447: --- [~markap14]: Why shouldn't we have linewise options for append and prepend? The functionality would append or prepend on each line. The same way ".*" will match the whole line and replace each line with the replacement value. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
[ https://issues.apache.org/jira/browse/NIFI-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-932: Fix Version/s: 0.4.0 > Allow ExecuteSQL to run periodically without any input FlowFile > --- > > Key: NIFI-932 > URL: https://issues.apache.org/jira/browse/NIFI-932 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > Fix For: 0.4.0 > > > Currently, ExecuteSQL will return immediately if run without a FlowFile. > Instead, it should be able to be configured to run (say once per hour) > without any input data. > The framework as-is does not provide this information, so the fact of whether > or not incoming connections exist would have to be provided by the > ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
[ https://issues.apache.org/jira/browse/NIFI-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731209#comment-14731209 ] Mark Payne commented on NIFI-932: - I agree. This should certainly be in the documentation. > Allow ExecuteSQL to run periodically without any input FlowFile > --- > > Key: NIFI-932 > URL: https://issues.apache.org/jira/browse/NIFI-932 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > Fix For: 0.4.0 > > > Currently, ExecuteSQL will return immediately if run without a FlowFile. > Instead, it should be able to be configured to run (say once per hour) > without any input data. > The framework as-is does not provide this information, so the fact of whether > or not incoming connections exist would have to be provided by the > ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
[ https://issues.apache.org/jira/browse/NIFI-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731205#comment-14731205 ] Aldrin Piri commented on NIFI-932: -- Our documentation is quite weak on ExecuteSQL in its current state. With this added functionality, let's be sure to provide good detail in the processor's usage/information about how these "modes" function. > Allow ExecuteSQL to run periodically without any input FlowFile > --- > > Key: NIFI-932 > URL: https://issues.apache.org/jira/browse/NIFI-932 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > > Currently, ExecuteSQL will return immediately if run without a FlowFile. > Instead, it should be able to be configured to run (say once per hour) > without any input data. > The framework as-is does not provide this information, so the fact of whether > or not incoming connections exist would have to be provided by the > ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
Mark Payne created NIFI-932: --- Summary: Allow ExecuteSQL to run periodically without any input FlowFile Key: NIFI-932 URL: https://issues.apache.org/jira/browse/NIFI-932 Project: Apache NiFi Issue Type: Improvement Reporter: Mark Payne Currently, ExecuteSQL will return immediately if run without a FlowFile. Instead, it should be able to be configured to run (say once per hour) without any input data. The framework as-is does not provide this information, so the fact of whether or not incoming connections exist would have to be provided by the ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731191#comment-14731191 ] Ryan Blue commented on NIFI-919: I think we should go with record strategy to start off with. The block strategy doesn't sound very useful to me since users don't typically know anything about Avro blocks. They are almost always an internal thing that users don't see. Users might want to break the file into chunks based on a target file size, which I think could be a good use case to support and makes AVRO-1726 a bit easier to implement as well. > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-931) Mock Framework for Controller Services doesn't call customValidate() before @OnEnabled
Brandon DeVries created NIFI-931: Summary: Mock Framework for Controller Services doesn't call customValidate() before @OnEnabled Key: NIFI-931 URL: https://issues.apache.org/jira/browse/NIFI-931 Project: Apache NiFi Issue Type: Bug Reporter: Brandon DeVries Priority: Minor Under the current mock model, a controller service's customValidate() method isn't called until after the methods annotated with \@OnEnabled. This can cause some unexpected behavior, as \@OnEnabled shouldn't run until the properties have been validated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-773) Mock Framework needs to provide better mechanisms for testing Controller Services
[ https://issues.apache.org/jira/browse/NIFI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731187#comment-14731187 ] Brandon DeVries commented on NIFI-773: -- sure, created NIFI-931. > Mock Framework needs to provide better mechanisms for testing Controller > Services > - > > Key: NIFI-773 > URL: https://issues.apache.org/jira/browse/NIFI-773 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Mark Payne > > Currently, in order to test a Controller Service with the TestRunner, you > have to create the TestRunner by creating a mocked out Processor. > We need to provide the ability to test Controller Services without doing this > type of thing. > Additionally, we then have to set properties by calling > TestRunner.setProperty(ControllerService, PropertyDescriptor, String). This > is fine if we are using a Controller Service to test a processor, but if the > Controller Service itself is what is being tested, there should be no need to > pass the controller service to the test runner each time. > Recommend we create a ControllerService-specific Test Runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-930) Need more details on running NiFi as non-root
Ali Bajwa created NIFI-930: -- Summary: Need more details on running NiFi as non-root Key: NIFI-930 URL: https://issues.apache.org/jira/browse/NIFI-930 Project: Apache NiFi Issue Type: Improvement Components: Documentation & Website Affects Versions: 0.3.0 Reporter: Ali Bajwa Priority: Minor When setting up to run NiFi as non-root user, Nifi kept going down and I kept getting errors about user not being in list of sudoers.After adding nifi user to /etc/sudoers it started working. Would be great to have better documentation on this e.g. point this out as a required manual step before installation. Also would be a good place to talk about run.as option and how it would be used in such scenario -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-773) Mock Framework needs to provide better mechanisms for testing Controller Services
[ https://issues.apache.org/jira/browse/NIFI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731165#comment-14731165 ] Mark Payne commented on NIFI-773: - Ohh... yeah, I bet that does cause some unexpected behavior. Do you mind creating a new ticket for that? > Mock Framework needs to provide better mechanisms for testing Controller > Services > - > > Key: NIFI-773 > URL: https://issues.apache.org/jira/browse/NIFI-773 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Mark Payne > > Currently, in order to test a Controller Service with the TestRunner, you > have to create the TestRunner by creating a mocked out Processor. > We need to provide the ability to test Controller Services without doing this > type of thing. > Additionally, we then have to set properties by calling > TestRunner.setProperty(ControllerService, PropertyDescriptor, String). This > is fine if we are using a Controller Service to test a processor, but if the > Controller Service itself is what is being tested, there should be no need to > pass the controller service to the test runner each time. > Recommend we create a ControllerService-specific Test Runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-928) Add ability to configure the pid directory
[ https://issues.apache.org/jira/browse/NIFI-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Bajwa updated NIFI-928: --- Issue Type: Improvement (was: New Feature) > Add ability to configure the pid directory > -- > > Key: NIFI-928 > URL: https://issues.apache.org/jira/browse/NIFI-928 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Affects Versions: 0.3.0 >Reporter: Ali Bajwa >Priority: Minor > > Per Mark P, there is currently no way to configure the pid dir e.g. via > nifi.properties. This would be useful for Ambari/Hadoop integration as for > all the other components, the pids are stored under /var/run/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-929) Ability to generate a 'true' pid file
Ali Bajwa created NIFI-929: -- Summary: Ability to generate a 'true' pid file Key: NIFI-929 URL: https://issues.apache.org/jira/browse/NIFI-929 Project: Apache NiFi Issue Type: Improvement Components: Configuration Affects Versions: 0.3.0 Reporter: Ali Bajwa Priority: Minor The nifi pid file does not seem to be a true pid file (it has other info as well). For integration with monitoring tools like Ambari it would be nice to have a true pid file (containing only the pid) created as well (in line with other Hadoop components). This is coming from the Ambari service I put together: I had to parse out the pid from Nifi pidfile and maintain another one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-928) Add ability to configure the pid directory
Ali Bajwa created NIFI-928: -- Summary: Add ability to configure the pid directory Key: NIFI-928 URL: https://issues.apache.org/jira/browse/NIFI-928 Project: Apache NiFi Issue Type: New Feature Components: Configuration Affects Versions: 0.3.0 Reporter: Ali Bajwa Priority: Minor Per Mark P, there is currently no way to configure the pid dir e.g. via nifi.properties. This would be useful for Ambari/Hadoop integration as for all the other components, the pids are stored under /var/run/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-773) Mock Framework needs to provide better mechanisms for testing Controller Services
[ https://issues.apache.org/jira/browse/NIFI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731082#comment-14731082 ] Brandon DeVries commented on NIFI-773: -- this comment will probably be OBE after the creation of a ControllerService-specific Test Runner, but under the current model a controller service's customValidate() method isn't called until after the methods annotated with \@onEnabled, which can cause some unexpected behavior. > Mock Framework needs to provide better mechanisms for testing Controller > Services > - > > Key: NIFI-773 > URL: https://issues.apache.org/jira/browse/NIFI-773 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Mark Payne > > Currently, in order to test a Controller Service with the TestRunner, you > have to create the TestRunner by creating a mocked out Processor. > We need to provide the ability to test Controller Services without doing this > type of thing. > Additionally, we then have to set properties by calling > TestRunner.setProperty(ControllerService, PropertyDescriptor, String). This > is fine if we are using a Controller Service to test a processor, but if the > Controller Service itself is what is being tested, there should be no need to > pass the controller service to the test runner each time. > Recommend we create a ControllerService-specific Test Runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-927) NiFi Spark Receiver not serializing data appropriately
[ https://issues.apache.org/jira/browse/NIFI-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731072#comment-14731072 ] Michael Griffiths commented on NIFI-927: I can confirm that this has solved the "com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException" being thrown in my Spark job. Thanks Mark! > NiFi Spark Receiver not serializing data appropriately > -- > > Key: NIFI-927 > URL: https://issues.apache.org/jira/browse/NIFI-927 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-927) NiFi Spark Receiver not serializing data appropriately
[ https://issues.apache.org/jira/browse/NIFI-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-927: Attachment: 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > NiFi Spark Receiver not serializing data appropriately > -- > > Key: NIFI-927 > URL: https://issues.apache.org/jira/browse/NIFI-927 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-927) NiFi Spark Receiver not serializing data appropriately
Mark Payne created NIFI-927: --- Summary: NiFi Spark Receiver not serializing data appropriately Key: NIFI-927 URL: https://issues.apache.org/jira/browse/NIFI-927 Project: Apache NiFi Issue Type: Bug Components: Tools and Build Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730985#comment-14730985 ] Bryan Bende commented on NIFI-919: -- I've been playing around with adding a new method to DataFileWriter which essentially works exactly the same as appendAllFrom(...), but it lets you specify the number of blocks: https://issues.apache.org/jira/browse/AVRO-1726 I think if we got that in there it would work nicely for splitting up the files based on number of blocks. Instead of making the "Split Size" property be the number of records (or approx. number), we would likely make it the number of blocks when Split Strategy = Block, and the number of records when Split Strategy = Record. I'm thinking maybe first pass of this processor only supports the record strategy, unless we figure out something that doesn't require adding functionality to Avro. > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730952#comment-14730952 ] Mark Payne commented on NIFI-447: - [~JPercivall]: I would say yes, it makes sense. Good catch. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730944#comment-14730944 ] Joseph Percivall commented on NIFI-447: --- I'm working on this now. Should the literal replace have a linewise option as well? > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
