[jira] [Created] (NIFI-2617) If no ZooKeeper quorum, NiFi does not respond to some web requests
Mark Payne created NIFI-2617: Summary: If no ZooKeeper quorum, NiFi does not respond to some web requests Key: NIFI-2617 URL: https://issues.apache.org/jira/browse/NIFI-2617 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.0.0 If I lose a ZooKeeper quorum, some web requests never return from NiFi, regardless of how long I wait. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2411) ModifyBytes should use long instead of int for offsets.
[
https://issues.apache.org/jira/browse/NIFI-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-2411:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ModifyBytes should use long instead of int for offsets.
> ---
>
> Key: NIFI-2411
> URL: https://issues.apache.org/jira/browse/NIFI-2411
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Joe Skora
>Assignee: Joe Skora
> Labels: easyfix
> Fix For: 1.0.0, 0.8.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> ModifyBytes.onTrigger() uses Java 32 bit {{int}} value for byte offsets
> limiting it to 2 Gigabytes, switching to {{long}} values will allow it to
> handle up to 15 Exabytes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2411) ModifyBytes should use long instead of int for offsets.
[
https://issues.apache.org/jira/browse/NIFI-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430953#comment-15430953
]
Mark Payne commented on NIFI-2411:
--
[~jskora] - looks good. +1, Merged to 0.x and master. Thanks!
> ModifyBytes should use long instead of int for offsets.
> ---
>
> Key: NIFI-2411
> URL: https://issues.apache.org/jira/browse/NIFI-2411
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Joe Skora
>Assignee: Joe Skora
> Labels: easyfix
> Fix For: 1.0.0, 0.8.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> ModifyBytes.onTrigger() uses Java 32 bit {{int}} value for byte offsets
> limiting it to 2 Gigabytes, switching to {{long}} values will allow it to
> handle up to 15 Exabytes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2035) Connection creation needs to verify that source and destination both exist during 1st phase of 2 phase commit
[ https://issues.apache.org/jira/browse/NIFI-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2035: - Resolution: Fixed Status: Resolved (was: Patch Available) > Connection creation needs to verify that source and destination both exist > during 1st phase of 2 phase commit > - > > Key: NIFI-2035 > URL: https://issues.apache.org/jira/browse/NIFI-2035 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > > When a connection is created, we don't verify in the first step that both the > source and destination exist. In the case of a Remote Process Group, where > the ports change in the background, this can become problematic, we one node > in the cluster may not have the same ports as another and the connection > creation could end up failing on that node, resulting in the node getting > kicked out of the cluster. We should also verify that the new destination > exists when updating a connection if its destination changes, as this could > happen there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2651) Remote Process Group does not stop immediately when using HTTP-based Site-to-Site
[ https://issues.apache.org/jira/browse/NIFI-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436066#comment-15436066 ] Mark Payne commented on NIFI-2651: -- The attached template should help to expose/replicate the issue. It generates a lot of large FlowFiles and sends them back to the same cluster. If we right click on the Remote Process Group and then click Disable Transmission, it should stop immediately but does not. > Remote Process Group does not stop immediately when using HTTP-based > Site-to-Site > - > > Key: NIFI-2651 > URL: https://issues.apache.org/jira/browse/NIFI-2651 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 1.0.0 > > Attachments: HTTP_Site-to-Site_with_Large_Data_Chunks.xml > > > If we stop a Remote Process Group that is using the RAW Site-to-Site > protocol, it stops immediately, interrupting any transaction that is in > progress. If we stop a Remote Process Group that is using HTTP Site-to-Site > Protocol, though, the RPG waits until the current transactions have finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2651) Remote Process Group does not stop immediately when using HTTP-based Site-to-Site
[ https://issues.apache.org/jira/browse/NIFI-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2651: - Status: Patch Available (was: Open) > Remote Process Group does not stop immediately when using HTTP-based > Site-to-Site > - > > Key: NIFI-2651 > URL: https://issues.apache.org/jira/browse/NIFI-2651 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 1.0.0 > > Attachments: HTTP_Site-to-Site_with_Large_Data_Chunks.xml > > > If we stop a Remote Process Group that is using the RAW Site-to-Site > protocol, it stops immediately, interrupting any transaction that is in > progress. If we stop a Remote Process Group that is using HTTP Site-to-Site > Protocol, though, the RPG waits until the current transactions have finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2651) Remote Process Group does not stop immediately when using HTTP-based Site-to-Site
Mark Payne created NIFI-2651: Summary: Remote Process Group does not stop immediately when using HTTP-based Site-to-Site Key: NIFI-2651 URL: https://issues.apache.org/jira/browse/NIFI-2651 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Priority: Critical Fix For: 1.0.0 If we stop a Remote Process Group that is using the RAW Site-to-Site protocol, it stops immediately, interrupting any transaction that is in progress. If we stop a Remote Process Group that is using HTTP Site-to-Site Protocol, though, the RPG waits until the current transactions have finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2651) Remote Process Group does not stop immediately when using HTTP-based Site-to-Site
[ https://issues.apache.org/jira/browse/NIFI-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2651: - Attachment: HTTP_Site-to-Site_with_Large_Data_Chunks.xml > Remote Process Group does not stop immediately when using HTTP-based > Site-to-Site > - > > Key: NIFI-2651 > URL: https://issues.apache.org/jira/browse/NIFI-2651 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 1.0.0 > > Attachments: HTTP_Site-to-Site_with_Large_Data_Chunks.xml > > > If we stop a Remote Process Group that is using the RAW Site-to-Site > protocol, it stops immediately, interrupting any transaction that is in > progress. If we stop a Remote Process Group that is using HTTP Site-to-Site > Protocol, though, the RPG waits until the current transactions have finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2459) HTTP Site-to-Site can't update remote peer topology if the "bootstrap url" node is down
[ https://issues.apache.org/jira/browse/NIFI-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2459: - Assignee: Koji Kawamura > HTTP Site-to-Site can't update remote peer topology if the "bootstrap url" > node is down > --- > > Key: NIFI-2459 > URL: https://issues.apache.org/jira/browse/NIFI-2459 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Assignee: Koji Kawamura > Fix For: 1.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > To establish a Site-to-Site protocol, a client NiFi uses the remote NiFi API > url that is specified as 'URL' at the creation of a Remote Process Group > (let's call it as 'bootstrap url'). > Currently, HTTP Site-to-Site keeps using the same 'bootstrap url', even if > the remote node goes down. If the bootstrap node went down, Site-to-Site data > transfer will keep working, but it can't update remote peer topology. > RAW socket version cares about this scenario by asking updated peer > information with every nodes that it has connected before, until it gets a > successful response. > HTTP version should follow how RAW version handles this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2659) Remote Process Group UI component shows Sent stats as Received stats
[
https://issues.apache.org/jira/browse/NIFI-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-2659:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Remote Process Group UI component shows Sent stats as Received stats
>
>
> Key: NIFI-2659
> URL: https://issues.apache.org/jira/browse/NIFI-2659
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
> Attachments: screenshot-1.png, screenshot-2-fixed.png
>
>
> RPG component shows Sent and Received statistics. Sent stats is shown
> correctly, but Received stats also uses the Sent stats. Because of that, the
> correct Received stats is not shown.
> Corresponding JS code uses 'sent' stats for 'received' wrongly:
> {code:title=nf-remote-process-group.js}
> // received count value
> updated.select('text.remote-process-group-received tspan.count')
> .text(function (d) {
> return
> nf.Common.substringBeforeFirst(d.status.aggregateSnapshot.sent, ' ');
> });
> // received size value
> updated.select('text.remote-process-group-received tspan.size')
> .text(function (d) {
> return ' ' +
> nf.Common.substringAfterFirst(d.status.aggregateSnapshot.sent, ' ');
> });
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-2739) ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to communicate with Kafka broker
Mark Payne created NIFI-2739: Summary: ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to communicate with Kafka broker Key: NIFI-2739 URL: https://issues.apache.org/jira/browse/NIFI-2739 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 If I use ConsumeKafka and point to a broker that is in a bad state, I see ConsumeKafka block indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-2739) ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to communicate with Kafka broker
[ https://issues.apache.org/jira/browse/NIFI-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-2739. -- Resolution: Fixed > ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to > communicate with Kafka broker > > > Key: NIFI-2739 > URL: https://issues.apache.org/jira/browse/NIFI-2739 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > If I use ConsumeKafka and point to a broker that is in a bad state, I see > ConsumeKafka block indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2628) NiFi node does not save flow returned by cluster coodinator to disk immediately
[ https://issues.apache.org/jira/browse/NIFI-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-2628: Assignee: Mark Payne > NiFi node does not save flow returned by cluster coodinator to disk > immediately > --- > > Key: NIFI-2628 > URL: https://issues.apache.org/jira/browse/NIFI-2628 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Arpit Gupta >Assignee: Mark Payne > Fix For: 1.1.0 > > > While investigating NIFI-2605 it was determined that NiFi node does not save > the flow returned by cluster coordinator to disk immediately. This in some > rare cases can lead to issues. > See for more detail > https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2628) NiFi node does not save flow returned by cluster coodinator to disk immediately
[ https://issues.apache.org/jira/browse/NIFI-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2628: - Status: Patch Available (was: Open) > NiFi node does not save flow returned by cluster coodinator to disk > immediately > --- > > Key: NIFI-2628 > URL: https://issues.apache.org/jira/browse/NIFI-2628 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Arpit Gupta >Assignee: Mark Payne > Fix For: 1.1.0 > > > While investigating NIFI-2605 it was determined that NiFi node does not save > the flow returned by cluster coordinator to disk immediately. This in some > rare cases can lead to issues. > See for more detail > https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2707) "Bring to Front" function in UI does not to work after exiting group
Mark Payne created NIFI-2707: Summary: "Bring to Front" function in UI does not to work after exiting group Key: NIFI-2707 URL: https://issues.apache.org/jira/browse/NIFI-2707 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.0.0 Reporter: Mark Payne Fix For: 1.1.0 I have some connections that overlap one another. I right-clicked on one and clicked Bring to Front. This worked as expected. However, after I left the Process Group and came back, the connection that was brought to the front is no longer in front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2746) Error when accessing User page
[ https://issues.apache.org/jira/browse/NIFI-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2746: - Assignee: Mark Payne Status: Patch Available (was: Open) > Error when accessing User page > -- > > Key: NIFI-2746 > URL: https://issues.apache.org/jira/browse/NIFI-2746 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.1.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.1.0 > > > When I try to navigate to the Users page in 1.1.0-SNAPSHOT, I am seeing an > error, and the logs show a Null Pointer: > 2016-09-07 16:42:25,129 ERROR [NiFi Web Server-33364] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException. Returning Internal Server Error response. > java.lang.NullPointerException: null > at > org.apache.nifi.web.api.ApplicationResource.ensureFlowInitialized(ApplicationResource.java:764) > ~[classes/:na] > at > org.apache.nifi.web.api.ApplicationResource.isReplicateRequest(ApplicationResource.java:402) > ~[classes/:na] > at > org.apache.nifi.web.api.TenantsResource.getUsers(TenantsResource.java:285) > ~[classes/:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_91] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_91] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_91] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] > at > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > ~[jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) > [jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) > [jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) > [jersey-server-1.19.jar:1.19] > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) > [jersey-server-1.19.jar:1.19] > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) > [jersey-servlet-1.19.jar:1.19] > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) > [jersey-servlet-1.19.jar:1.19] > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) > [jersey-servlet-1.19.jar:1.19] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > [javax.servlet-api-3.1.0.jar:3.1.0] > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) > [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) > [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) > [jetty-servlets-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) > [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) > [classes/:na] > at >
[jira] [Created] (NIFI-2746) Error when accessing User page
Mark Payne created NIFI-2746: Summary: Error when accessing User page Key: NIFI-2746 URL: https://issues.apache.org/jira/browse/NIFI-2746 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.1.0 Reporter: Mark Payne Priority: Blocker Fix For: 1.1.0 When I try to navigate to the Users page in 1.1.0-SNAPSHOT, I am seeing an error, and the logs show a Null Pointer: 2016-09-07 16:42:25,129 ERROR [NiFi Web Server-33364] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response. java.lang.NullPointerException: null at org.apache.nifi.web.api.ApplicationResource.ensureFlowInitialized(ApplicationResource.java:764) ~[classes/:na] at org.apache.nifi.web.api.ApplicationResource.isReplicateRequest(ApplicationResource.java:402) ~[classes/:na] at org.apache.nifi.web.api.TenantsResource.getUsers(TenantsResource.java:285) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) [jetty-servlets-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at ... This appears to affect only the master branch and no previously released artifacts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (NIFI-1966) Address Issue of Nodes Disagreeing About Flow Contents on Startup
[ https://issues.apache.org/jira/browse/NIFI-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reopened NIFI-1966: -- > Address Issue of Nodes Disagreeing About Flow Contents on Startup > - > > Key: NIFI-1966 > URL: https://issues.apache.org/jira/browse/NIFI-1966 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Mark Payne >Priority: Blocker > Labels: migration > Fix For: 1.1.0 > > > During startup a coordinator is selected at random to act as the manager of > data flow and distributing it to other nodes. This works fine unless that > selected coordinator happens to have a uninheritable data flow. When this > occurs, the other nodes either are > - Able to connect to Coordinator and then unable to join due to an > uninheritable flow exception due to being presented with a bad data flow. In > this case, the node shut down. > - Unable to connect to the Coordinator due to lack of quorem and never > present with the bad data flow. In this case, the node ran its local flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1966) Address Issue of Nodes Disagreeing About Flow Contents on Startup
[ https://issues.apache.org/jira/browse/NIFI-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472335#comment-15472335 ] Mark Payne commented on NIFI-1966: -- Re-opening the ticket, as I have come across an issue. If all nodes in the cluster have an empty flow, but the flow is different between two of the nodes, we end up with the following warning in the logs: 2016-09-07 21:23:47,649 WARN [Process Cluster Protocol Request-9] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.util.NoSuchElementException: No value present java.util.NoSuchElementException: No value present at java.util.Optional.get(Optional.java:135) ~[na:1.8.0_101] at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElection.performElection(PopularVoteFlowElection.java:166) ~[nifi-framework-cluster-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElection.isElectionComplete(PopularVoteFlowElection.java:87) ~[nifi-framework-cluster-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElection.castVote(PopularVoteFlowElection.java:118) ~[nifi-framework-cluster-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.handleConnectionRequest(NodeClusterCoordinator.java:860) ~[nifi-framework-cluster-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.handle(NodeClusterCoordinator.java:740) ~[nifi-framework-cluster-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:167) ~[nifi-framework-cluster-protocol-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) ~[nifi-socket-utils-1.0.0.2.0.0.0-556.jar:1.0.0.2.0.0.0-556] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Address Issue of Nodes Disagreeing About Flow Contents on Startup > - > > Key: NIFI-1966 > URL: https://issues.apache.org/jira/browse/NIFI-1966 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Mark Payne >Priority: Blocker > Labels: migration > Fix For: 1.1.0 > > > During startup a coordinator is selected at random to act as the manager of > data flow and distributing it to other nodes. This works fine unless that > selected coordinator happens to have a uninheritable data flow. When this > occurs, the other nodes either are > - Able to connect to Coordinator and then unable to join due to an > uninheritable flow exception due to being presented with a bad data flow. In > this case, the node shut down. > - Unable to connect to the Coordinator due to lack of quorem and never > present with the bad data flow. In this case, the node ran its local flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2717) Site-to-Site needs to ensure that the remote instance commits the transaction before the local instance
[ https://issues.apache.org/jira/browse/NIFI-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2717: - Assignee: Mark Payne Status: Patch Available (was: Open) > Site-to-Site needs to ensure that the remote instance commits the transaction > before the local instance > --- > > Key: NIFI-2717 > URL: https://issues.apache.org/jira/browse/NIFI-2717 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.1.0 > > > When using site-to-site, if NiFi P sends to NiFi Q, right now NiFi P commits > the transaction before Q. As a result, if Q fails to commit the transaction, > there can be loss of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2715) When performing a Queue Listing, if FlowFile has no content, UI does not indicate this fact
Mark Payne created NIFI-2715: Summary: When performing a Queue Listing, if FlowFile has no content, UI does not indicate this fact Key: NIFI-2715 URL: https://issues.apache.org/jira/browse/NIFI-2715 Project: Apache NiFi Issue Type: Bug Components: Core UI Reporter: Mark Payne Priority: Minor Fix For: 1.1.0 I have a queue that has FlowFiles with no content (this is the result of running GetHTMLElement with the destination set to attribute and sending the matching FlowFiles to this queue). When I view the queue listing, the FlowFiles have no Content Claim, but there is no indicator that there is no Content Claim. The UI should specifically indicate this, as it leaves the user confused as to why they are unable to view the content. Also, while looking at this, when I view the details of a FlowFile, I found that the File Size was listed as 'no previous value set' because the FlowFile had no content claim. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2717) Site-to-Site needs to ensure that the remote instance commits the transaction before the local instance
Mark Payne created NIFI-2717: Summary: Site-to-Site needs to ensure that the remote instance commits the transaction before the local instance Key: NIFI-2717 URL: https://issues.apache.org/jira/browse/NIFI-2717 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Priority: Blocker Fix For: 1.1.0 When using site-to-site, if NiFi P sends to NiFi Q, right now NiFi P commits the transaction before Q. As a result, if Q fails to commit the transaction, there can be loss of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2773) Allow search results to be kept open
Mark Payne created NIFI-2773: Summary: Allow search results to be kept open Key: NIFI-2773 URL: https://issues.apache.org/jira/browse/NIFI-2773 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Mark Payne Fix For: 1.1.0 I wanted to make a change to each instance of the PublishKafka processors on my canvas. I have 5 instances. To do this, I had to search for PublishKafka, select the first result, change it, search for PublishKafka, select the second result, change it, and so on. This is time consuming and gets much more difficult if there are more search results. We should allow the user to 'pin' the search results or something of that nature so that the results do not go away when one is selected. Instead, they should go away only after I choose to close them. This way, I could search for PublishKafka, update the first one, then just click the next result and update it, click the next result, and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2771) REST API does not compress responses
Mark Payne created NIFI-2771: Summary: REST API does not compress responses Key: NIFI-2771 URL: https://issues.apache.org/jira/browse/NIFI-2771 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.0.0 Reporter: Mark Payne Priority: Critical Fix For: 1.1.0 Responses from the REST API do not appear to be compressed. In the logs, we see warnings: 2016-09-13 15:22:23,124 WARN [main] o.eclipse.jetty.util.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlets.GzipFilter 2016-09-13 15:22:23,124 WARN [main] org.eclipse.jetty.servlets.GzipFilter GzipFilter is deprecated. Use GzipHandler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2777) Provenance Events' Node Identifier not set when querying only 1 node in cluster
Mark Payne created NIFI-2777: Summary: Provenance Events' Node Identifier not set when querying only 1 node in cluster Key: NIFI-2777 URL: https://issues.apache.org/jira/browse/NIFI-2777 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 If I open the Provenance page and search for a FlowFile UUID and restrict the search to a specific node, the Node Identifier is not populated in the events that are returned. As a result, I cannot view the lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2778) If a Provenance Query is canceled, the repository doesn't stop immediately
Mark Payne created NIFI-2778: Summary: If a Provenance Query is canceled, the repository doesn't stop immediately Key: NIFI-2778 URL: https://issues.apache.org/jira/browse/NIFI-2778 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Fix For: 1.1.0 When a Provenance Query is issued and then canceled, the result object is marked as canceled, but repository continues to search. It should instead stop querying lucene and stop reading events from the provenance log files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2751) When a processor pulls a batch of FlowFiles, it keeps pulling from the same inbound connection instead of round-robin'ing
Mark Payne created NIFI-2751: Summary: When a processor pulls a batch of FlowFiles, it keeps pulling from the same inbound connection instead of round-robin'ing Key: NIFI-2751 URL: https://issues.apache.org/jira/browse/NIFI-2751 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Fix For: 1.1.0 When a Processor calls ProcessSession.get(int) or ProcessSession.get(FlowFileFilter), the FlowFiles only come from the first inbound connection, unless that connection is empty or doesn't have enough FlowFiles to complete the get() call. It should instead round-robin between the incoming connections, just as ProcessSession.get() does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2865) Address issues of PublishKafka blocking when having trouble communicating with Kafka broker and improve performance
Mark Payne created NIFI-2865: Summary: Address issues of PublishKafka blocking when having trouble communicating with Kafka broker and improve performance Key: NIFI-2865 URL: https://issues.apache.org/jira/browse/NIFI-2865 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 When NiFi is unable to communicate properly with the Kafka broker, we see the NiFi threads occasionally block. This should be resolvable by calling the wakeup() method of the client. Additionally, if Kafka takes too long to respond, we should be able to route the FlowFile to failure and move on. PublishKafka has a nice feature that allows a demarcated stream to be sent as separate messages, so that a large number of messages can be sent as a single FlowFile. However, in the case of individual messages per FlowFile, the performance could be improved by batching together multiple FlowFiles per session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2874) StreamDemarcator can return wrong data for token
Mark Payne created NIFI-2874:
Summary: StreamDemarcator can return wrong data for token
Key: NIFI-2874
URL: https://issues.apache.org/jira/browse/NIFI-2874
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 1.1.0, 0.7.1
There is a case where StreamDemarcator can return the wrong data for a token.
If a token ends at the end of the buffer, and the next token is smaller than
the previous, it can result in the next token keeping part of the buffer's
content. The code below is a unit test that exposes this:
{code}
@Test
public void testOnBufferSplitNoTrailingDelimiter() throws IOException {
final byte[] inputData = "Yes\nNo".getBytes(StandardCharsets.UTF_8);
ByteArrayInputStream is = new ByteArrayInputStream(inputData);
StreamDemarcator scanner = new StreamDemarcator(is, "\n".getBytes(),
1000, 3);
final byte[] first = scanner.nextToken();
final byte[] second = scanner.nextToken();
assertNotNull(first);
assertNotNull(second);
assertArrayEquals(first, new byte[] {'Y', 'e', 's'});
assertArrayEquals(second, new byte[] {'N', 'o'});
}
{code}
In this case, the second token, which should be 'No' comes back as 'Nos'
because it contains the 's' from the previous token.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2874) StreamDemarcator can return wrong data for token
[
https://issues.apache.org/jira/browse/NIFI-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1085#comment-1085
]
Mark Payne commented on NIFI-2874:
--
This was addressed in the 1.1.0 baseline already via NIFI-2865, as it was found
when testing the implementation of that ticket. So no PR is needed for the 1.x
baseline. Will create a PR to address in the 0.x baseline.
> StreamDemarcator can return wrong data for token
>
>
> Key: NIFI-2874
> URL: https://issues.apache.org/jira/browse/NIFI-2874
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.1.0, 0.7.1
>
>
> There is a case where StreamDemarcator can return the wrong data for a token.
> If a token ends at the end of the buffer, and the next token is smaller than
> the previous, it can result in the next token keeping part of the buffer's
> content. The code below is a unit test that exposes this:
> {code}
> @Test
> public void testOnBufferSplitNoTrailingDelimiter() throws IOException {
> final byte[] inputData = "Yes\nNo".getBytes(StandardCharsets.UTF_8);
> ByteArrayInputStream is = new ByteArrayInputStream(inputData);
> StreamDemarcator scanner = new StreamDemarcator(is, "\n".getBytes(),
> 1000, 3);
> final byte[] first = scanner.nextToken();
> final byte[] second = scanner.nextToken();
> assertNotNull(first);
> assertNotNull(second);
> assertArrayEquals(first, new byte[] {'Y', 'e', 's'});
> assertArrayEquals(second, new byte[] {'N', 'o'});
> }
> {code}
> In this case, the second token, which should be 'No' comes back as 'Nos'
> because it contains the 's' from the previous token.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-2875) Provide a Caching FlowFileQueue that will cache a small number of FlowFiles to improve performance
Mark Payne created NIFI-2875: Summary: Provide a Caching FlowFileQueue that will cache a small number of FlowFiles to improve performance Key: NIFI-2875 URL: https://issues.apache.org/jira/browse/NIFI-2875 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Some Processors expect to pull a very large number of FlowFiles at a very fast rate. When this happens, there can be a great deal of contention of the StandardFlowFileQueue. We should create a CachingFlowFileQueue that will wrap another FlowFileQueue. When poll() is called, it will call poll(int) on the underlying FlowFileQueue to poll some number of FlowFiles (10 or 100, perhaps). It can then cache these FlowFiles so that subsequent calls to poll() returns the cached FlowFiles. This avoid a great deal of lock contention on StandardFlowFileQueue because that queue must perform a significant amount of work within its lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2850) Provide ability for a FlowFile to be migrated from one Process Session to another
[ https://issues.apache.org/jira/browse/NIFI-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2850: - Status: Patch Available (was: Open) > Provide ability for a FlowFile to be migrated from one Process Session to > another > - > > Key: NIFI-2850 > URL: https://issues.apache.org/jira/browse/NIFI-2850 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > Currently, the MergeContent processor creates a separate ProcessSession for > each FlowFile that it pulls. This is done so that we can ensure that we can > commit all Process Sessions when a bin is full. Unfortunately, this means > that MergeContent is required to call ProcessSession.get() many times, which > adds a lot of contention on the FlowFile Queue. If we allow FlowFiles to be > migrated from 1 session to another, we can have a session per bin, and then > use ProcessSession.get(100) to greatly reduce lock contention. This will > likely have benefits in other processors as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2874) StreamDemarcator can return wrong data for token
[
https://issues.apache.org/jira/browse/NIFI-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-2874:
-
Status: Patch Available (was: Open)
> StreamDemarcator can return wrong data for token
>
>
> Key: NIFI-2874
> URL: https://issues.apache.org/jira/browse/NIFI-2874
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.1.0, 0.7.1
>
>
> There is a case where StreamDemarcator can return the wrong data for a token.
> If a token ends at the end of the buffer, and the next token is smaller than
> the previous, it can result in the next token keeping part of the buffer's
> content. The code below is a unit test that exposes this:
> {code}
> @Test
> public void testOnBufferSplitNoTrailingDelimiter() throws IOException {
> final byte[] inputData = "Yes\nNo".getBytes(StandardCharsets.UTF_8);
> ByteArrayInputStream is = new ByteArrayInputStream(inputData);
> StreamDemarcator scanner = new StreamDemarcator(is, "\n".getBytes(),
> 1000, 3);
> final byte[] first = scanner.nextToken();
> final byte[] second = scanner.nextToken();
> assertNotNull(first);
> assertNotNull(second);
> assertArrayEquals(first, new byte[] {'Y', 'e', 's'});
> assertArrayEquals(second, new byte[] {'N', 'o'});
> }
> {code}
> In this case, the second token, which should be 'No' comes back as 'Nos'
> because it contains the 's' from the previous token.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2856) Queue Listing - Cluster node id is missing
[
https://issues.apache.org/jira/browse/NIFI-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-2856:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Queue Listing - Cluster node id is missing
> --
>
> Key: NIFI-2856
> URL: https://issues.apache.org/jira/browse/NIFI-2856
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.1.0
>
>
> The cluster node id appears to be missing when performing a queue listing in
> clustered mode. As a result, subsequent attempts to directly access that
> flowfile fail with a message indicating that
> {noformat}The id of the node in the cluster is required{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2777) Provenance Events' Node Identifier not set when querying only 1 node in cluster
[ https://issues.apache.org/jira/browse/NIFI-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2777: - Resolution: Fixed Status: Resolved (was: Patch Available) > Provenance Events' Node Identifier not set when querying only 1 node in > cluster > --- > > Key: NIFI-2777 > URL: https://issues.apache.org/jira/browse/NIFI-2777 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Matt Gilman > Fix For: 1.1.0 > > > If I open the Provenance page and search for a FlowFile UUID and restrict the > search to a specific node, the Node Identifier is not populated in the events > that are returned. As a result, I cannot view the lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1280) Create FilterCSVColumns Processor
[ https://issues.apache.org/jira/browse/NIFI-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549852#comment-15549852 ] Mark Payne commented on NIFI-1280: -- [~Toivo Adams] I'm not sure what you mean by "indexing FlowFile content." In general, the content won't need to be read more than once. Even if the content has to be consumed multiple times, modern OS's have phenomenal disk caching mechanisms. If you heavy volumes of data through NiFi and you run "iostat -xmh 5" on a linux system, for instance, you'll see that the number of disk reads is almost 0 because the disk caching is so efficient. So I'm not too concerned about rereading the data. I did make some modifications to the processor... allowed the Enumerator to be created multiple times and refactored so that the data format can be extracted away from the core logic. At this point, the processor still only accepts CSV data but I'm hoping to open it up to more data formats than that. I will hopefully put a PR up soon that has the updated code, though. Would love for you to check it out and see if you've got any feedback. > Create FilterCSVColumns Processor > - > > Key: NIFI-1280 > URL: https://issues.apache.org/jira/browse/NIFI-1280 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Toivo Adams > > We should have a Processor that allows users to easily filter out specific > columns from CSV data. For instance, a user would configure two different > properties: "Columns of Interest" (a comma-separated list of column indexes) > and "Filtering Strategy" (Keep Only These Columns, Remove Only These Columns). > We can do this today with ReplaceText, but it is far more difficult than it > would be with this Processor, as the user has to use Regular Expressions, > etc. with ReplaceText. > Eventually a Custom UI could even be built that allows a user to upload a > Sample CSV and choose which columns from there, similar to the way that Excel > works when importing CSV by dragging and selecting the desired columns? That > would certainly be a larger undertaking and would not need to be done for an > initial implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1280) Create FilterCSVColumns Processor
[ https://issues.apache.org/jira/browse/NIFI-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549853#comment-15549853 ] Mark Payne commented on NIFI-1280: -- [~Toivo Adams] I'm not sure what you mean by "indexing FlowFile content." In general, the content won't need to be read more than once. Even if the content has to be consumed multiple times, modern OS's have phenomenal disk caching mechanisms. If you heavy volumes of data through NiFi and you run "iostat -xmh 5" on a linux system, for instance, you'll see that the number of disk reads is almost 0 because the disk caching is so efficient. So I'm not too concerned about rereading the data. I did make some modifications to the processor... allowed the Enumerator to be created multiple times and refactored so that the data format can be extracted away from the core logic. At this point, the processor still only accepts CSV data but I'm hoping to open it up to more data formats than that. I will hopefully put a PR up soon that has the updated code, though. Would love for you to check it out and see if you've got any feedback. > Create FilterCSVColumns Processor > - > > Key: NIFI-1280 > URL: https://issues.apache.org/jira/browse/NIFI-1280 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Toivo Adams > > We should have a Processor that allows users to easily filter out specific > columns from CSV data. For instance, a user would configure two different > properties: "Columns of Interest" (a comma-separated list of column indexes) > and "Filtering Strategy" (Keep Only These Columns, Remove Only These Columns). > We can do this today with ReplaceText, but it is far more difficult than it > would be with this Processor, as the user has to use Regular Expressions, > etc. with ReplaceText. > Eventually a Custom UI could even be built that allows a user to upload a > Sample CSV and choose which columns from there, similar to the way that Excel > works when importing CSV by dragging and selecting the desired columns? That > would certainly be a larger undertaking and would not need to be done for an > initial implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2836) Cluster - Request Replication
[ https://issues.apache.org/jira/browse/NIFI-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2836: - Status: Patch Available (was: Open) > Cluster - Request Replication > - > > Key: NIFI-2836 > URL: https://issues.apache.org/jira/browse/NIFI-2836 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Matt Gilman >Assignee: Mark Payne >Priority: Critical > Fix For: 1.1.0 > > > When a request is replicated throughout the cluster, the cluster coordinator > needs to ensure that request completes (including both parts of the two-phase > commit if applicable) prior to replicating the next request. > Looks like a write lock is obtained while submitted a background task. The > actually work completes outside the locks scope. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2836) Cluster - Request Replication
[ https://issues.apache.org/jira/browse/NIFI-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-2836: Assignee: Mark Payne > Cluster - Request Replication > - > > Key: NIFI-2836 > URL: https://issues.apache.org/jira/browse/NIFI-2836 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Matt Gilman >Assignee: Mark Payne >Priority: Critical > Fix For: 1.1.0 > > > When a request is replicated throughout the cluster, the cluster coordinator > needs to ensure that request completes (including both parts of the two-phase > commit if applicable) prior to replicating the next request. > Looks like a write lock is obtained while submitted a background task. The > actually work completes outside the locks scope. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2794) Expose a getOneFromEachConnection method in ProcessSession
[ https://issues.apache.org/jira/browse/NIFI-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509891#comment-15509891 ] Mark Payne commented on NIFI-2794: -- [~pvillard] - the problem with what you have laid out above is that in order for this to work, you'd need to be able to guarantee that the queues between B/D and C/D are "in sync" so that the first FlowFile in B/D matches up with the first FlowFile in C/D and the second FlowFile in B/D matches up with the second FlowFIle in C/D, etc. NiFi does not provide any sort of ordering guarantee that spans multiple connections. Instead, the approach that we take is to have processor A add some sort of "correlation id" attribute, and then let D group the FlowFiles by this attribute internally. This is how Merge Content works, for example. Using this approach, we can guarantee that FlowFiles are properly grouped together, and it means that we don't need to care about how many incoming Connections a Processor has. > Expose a getOneFromEachConnection method in ProcessSession > -- > > Key: NIFI-2794 > URL: https://issues.apache.org/jira/browse/NIFI-2794 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > Attachments: image.png > > > In case a processor has multiple incoming connections, it could be > interesting to expose a method allowing users to get exactly one flow file > per incoming connection in a row. > That could unlock opportunities such as performing join operations on > datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2545) Cannot tell a processor is "STOPPING", can edit, cannot restart it
[
https://issues.apache.org/jira/browse/NIFI-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-2545:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Cannot tell a processor is "STOPPING", can edit, cannot restart it
> --
>
> Key: NIFI-2545
> URL: https://issues.apache.org/jira/browse/NIFI-2545
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Core UI
>Affects Versions: 1.0.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.0.0
>
>
> Testing ListenSMTP processor. It has some bad behavior whereby it will not
> stop when told to. But on the UI i cannot tell that at all. So, i went to
> edit the processor config which I was able to do. Then went to start it and
> I get
> {quote}
> 7667f0c7-0156-1000-8181-5b556f8544da cannot be started because it is not
> stopped. Current state is STOPPING
> {quote}
> There should be a visual indicator of its status and we should not be able to
> edit a processor which is not stopped.
> More details on that specific case here
> https://issues.apache.org/jira/browse/NIFI-2519#
> But key is that a bad behaving processor can allow the user to do things they
> should not be able to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426409#comment-15426409 ] Mark Payne commented on NIFI-2383: -- There was a discussion about this on the mailing list a while back. We should really have a configurable way for the user to specify how "new files" should be detected. The current approach does indeed leave out any file that is copied into the directory if it is copied in a way that preserves the Last Modified timestamp. However, we went with this approach for the initial solution because it is extremely efficient and it allows us to avoid storing very much information. If we were to store enough information to pick up a second copy of that file, we would have to store information about each file that is in the directory, etc., and that can be very prohibitive in a clustered environment. So offering the ability to let the user configure which approach makes sense for them is a win-win. That being said, though, I don't think this is an improvement that we need to be scheduling for a 1.0.0 release. We are closing in on the release, and I suspect within the next couple of days we will have an RC out, so I would avoid putting anything on the 1.0.0 release unless it is of a critical bug-fix nature. I'll remove the fix version for now. If someone wants to pick this up though and supply a patch we can figure out at that point what version it should belong in. > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: n h > Labels: Core, ListFile > Fix For: 1.0.0, 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2382) Typo in ListFile Documentation
[ https://issues.apache.org/jira/browse/NIFI-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426416#comment-15426416 ] Mark Payne commented on NIFI-2382: -- Good catch. It was indeed supposed to be "FetchFile." Fixed and pushed to both 0.x and master branches. > Typo in ListFile Documentation > -- > > Key: NIFI-2382 > URL: https://issues.apache.org/jira/browse/NIFI-2382 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: n h >Priority: Minor > Labels: Documentation, ListFile > Fix For: 1.0.0, 0.7.1 > > > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with ListFile." > I guess it might be: > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with FetchFile." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2382) Typo in ListFile Documentation
[ https://issues.apache.org/jira/browse/NIFI-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2382: - Summary: Typo in ListFile Documentation (was: ListFile Documentation) > Typo in ListFile Documentation > -- > > Key: NIFI-2382 > URL: https://issues.apache.org/jira/browse/NIFI-2382 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: n h >Priority: Minor > Labels: Documentation, ListFile > Fix For: 1.0.0, 0.7.1 > > > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with ListFile." > I guess it might be: > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with FetchFile." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2383: - Issue Type: Improvement (was: Bug) > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: n h > Labels: ListFile > Fix For: 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2383: - Labels: ListFile (was: Core ListFile) > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: n h > Labels: ListFile > Fix For: 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2383: - Component/s: (was: Core Framework) Extensions > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: n h > Labels: ListFile > Fix For: 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2383: - Fix Version/s: (was: 1.0.0) > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: n h > Labels: ListFile > Fix For: 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-2382) Typo in ListFile Documentation
[ https://issues.apache.org/jira/browse/NIFI-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-2382. -- Resolution: Fixed > Typo in ListFile Documentation > -- > > Key: NIFI-2382 > URL: https://issues.apache.org/jira/browse/NIFI-2382 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: n h >Priority: Minor > Labels: Documentation, ListFile > Fix For: 1.0.0, 0.7.1 > > > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with ListFile." > I guess it might be: > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with FetchFile." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2382) Typo in ListFile Documentation
[ https://issues.apache.org/jira/browse/NIFI-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-2382: Assignee: Mark Payne > Typo in ListFile Documentation > -- > > Key: NIFI-2382 > URL: https://issues.apache.org/jira/browse/NIFI-2382 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: n h >Assignee: Mark Payne >Priority: Minor > Labels: Documentation, ListFile > Fix For: 1.0.0, 0.7.1 > > > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with ListFile." > I guess it might be: > "For each file that is listed,creates a FlowFile that represents the file so > that it can be fetched in conjunction with FetchFile." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2597) Unable to create self-loop connection from existing connection
Mark Payne created NIFI-2597: Summary: Unable to create self-loop connection from existing connection Key: NIFI-2597 URL: https://issues.apache.org/jira/browse/NIFI-2597 Project: Apache NiFi Issue Type: Bug Components: Core UI Reporter: Mark Payne Fix For: 1.0.0 If I have a connection that goes from Processor A to Processor B (both stopped), and then I drag the destination of the connection from Processor B to Processor A, so that it is a self-loop for Processor A, the connection stays connected to Processor B. The UI doesn't create the multiple bend-points on the connection but does appear to make the connection self-looping. When I click on the 'blue dot' on the connection that represents its destination, the connection is re-drawn to go to the destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2600) Provenance Repository can close Index Reader too soon
Mark Payne created NIFI-2600: Summary: Provenance Repository can close Index Reader too soon Key: NIFI-2600 URL: https://issues.apache.org/jira/browse/NIFI-2600 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.0.0 In NIFI-2452, we addressed an issue where the Persistent Provenance Repo closed Index Readers / Index Searchers before they should. I have found another case in which that can happen. Using a separate JIRA rather than re-opening the other, since NIFI-2452 has a fix Version of both 1.0.0-BETA and 1.0.0 and 1.0.0-BETA has already been released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2383) ListFile does not detect new files
[ https://issues.apache.org/jira/browse/NIFI-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427028#comment-15427028 ] Mark Payne commented on NIFI-2383: -- When you say a watching service, do you mean have ListFile watch a directory and be notified of changes to the directory? That could certainly be a good approach to take, for efficiency purposes, but we have to be very careful because we cannot rely solely on that; if NiFi restarts we will not have received those notifications, so we will have to have a way to detect new files after restart as well. > ListFile does not detect new files > -- > > Key: NIFI-2383 > URL: https://issues.apache.org/jira/browse/NIFI-2383 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: n h > Labels: ListFile > Fix For: 0.7.1 > > > ListFile does not detect directory updates correctly. It seems it only check > for file name and modified date. > Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to > finally delete files. Following scenario fails: > 1. Create file1.txt somewhere outside of watching directory. > 2. Copy file1.txt to watching directory. > 3. Wait for FetchFile to fetch and delete file1.txt > 4. Copy file1.txt again to watching directory. > In the 2nd time ListFile does not detect "new copy of the same file > file1.txt" because file name and modified time are the same. Only creation > time is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2587) Need additional authorization on Component History
[ https://issues.apache.org/jira/browse/NIFI-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2587: - Resolution: Fixed Status: Resolved (was: Patch Available) > Need additional authorization on Component History > -- > > Key: NIFI-2587 > URL: https://issues.apache.org/jira/browse/NIFI-2587 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Critical > Fix For: 1.0.0 > > > When loading property history for a given component, need additional > authorization checks for the underlying component. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2597) Unable to create self-loop connection from existing connection
[ https://issues.apache.org/jira/browse/NIFI-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2597: - Resolution: Fixed Status: Resolved (was: Patch Available) > Unable to create self-loop connection from existing connection > -- > > Key: NIFI-2597 > URL: https://issues.apache.org/jira/browse/NIFI-2597 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Mark Payne >Assignee: Matt Gilman > Fix For: 1.0.0 > > > If I have a connection that goes from Processor A to Processor B (both > stopped), and then I drag the destination of the connection from Processor B > to Processor A, so that it is a self-loop for Processor A, the connection > stays connected to Processor B. The UI doesn't create the multiple > bend-points on the connection but does appear to make the connection > self-looping. When I click on the 'blue dot' on the connection that > represents its destination, the connection is re-drawn to go to the > destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2669) HTTP-based Site-to-Site leaks threads/memory when unable to communicate with remote instance
Mark Payne created NIFI-2669: Summary: HTTP-based Site-to-Site leaks threads/memory when unable to communicate with remote instance Key: NIFI-2669 URL: https://issues.apache.org/jira/browse/NIFI-2669 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Priority: Blocker Fix For: 1.0.0 I am using HTTP-based site-to-site to communicate with a remote instance of NiFi. I am seeing problems communicating, which is okay. But I am seeing CPU load of 600-700 on my system with 32 cores. I am also seeing in a thread dump 10's to 100's threads with the name "I/O dispatcher" and am seeing OOME's. A heap dump shows that the OOME is caused by 64K byte arrays, each being filled with 0's. The byte[] is coming from class org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2603) Bringing Some UI Color Back
[ https://issues.apache.org/jira/browse/NIFI-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439321#comment-15439321 ] Mark Payne commented on NIFI-2603: -- I'm onboard with using a red color to indicate that a processor is stopped, but that would help to draw attention to the fact that you have stopped processors. Generally, I don't think we should have processors stopped for a long period of time. If the idea is that we don't want the processor running for a long period of time, we should probably disable the processor. I've certainly seen cases though where we stop a processor to change configuration and then forget to start it again. So I think the red color on the graph would help with that. However, the 'normal' case is for processors to be running, and personally I think it should stay the same color as it is now. I find the green to be a bit distracting because it stands out and draws attention to a 'normal' condition. With respect to changing the 'warning' icon color from red to yellow, I personally prefer the red but don't have a strong preference. > Bringing Some UI Color Back > --- > > Key: NIFI-2603 > URL: https://issues.apache.org/jira/browse/NIFI-2603 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Peter Wicks >Priority: Minor > > In the new 1.0 UI all of the colors associated with status (except the orange > triangle) are gone; replaced with a dark gray color. > I propose bringing back color. The screenshots are in the format of before > on the top and after on the bottom, except were labeled in the picture itself: > - Top Status Menu: https://goo.gl/photos/se1JnvhRwU7N4Fap7 > - Process Group: https://goo.gl/photos/dqjG4KvC6xqxQfgT7 > - Processes (Running/Stopped/Invalid): > https://goo.gl/photos/dSS8vgE2RkrXtc77A > - Operate Play/Stop buttons (only on mouse hover): > https://goo.gl/photos/Am5SUEEn7G9RjmMe6 > - Processor/Processor Group Context Menu: > https://goo.gl/photos/Jq3qFg4ezaN91qms5 > This is not a "100% done, I've covered everything" before/after list. I know > I need to do the NiFi summary page also at the minimum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2678) When clicking 'stop' on a processor, the UI sometimes changes the icon to stop and then immediately back to start.
Mark Payne created NIFI-2678: Summary: When clicking 'stop' on a processor, the UI sometimes changes the icon to stop and then immediately back to start. Key: NIFI-2678 URL: https://issues.apache.org/jira/browse/NIFI-2678 Project: Apache NiFi Issue Type: Bug Components: Core UI Reporter: Mark Payne Priority: Minor Fix For: 1.1.0 When I click 'Stop' on a processor, sometimes the icon changes to show that it's stopped but then a split second later changes back to showing the processor as running. If I right-click and refresh, the icon changes back to stop. This appears to be a race condition between the 'stop' request and the background 'refresh status' request. I believe we have the revision information included in the status request, so we should be able to avoid changing the icon if the status request indicates a lower revision than the current one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2679) Unable to disable processor that is invalid through Operate palette
Mark Payne created NIFI-2679: Summary: Unable to disable processor that is invalid through Operate palette Key: NIFI-2679 URL: https://issues.apache.org/jira/browse/NIFI-2679 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.0.0 Reporter: Mark Payne Priority: Minor Fix For: 1.1.0 If I have an invalid processor, I cannot click the 'disable' button in the operate palette. The UI appears to check only if the processor is stopped, rather than checking if it's disabled. I can, however, click Configure and un-check the 'Enabled' checkbox from the Configure dialog. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2679) Unable to disable processor that is invalid through Operate palette
[ https://issues.apache.org/jira/browse/NIFI-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2679: - Description: If I have an invalid processor, I cannot click the 'disable' button in the operate palette. The UI appears to check only if the processor is stopped, rather than checking if it's stopped or invalid. I can, however, click Configure and un-check the 'Enabled' checkbox from the Configure dialog. was: If I have an invalid processor, I cannot click the 'disable' button in the operate palette. The UI appears to check only if the processor is stopped, rather than checking if it's disabled. I can, however, click Configure and un-check the 'Enabled' checkbox from the Configure dialog. > Unable to disable processor that is invalid through Operate palette > --- > > Key: NIFI-2679 > URL: https://issues.apache.org/jira/browse/NIFI-2679 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Mark Payne >Priority: Minor > Fix For: 1.1.0 > > > If I have an invalid processor, I cannot click the 'disable' button in the > operate palette. The UI appears to check only if the processor is stopped, > rather than checking if it's stopped or invalid. > I can, however, click Configure and un-check the 'Enabled' checkbox from the > Configure dialog. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2681) Avoid caching Provenance Index Searchers
Mark Payne created NIFI-2681: Summary: Avoid caching Provenance Index Searchers Key: NIFI-2681 URL: https://issues.apache.org/jira/browse/NIFI-2681 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Priority: Critical Fix For: 1.1.0 In NIFI-2600 and NIFI-2452, we addressed two bugs where the Provenance Repository closes a cached IndexSearcher too soon. The IndexManager keeps the searchers cached in an effort to offer better performance when performing a Provenance Query. This was done because it was recommended in the Lucene documentation. However, we occasionally still see nodes crashing with segfaults due to the Lucene Searching. We should update the Persistent Provenance Repository to stop caching Index Searchers in order to trade a slight performance improvement for significantly better reliability. Playing around with the idea in order to test it out shows very favorable results. On a system where I could cause a seg fault almost every time that I ran a large provenance query, I updated the code to no longer cache the readers and saw perfect stability with no noticeable performance degradation. I will cleanup the code and submit a PR for these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2447) When calculating lineage for a FlowFile, should provide a mechanism for starting with a particular event instead of FlowFile UUID
[ https://issues.apache.org/jira/browse/NIFI-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2447: - Fix Version/s: (was: 1.1.0) 1.0.0 > When calculating lineage for a FlowFile, should provide a mechanism for > starting with a particular event instead of FlowFile UUID > - > > Key: NIFI-2447 > URL: https://issues.apache.org/jira/browse/NIFI-2447 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Mark Payne > Fix For: 1.0.0 > > > Currently, whenever a user clicks the lineage button in the Provenance UI, we > have the ID of the Provenance Event that the user wants to generate a lineage > for. However, we pass in only the UUID of the FlowFile. > If we pass in the Event ID instead, we are able to determine the Lineage > Start Date from that event, which is very efficient. We can then search for > events in the lineage with that start date. Currently, we search the entirety > of the Provenance Repository in order to create a lineage, but this is very > inefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-2447) When calculating lineage for a FlowFile, should provide a mechanism for starting with a particular event instead of FlowFile UUID
[ https://issues.apache.org/jira/browse/NIFI-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-2447. -- Resolution: Fixed Assignee: Mark Payne > When calculating lineage for a FlowFile, should provide a mechanism for > starting with a particular event instead of FlowFile UUID > - > > Key: NIFI-2447 > URL: https://issues.apache.org/jira/browse/NIFI-2447 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > > Currently, whenever a user clicks the lineage button in the Provenance UI, we > have the ID of the Provenance Event that the user wants to generate a lineage > for. However, we pass in only the UUID of the FlowFile. > If we pass in the Event ID instead, we are able to determine the Lineage > Start Date from that event, which is very efficient. We can then search for > events in the lineage with that start date. Currently, we search the entirety > of the Provenance Repository in order to create a lineage, but this is very > inefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2447) When calculating lineage for a FlowFile, should provide a mechanism for starting with a particular event instead of FlowFile UUID
[ https://issues.apache.org/jira/browse/NIFI-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15444276#comment-15444276 ] Mark Payne commented on NIFI-2447: -- This issue was resolved in the PR for NIFI-2452. > When calculating lineage for a FlowFile, should provide a mechanism for > starting with a particular event instead of FlowFile UUID > - > > Key: NIFI-2447 > URL: https://issues.apache.org/jira/browse/NIFI-2447 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Mark Payne > Fix For: 1.1.0 > > > Currently, whenever a user clicks the lineage button in the Provenance UI, we > have the ID of the Provenance Event that the user wants to generate a lineage > for. However, we pass in only the UUID of the FlowFile. > If we pass in the Event ID instead, we are able to determine the Lineage > Start Date from that event, which is very efficient. We can then search for > events in the lineage with that start date. Currently, we search the entirety > of the Provenance Repository in order to create a lineage, but this is very > inefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2606) FingerprintFactory and default property values
[ https://issues.apache.org/jira/browse/NIFI-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2606: - Resolution: Fixed Status: Resolved (was: Patch Available) > FingerprintFactory and default property values > -- > > Key: NIFI-2606 > URL: https://issues.apache.org/jira/browse/NIFI-2606 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Matt Gilman > Fix For: 1.0.0 > > > When a node joins a cluster, it checks the proposed flow (of the cluster) > against it's local flow. The proposed flow will include default property > values if a property is not explicitly set. This is accounted for when > comparing a Processor but is not for Controller Services and Reporting Tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2634) Cannot import template from 0.x that contains Remote Process Group
[ https://issues.apache.org/jira/browse/NIFI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432862#comment-15432862 ] Mark Payne commented on NIFI-2634: -- Excellent, thanks, Joe! I should be able to kick out a PR for this in just a bit. > Cannot import template from 0.x that contains Remote Process Group > -- > > Key: NIFI-2634 > URL: https://issues.apache.org/jira/browse/NIFI-2634 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.0.0 > > > As a user, when attempting to add a template from 0.x that contains an RPG to > the graph I get a screen saying "unexpected error". After returning to the > canvas, all the processors were added to the graph but there are not process > groups, connections or remote process groups. In the user logs there is the > below exception. > I believe this is due to the template not having the type of RPG (raw vs > http). > 2016-08-23 10:03:41,157 ERROR [NiFi Web Server-138] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException: Name is null. Returning Internal Server Error > response. > java.lang.NullPointerException: Name is null > at java.lang.Enum.valueOf(Enum.java:236) ~[na:1.8.0_74] > at > org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol.valueOf(SiteToSiteTransportProtocol.java:19) > ~[nifi-site-to-site-client-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.FlowController.instantiateSnippet(FlowController.java:1744) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.web.dao.impl.StandardTemplateDAO.instantiateTemplate(StandardTemplateDAO.java:134) > ~[classes/:na] > at > org.apache.nifi.web.StandardNiFiServiceFacade.createTemplateInstance(StandardNiFiServiceFacade.java:1727) > ~[classes/:1.0.0-SNAPSHOT] > at > org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke() > ~[classes/:1.0.0-SNAPSHOT] > at > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > ~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.apache.nifi.web.NiFiServiceFacadeLock.createLock(NiFiServiceFacadeLock.java:41) > ~[classes/:1.0.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_74] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_74] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_74] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74] > at > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) > ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE] > at > org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$3ebe455f.createTemplateInstance() > ~[classes/:1.0.0-SNAPSHOT] > at > org.apache.nifi.web.api.ProcessGroupResource.instantiateTemplate(ProcessGroupResource.java:1797) > ~[classes/:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >
[jira] [Commented] (NIFI-2634) Templates assume RPGs populate the type field
[ https://issues.apache.org/jira/browse/NIFI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432853#comment-15432853 ] Mark Payne commented on NIFI-2634: -- [~JPercivall] can you share the full stack trace, if you have it? > Templates assume RPGs populate the type field > - > > Key: NIFI-2634 > URL: https://issues.apache.org/jira/browse/NIFI-2634 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Priority: Blocker > Fix For: 1.0.0 > > > When adding a template from the 0.x line that uses RPGs into 1.X it fails > with the following error in the user logs: > 2016-08-23 10:03:41,157 ERROR [NiFi Web Server-138] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException: Name is null. Returning Internal Server Error > response. > java.lang.NullPointerException: Name is null > This is due to the template not having the type of RPG (raw vs http). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2634) Templates assume RPGs populate the type field
[ https://issues.apache.org/jira/browse/NIFI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-2634: Assignee: Mark Payne > Templates assume RPGs populate the type field > - > > Key: NIFI-2634 > URL: https://issues.apache.org/jira/browse/NIFI-2634 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.0.0 > > > When adding a template from the 0.x line that uses RPGs into 1.X it fails > with the following error in the user logs: > 2016-08-23 10:03:41,157 ERROR [NiFi Web Server-138] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException: Name is null. Returning Internal Server Error > response. > java.lang.NullPointerException: Name is null > This is due to the template not having the type of RPG (raw vs http). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2669) HTTP-based Site-to-Site leaks threads/memory when unable to communicate with remote instance
[ https://issues.apache.org/jira/browse/NIFI-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2669: - Status: Patch Available (was: Open) > HTTP-based Site-to-Site leaks threads/memory when unable to communicate with > remote instance > > > Key: NIFI-2669 > URL: https://issues.apache.org/jira/browse/NIFI-2669 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.0.0 > > > I am using HTTP-based site-to-site to communicate with a remote instance of > NiFi. I am seeing problems communicating, which is okay. But I am seeing CPU > load of 600-700 on my system with 32 cores. I am also seeing in a thread dump > 10's to 100's threads with the name "I/O dispatcher" and am seeing OOME's. A > heap dump shows that the OOME is caused by 64K byte arrays, each being filled > with 0's. The byte[] is coming from class > org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2670) ConsumeKafka should support a configurable option for how to encode kafka key as attribute
[ https://issues.apache.org/jira/browse/NIFI-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-2670: Assignee: Mark Payne > ConsumeKafka should support a configurable option for how to encode kafka key > as attribute > -- > > Key: NIFI-2670 > URL: https://issues.apache.org/jira/browse/NIFI-2670 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 1.0.0 > > > Previously, ConsumeKafka / GetKafka added the kafka key as an attribute to > the flowfile, assuming that the key was in UTF-8 format. Recently, this was > changed to use a hex encoding. This could break flows, as it is not backward > compatible. We should instead make this a configurable option with the > default being UTF-8 Encoded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2670) ConsumeKafka should support a configurable option for how to encode kafka key as attribute
Mark Payne created NIFI-2670: Summary: ConsumeKafka should support a configurable option for how to encode kafka key as attribute Key: NIFI-2670 URL: https://issues.apache.org/jira/browse/NIFI-2670 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Mark Payne Priority: Critical Fix For: 1.0.0 Previously, ConsumeKafka / GetKafka added the kafka key as an attribute to the flowfile, assuming that the key was in UTF-8 format. Recently, this was changed to use a hex encoding. This could break flows, as it is not backward compatible. We should instead make this a configurable option with the default being UTF-8 Encoded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2854) Allow Provenance Repository to roll back to a previous implementation
Mark Payne created NIFI-2854: Summary: Allow Provenance Repository to roll back to a previous implementation Key: NIFI-2854 URL: https://issues.apache.org/jira/browse/NIFI-2854 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 Currently, if we start up a new version of NiFi and the format of the Data Provenance data has changed, we are not able to roll back to a previous version of NiFi. If we do, NiFi will fail to read the Provenance Data and not start up. We should instead provide the ability to write data to the repository in such a way that old versions of the repository will still be able to read the data, so that we can roll back -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2850) Provide ability for a FlowFile to be migrated from one Process Session to another
Mark Payne created NIFI-2850: Summary: Provide ability for a FlowFile to be migrated from one Process Session to another Key: NIFI-2850 URL: https://issues.apache.org/jira/browse/NIFI-2850 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 Currently, the MergeContent processor creates a separate ProcessSession for each FlowFile that it pulls. This is done so that we can ensure that we can commit all Process Sessions when a bin is full. Unfortunately, this means that MergeContent is required to call ProcessSession.get() many times, which adds a lot of contention on the FlowFile Queue. If we allow FlowFiles to be migrated from 1 session to another, we can have a session per bin, and then use ProcessSession.get(100) to greatly reduce lock contention. This will likely have benefits in other processors as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2851) Improve performance of SplitText
Mark Payne created NIFI-2851: Summary: Improve performance of SplitText Key: NIFI-2851 URL: https://issues.apache.org/jira/browse/NIFI-2851 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0 SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 1.4 million line text file into 5k line chunks and then splits those 5k line chunks into 1 line chunks is only capable of pushing through about 10k lines per second. This equates to about 10 MB/sec. JVisualVM shows that the majority of the time is spent in the locateSplitPoint() method. Isolating this code and inspecting how it works, and using some micro-benchmarking, it appears that if we refactor the calls to InputStream.read() to instead read into a byte array, we can improve performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1991) Zero Master Cluster: Modifying a disconnected node causes other nodes to not be able to join cluster
[ https://issues.apache.org/jira/browse/NIFI-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563064#comment-15563064 ] Mark Payne commented on NIFI-1991: -- [~andrewmlim] we have not really done anything in 1.1.0 at this time that explicitly was designed to address this issue. However, I think that this has been addressed or greatly mitigated by other features in 1.1.0. Specifically, nodes in a cluster must now vote on which flow is the correct flow (NIFI-1966). The flow won't be available until either enough nodes have joined the cluster or some timeout occurs. This means that if you restart an entire cluster at the same time, you will not be able to modify anything until at least some configurable nodes have connected to the cluster. In addition to this, if you browse to a NiFi instance that has not yet connected to the cluster, you will receive a warning pop-up notifying you that the node is configured to join a cluster but has not yet been able to join. Do you think this is sufficient, or do you have other ideas that you'd like to see implemented here? > Zero Master Cluster: Modifying a disconnected node causes other nodes to not > be able to join cluster > - > > Key: NIFI-1991 > URL: https://issues.apache.org/jira/browse/NIFI-1991 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Andrew Lim >Priority: Blocker > Fix For: 1.1.0 > > > I created a zero master cluster with 3 nodes. All three were up and running. > I stopped all 3 instances and then brought up just Node1. > In the Node1 UI, I made some edits to the flow. > I attempted but was not able to bring up Node2 and Node3 to join the cluster. > Here is what I saw in the logs: > 2016-06-09 14:35:47,051 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > java.lang.Exception: Unable to load flow due to: java.io.IOException: > org.apache.nifi.controller.UninheritableFlowException: Failed to connect node > to cluster because local flow is different than cluster flow. > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753) > ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.NiFi.(NiFi.java:137) > [nifi-runtime-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.NiFi.main(NiFi.java:227) > [nifi-runtime-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > Caused by: java.io.IOException: > org.apache.nifi.controller.UninheritableFlowException: Failed to connect node > to cluster because local flow is different than cluster flow. > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:501) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:744) > ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > ... 2 common frames omitted > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to > connect node to cluster because local flow is different than cluster flow. > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:862) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > ... 3 common frames omitted > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed > configuration is not inheritable by the flow controller because of flow > differences: Found difference in Flows: > Local Fingerprint contains additional configuration from Cluster Fingerprint: > eba77dac-32ad-356b-8a82-f669d046aa21eba77dac-32ad-356b-8a82-f669d046aa21org.apache.nifi.processors.standard.LogAttributeNO_VALUEAttributes > to IgnoreNO_VALUEAttributes to LogNO_VALUELog prefixNO_VALUEs > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:219) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1329) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:75) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:668) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:839) >
[jira] [Commented] (NIFI-2991) JSON.org license is now CatX
[
https://issues.apache.org/jira/browse/NIFI-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15650948#comment-15650948
]
Mark Payne commented on NIFI-2991:
--
For nifi-standard-nar, as Joe mentioned, it is brought in through
nifi-processors. It appears that this is brought in for the ValidationJson
Processor. However, this processor was added only on master and has not yet
been released. So we can remove this processor for the time being, if
necessary, until this is resolved, without it causing any issues with backward
compatibility.
> JSON.org license is now CatX
>
>
> Key: NIFI-2991
> URL: https://issues.apache.org/jira/browse/NIFI-2991
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Sean Busbey
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.1.0
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list.
> Prior to this, use of the JSON Java library was allowed. See Debian's page
> for a list of alternatives.
> {quote}
> I don't know how many of our versions include stuff under this license, it's
> definitely currently in master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2603) Bringing Some UI Color Back
[ https://issues.apache.org/jira/browse/NIFI-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2603: - Resolution: Fixed Status: Resolved (was: Patch Available) > Bringing Some UI Color Back > --- > > Key: NIFI-2603 > URL: https://issues.apache.org/jira/browse/NIFI-2603 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Peter Wicks >Assignee: Matt Gilman >Priority: Minor > Fix For: 1.1.0 > > Attachments: running-disabled-adjustment.png, > settings-shell-controller-services.png, settings-shell-reporting-tasks.png, > status-bars-and-components.png, summary-shell.png > > > In the new 1.0 UI all of the colors associated with status (except the orange > triangle) are gone; replaced with a dark gray color. > I propose bringing back color. The screenshots are in the format of before > on the top and after on the bottom, except were labeled in the picture itself: > - Top Status Menu: https://goo.gl/photos/se1JnvhRwU7N4Fap7 > - Process Group: https://goo.gl/photos/dqjG4KvC6xqxQfgT7 > - Processes (Running/Stopped/Invalid): > https://goo.gl/photos/dSS8vgE2RkrXtc77A > - Operate Play/Stop buttons (only on mouse hover): > https://goo.gl/photos/Am5SUEEn7G9RjmMe6 > - Processor/Processor Group Context Menu: > https://goo.gl/photos/Jq3qFg4ezaN91qms5 > This is not a "100% done, I've covered everything" before/after list. I know > I need to do the NiFi summary page also at the minimum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3002) Remove legacy code for retrieving Site To Site details
[ https://issues.apache.org/jira/browse/NIFI-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3002: - Resolution: Fixed Status: Resolved (was: Patch Available) > Remove legacy code for retrieving Site To Site details > -- > > Key: NIFI-3002 > URL: https://issues.apache.org/jira/browse/NIFI-3002 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Matt Gilman > Fix For: 1.1.0 > > > Earlier versions of Apache NiFi could potentially enter a distributed > deadlock when load balancing with a Remote Process Group. Retrieval of these > details was obtained within the framework as a remote call was technically > unnecessary. Subsequently, these actions were made asynchronous but the port > retrieval was left in place. This could lead to some confusion as the UI may > report some ports as available when technically policies may prevent data > transfers. By removing the legacy code, the UI will accurately reflect the > port availability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3022) DROP Provenance Events is not shown when user right-clicks on connection and chooses to Empty queue
Mark Payne created NIFI-3022: Summary: DROP Provenance Events is not shown when user right-clicks on connection and chooses to Empty queue Key: NIFI-3022 URL: https://issues.apache.org/jira/browse/NIFI-3022 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.0.0 Reporter: Mark Payne Fix For: 1.1.0 I dropped the FlowFiles from a FlowFile Queue and then was unable to see the Provenance Events. It appears that the events are created and read from the repository but are not being authorized properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2854: - Status: Patch Available (was: Open) > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2920) Swapped FlowFiles are not removed from content repo when a queue is emptied.
[ https://issues.apache.org/jira/browse/NIFI-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592346#comment-15592346 ] Mark Payne commented on NIFI-2920: -- After looking into this some more, I believe that what I have replicated here is actually a different bug that affects only version 1.0. The symptoms are similar if not identical. However, the problem appears to not be the same, as I can easily reproduce the problem in version 1.0 but cannot reproduce the problem in the 0.x baseline. I tried in both 0.7.0 and 0.6.1. Given that you have reported this as a bug in 0.6.1, I think they are different. I have created a new ticket (NIFI-2925) to address the issue that affects 1.0 and will submit a PR for that problem momentarily. I can see in the code that problem in 1.0 would not affect the 0.x branch. So I have marked the tickets as related but not the same. > Swapped FlowFiles are not removed from content repo when a queue is emptied. > > > Key: NIFI-2920 > URL: https://issues.apache.org/jira/browse/NIFI-2920 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 > Environment: Linux >Reporter: Matthew Clarke >Priority: Critical > > If a queue contains enough FlowFiles to trigger swapping to occur and a user > selects "empty queue" or sets file expiration, only the content claims > associated to FlowFiles that were not swapped get removed from the content > repository. All other content claims are left in the content repository and > are not moved to archive and/or purged. > A restart of NiFi will produce app log messages about these unknown files > left in the content repo and will at that time move them to archive or purge > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2925) FlowFiles that are swapped out are never released from Content Repository
Mark Payne created NIFI-2925:
Summary: FlowFiles that are swapped out are never released from
Content Repository
Key: NIFI-2925
URL: https://issues.apache.org/jira/browse/NIFI-2925
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.0.0
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Blocker
Fix For: 1.1.0
To reproduce this, I created a simple Flow: GenerateFlowFile (1 KB file size)
with success going to 2 different UpdateAttribute Processors (so that the same
Content Claim is held by 2 different FlowFiles). I let about 150,000 FlowFiles
queue up (with backpressure turned off). I then start one of the
UpdateAttribute processors. This drained its queue. I could then look at my
content repo for any files not archived:
{code}
content_repository $ find . -type f | grep -v archive | wc -l
192
{code}
After a few minutes, the FlowFile repo is checkpointed, which will result in
things getting cleaned up if they can. The above command shows the same result
(expected, since the FlowFiles are still held. I then empty the queue. After
the FlowFile checkpoints again, I should see nothing in the content repo
outside of archive, but I see:
{code}
content_repository $ find . -type f | grep -v archive | wc -l
167
{code}
I see the same thing happening if I turn on expiration to remove the FlowFiles
instead of clicking Empty Queue, or if a processor runs and completes the
processing of the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2920) Swapped FlowFiles are not removed from content repo when a queue is emptied.
[ https://issues.apache.org/jira/browse/NIFI-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2920: - Affects Version/s: (was: 1.0.0) > Swapped FlowFiles are not removed from content repo when a queue is emptied. > > > Key: NIFI-2920 > URL: https://issues.apache.org/jira/browse/NIFI-2920 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 > Environment: Linux >Reporter: Matthew Clarke >Priority: Critical > > If a queue contains enough FlowFiles to trigger swapping to occur and a user > selects "empty queue" or sets file expiration, only the content claims > associated to FlowFiles that were not swapped get removed from the content > repository. All other content claims are left in the content repository and > are not moved to archive and/or purged. > A restart of NiFi will produce app log messages about these unknown files > left in the content repo and will at that time move them to archive or purge > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2920) Swapped FlowFiles are not removed from content repo when a queue is emptied.
[ https://issues.apache.org/jira/browse/NIFI-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2920: - Affects Version/s: 1.0.0 > Swapped FlowFiles are not removed from content repo when a queue is emptied. > > > Key: NIFI-2920 > URL: https://issues.apache.org/jira/browse/NIFI-2920 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.6.1 > Environment: Linux >Reporter: Matthew Clarke >Priority: Critical > > If a queue contains enough FlowFiles to trigger swapping to occur and a user > selects "empty queue" or sets file expiration, only the content claims > associated to FlowFiles that were not swapped get removed from the content > repository. All other content claims are left in the content repository and > are not moved to archive and/or purged. > A restart of NiFi will produce app log messages about these unknown files > left in the content repo and will at that time move them to archive or purge > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2920) Swapped FlowFiles are not removed from content repo when a queue is emptied.
[ https://issues.apache.org/jira/browse/NIFI-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2920: - Priority: Critical (was: Major) > Swapped FlowFiles are not removed from content repo when a queue is emptied. > > > Key: NIFI-2920 > URL: https://issues.apache.org/jira/browse/NIFI-2920 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.6.1 > Environment: Linux >Reporter: Matthew Clarke >Priority: Critical > > If a queue contains enough FlowFiles to trigger swapping to occur and a user > selects "empty queue" or sets file expiration, only the content claims > associated to FlowFiles that were not swapped get removed from the content > repository. All other content claims are left in the content repository and > are not moved to archive and/or purged. > A restart of NiFi will produce app log messages about these unknown files > left in the content repo and will at that time move them to archive or purge > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2920) Swapped FlowFiles are not removed from content repo when a queue is emptied.
[
https://issues.apache.org/jira/browse/NIFI-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589332#comment-15589332
]
Mark Payne commented on NIFI-2920:
--
I have been able to reproduce this locally. To do so, I created a simple Flow:
GenerateFlowFile (1 KB file size) with success going to 2 different
UpdateAttribute Processors (so that the same Content Claim is held by 2
different FlowFiles). I let about 150,000 FlowFiles queue up (with backpressure
turned off). I then start one of the UpdateAttribute processors. This drained
its queue. I could then look at my content repo for any files not archived:
{code}
content_repository $ find . -type f | grep -v archive | wc -l
192
{code}
After a few minutes, the FlowFile repo is checkpointed, which will result in
things getting cleaned up if they can. The above command shows the same result
(expected, since the FlowFiles are still held. I then empty the queue. After
the FlowFile checkpoints again, I should see nothing in the content repo
outside of archive, but I see:
{code}
content_repository $ find . -type f | grep -v archive | wc -l
167
{code}
I see the same thing happening if I turn on expiration to remove the FlowFiles
instead of clicking Empty Queue.
> Swapped FlowFiles are not removed from content repo when a queue is emptied.
>
>
> Key: NIFI-2920
> URL: https://issues.apache.org/jira/browse/NIFI-2920
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.0.0, 0.6.1
> Environment: Linux
>Reporter: Matthew Clarke
>Priority: Critical
>
> If a queue contains enough FlowFiles to trigger swapping to occur and a user
> selects "empty queue" or sets file expiration, only the content claims
> associated to FlowFiles that were not swapped get removed from the content
> repository. All other content claims are left in the content repository and
> are not moved to archive and/or purged.
> A restart of NiFi will produce app log messages about these unknown files
> left in the content repo and will at that time move them to archive or purge
> them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2854: - Fix Version/s: (was: 1.2.0) 1.1.0 > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3027) ExecuteProcess supports the expression language
[
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-3027:
-
Fix Version/s: 1.1.0
> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}',
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (NIFI-3027) ExecuteProcess supports the expression language
[
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-3027.
--
Resolution: Fixed
> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}',
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (NIFI-1526) Allow components to provide default values for Yield Duration and Run Schedule
[ https://issues.apache.org/jira/browse/NIFI-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-1526. -- Resolution: Fixed > Allow components to provide default values for Yield Duration and Run Schedule > -- > > Key: NIFI-1526 > URL: https://issues.apache.org/jira/browse/NIFI-1526 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Priority: Minor > Fix For: 1.1.0 > > > It would be nice for developers of processors (and maybe reporting tasks and > controller services) to be able to specify a default value for Yield duration > and Run Schedule. > Currently Yield defaults to 1 second and Run Schedule defaults to 0 seconds. > There may be cases where these are not the best default values and the > developer wants to start off with better defaults, still allowing the user to > tune as needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3039) Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits
[
https://issues.apache.org/jira/browse/NIFI-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684567#comment-15684567
]
Mark Payne commented on NIFI-3039:
--
[~jskora] - Thanks for digging into this guy... the code can get pretty intense
here :)
The ramifications of this change do concern me a bit, though. I can see why you
may get smoother performance when you start aging off aggressively at 90%.
Given the code change and the difference that it makes, though, my guess is
that the "thrashing" that you mention is due to large spikes of data, rather
than data that is processed at a very consistent data rate. As a result, this
change would give smoother performance as long as that spike didn't last "too
long." I say this because if you hit 90% and then you see it continue to rise
up to 100% usage, that means that the deletions were not keeping up. As a
result, this change would give the system more time, allowing for 30% of the
repo size to be filled up while the deletions were falling behind instead of
the existing 10%. We are doing this, though, at the expense of giving up a
pretty large amount of storage. Consider, for instance, if I have a 1 TB
provenance repo. This means that once I hit 900 GB usage I'll not stop deleting
until I'm down to only 700 GB used -- I'd be giving up around 300 GB of
provenance data.
I would just be very hesitant to start trading storage for potential
performance consistency when it's not necessary in all cases. If this change is
needed, perhaps we should make these values configurable, defaulting to
something 90% as the high-water mark and 87% as the low-water mark? This would
leave it fairly consistent with how it operates now but provides the
flexibility needed for the different cases.
re: rollover cutoffs, the idea there was that we hope that setting the ageoff
to 90% will result in not exceeding the max. But in a 'last ditch effort' if
cannot keep up, so that we end up significantly exceeding the max configured
value (110% of this value, as you noted), we will block to avoid writing to the
repo at all until ageoff occurs. I'm not opposed to changing that to 100% as
you outline above. However, the PR seems to change it to 90%. I don't think we
should block writing to the repo just because we hit 90% utilization.
Thoughts here?
> Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits
> --
>
> Key: NIFI-3039
> URL: https://issues.apache.org/jira/browse/NIFI-3039
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.0.0, 1.1.0, 0.8.0, 0.7.1
>Reporter: Joe Skora
>Assignee: Joe Skora
>
> Current {purgeOldEvents} logic triggers cleanup when 90% of space is used,
> but it only removes one file if usage is under 100%, causing thrashing around
> 100% usage. In testing, cleanup up to 70% after hitting 90% makes the system
> run more smoothly.
> Also, {rollover} will not trigger cleanup unless 110% of the allowed space is
> in use, changing this to 100% also make a difference in testing.
> Before these changes, a test system that generates huge amounts of provenance
> would become unstable and stop processing provenance until restarted. With
> these changes, the system consistently recovers even under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684606#comment-15684606 ] Mark Payne commented on NIFI-949: - [~bicore] thanks for the feedback. I'm actually curious if what you're seeing is NIFI-2925. Whenever FlowFiles are swapped out, they can fail to ever get removed from the content repo. On restart, it will go through and prune them, but this can take quite a long time. > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3088) NPE occurs on start-up if there are orphaned FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3088: - Assignee: Mark Payne Status: Patch Available (was: Open) > NPE occurs on start-up if there are orphaned FlowFiles > -- > > Key: NIFI-3088 > URL: https://issues.apache.org/jira/browse/NIFI-3088 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.1.0 > > > The repo serialization was re-done in this commit: > https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd > It causes the following exception on start-up if there are orphaned FlowFiles > (FlowFiles that are associated with a queue id that no longer exists in the > Flow). > org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. > at > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at org.eclipse.jetty.server.Server.start(Server.java:411) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at org.eclipse.jetty.server.Server.doStart(Server.java:378) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >
[jira] [Updated] (NIFI-3091) Processors sometimes creating 0-byte FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3091: - Status: Patch Available (was: Open) > Processors sometimes creating 0-byte FlowFiles > -- > > Key: NIFI-3091 > URL: https://issues.apache.org/jira/browse/NIFI-3091 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.1.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.1.0 > > > This bug only applies to the 1.1.0-SNAPSHOT release, which has not yet been > released. It does not appear to affect any released versions of NiFi. > Some Processors (for example ConvertJSONToAvro) improperly create 0-byte > FlowFiles even though the input has data and data is expected to be written > out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3091) Processors sometimes creating 0-byte FlowFiles
Mark Payne created NIFI-3091: Summary: Processors sometimes creating 0-byte FlowFiles Key: NIFI-3091 URL: https://issues.apache.org/jira/browse/NIFI-3091 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.1.0 Reporter: Mark Payne Assignee: Mark Payne Priority: Blocker Fix For: 1.1.0 This bug only applies to the 1.1.0-SNAPSHOT release, which has not yet been released. It does not appear to affect any released versions of NiFi. Some Processors (for example ConvertJSONToAvro) improperly create 0-byte FlowFiles even though the input has data and data is expected to be written out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3079) Regression in session read counting
[ https://issues.apache.org/jira/browse/NIFI-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3079: - Assignee: Mark Payne Status: Patch Available (was: Open) > Regression in session read counting > --- > > Key: NIFI-3079 > URL: https://issues.apache.org/jira/browse/NIFI-3079 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.1.0 >Reporter: Joseph Witt >Assignee: Mark Payne >Priority: Critical > Fix For: 1.1.0 > > Attachments: Screen Shot 2016-11-21 at 11.26.28 PM.png > > > See attached screenshot. The count for bytes read should not be zero on > compress content. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3084) Provenance Repo Build Failures on Windows
[ https://issues.apache.org/jira/browse/NIFI-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3084: - Resolution: Fixed Status: Resolved (was: Patch Available) > Provenance Repo Build Failures on Windows > - > > Key: NIFI-3084 > URL: https://issues.apache.org/jira/browse/NIFI-3084 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Blocker > Fix For: 1.1.0 > > > Build fails with the following errors on the current master when built on > Windows 8. When running the units tests from Intellij I see a subset of the > below errors. Typically it is: > "testIndexAndCompressOnRolloverAndSubsequentEmptySearch", "testBackPressure" > and "testIndexOnRolloverAndSubsequentSearch" > --- > T E S T S > --- > Running org.apache.nifi.provenance.lucene.TestCachingIndexManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.391 sec - > in org.apache.nifi.provenance.lucene.TestCachingIndexManager > Running org.apache.nifi.provenance.lucene.TestSimpleIndexManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec - > in org.apache.nifi.provenance.lucene.TestSimpleIndexManager > Running org.apache.nifi.provenance.TestPersistentProvenanceRepository > Tests run: 30, Failures: 1, Errors: 23, Skipped: 2, Time elapsed: 160.653 sec > <<< FAILURE! - in > org.apache.nifi.provenance.TestPersistentProvenanceRepository > testNotAuthorizedGetSpecificEvent(org.apache.nifi.provenance.TestPersistentProvenanceRepository) > Time elapsed: 5.047 sec <<< ERROR! > java.io.IOException: Unable to delete > C:\nifi\nifi-nar-bundles\nifi-provenance-repository-bundle\nifi-persistent-provenance-repository\target\storage\1ec64b68-d0b3-41a9-8c22-ae272e889ea1\0.prov.gz > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:252) > at org.apache.nifi.util.file.FileUtils.deleteFiles(FileUtils.java:242) > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:248) > at > org.apache.nifi.provenance.TestPersistentProvenanceRepository.closeRepo(TestPersistentProvenanceRepository.java:149) > testModifyIndexWhileSearching(org.apache.nifi.provenance.TestPersistentProvenanceRepository) > Time elapsed: 10.031 sec <<< ERROR! > java.io.IOException: Unable to delete > C:\nifi\nifi-nar-bundles\nifi-provenance-repository-bundle\nifi-persistent-provenance-repository\target\storage\99589698-4b40-443f-9aaf-cac69807c243\0.prov.gz > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:252) > at org.apache.nifi.util.file.FileUtils.deleteFiles(FileUtils.java:242) > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:248) > at > org.apache.nifi.provenance.TestPersistentProvenanceRepository.closeRepo(TestPersistentProvenanceRepository.java:149) > testNotAuthorizedQuery(org.apache.nifi.provenance.TestPersistentProvenanceRepository) > Time elapsed: 5.11 sec <<< ERROR! > java.io.IOException: Unable to delete > C:\nifi\nifi-nar-bundles\nifi-provenance-repository-bundle\nifi-persistent-provenance-repository\target\storage\107b3965-7143-429c-a4e3-8f40dde92250\0.prov.gz > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:252) > at org.apache.nifi.util.file.FileUtils.deleteFiles(FileUtils.java:242) > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:248) > at > org.apache.nifi.provenance.TestPersistentProvenanceRepository.closeRepo(TestPersistentProvenanceRepository.java:149) > testIndexOnRolloverWithImmenseAttribute(org.apache.nifi.provenance.TestPersistentProvenanceRepository) > Time elapsed: 5.141 sec <<< ERROR! > java.io.IOException: Unable to delete > C:\nifi\nifi-nar-bundles\nifi-provenance-repository-bundle\nifi-persistent-provenance-repository\target\storage\5b5b1185-d38a-4284-91f1-cde93bfabebb\0.prov.gz > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:252) > at org.apache.nifi.util.file.FileUtils.deleteFiles(FileUtils.java:242) > at org.apache.nifi.util.file.FileUtils.deleteFile(FileUtils.java:248) > at > org.apache.nifi.provenance.TestPersistentProvenanceRepository.closeRepo(TestPersistentProvenanceRepository.java:149) > testIndexAndCompressOnRolloverAndSubsequentEmptySearch(org.apache.nifi.provenance.TestPersistentProvenanceRepository) > Time elapsed: 8.39 sec <<< ERROR! > java.io.IOException: Unable to delete > C:\nifi\nifi-nar-bundles\nifi-provenance-repository-bundle\nifi-persistent-provenance-repository\target\storage\c8c56a43-c930-4f64-a496-66edcce8e39d\0.prov.gz > at
[jira] [Created] (NIFI-3111) Clarify error message that is generated when @OnScheduled method cannot be invoked
Mark Payne created NIFI-3111: Summary: Clarify error message that is generated when @OnScheduled method cannot be invoked Key: NIFI-3111 URL: https://issues.apache.org/jira/browse/NIFI-3111 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Priority: Minor Whenever a Processor throws an Exception from a method annotated with @OnScheduled, we get an error message and bulletin such as: MyProcessor[id=ac88775c-0158-1000-4de6-52946f9264b5] MyProcessor[id=ac88775c-0158-1000-4de6-52946f9264b5] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. This is repetition (contains Processor's toString() twice) and also contains the RuntimeException's toString() twice without actually indicating what the problem is. More details are logged in nifi-app.log but the bulletin should be more specific. I have not verified yet, but a similar problem likely exists with @OnStopped, @OnUnscheduled, @OnScheduled, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2363) CLUSTER scope state should be viewed/cleared from primary node
[ https://issues.apache.org/jira/browse/NIFI-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2363: - Fix Version/s: 1.2.0 > CLUSTER scope state should be viewed/cleared from primary node > -- > > Key: NIFI-2363 > URL: https://issues.apache.org/jira/browse/NIFI-2363 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Assignee: Koji Kawamura > Fix For: 1.2.0 > > > Currently, every node in a NiFi cluster retrieves or clears CLUSTER scope > state when user view/clear component state from NiFi UI. > Since the cluster-wide state is shared the same value among nodes, those > operation should be executed only from a primary node. > See this discussion for detail: > https://github.com/apache/nifi/pull/563#issuecomment-231393086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2975) NiFi attempts to expire FlowFiles before all have been restored
[ https://issues.apache.org/jira/browse/NIFI-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-2975: - Status: Patch Available (was: Open) > NiFi attempts to expire FlowFiles before all have been restored > --- > > Key: NIFI-2975 > URL: https://issues.apache.org/jira/browse/NIFI-2975 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > Occasionally, I have seen the following log message in logs/nifi-app.log: > ERROR [Framework Task Thread Thread-1] o.a.n.controller.tasks.ExpireFlowFiles > Failed to expire FlowFiles due to java.lang.IllegalStateException: Cannot > update repository until record recovery has been performed > java.lang.IllegalStateException: Cannot update repository until record > recovery has been performed > at > org.wali.MinimalLockingWriteAheadLog.update(MinimalLockingWriteAheadLog.java:199) > ~[nifi-write-ahead-log-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:219) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:187) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:357) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:299) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.tasks.ExpireFlowFiles.expireFlowFiles(ExpireFlowFiles.java:85) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.tasks.ExpireFlowFiles.expireFlowFiles(ExpireFlowFiles.java:90) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.tasks.ExpireFlowFiles.expireFlowFiles(ExpireFlowFiles.java:116) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.tasks.ExpireFlowFiles.run(ExpireFlowFiles.java:56) > ~[nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > org.apache.nifi.controller.scheduling.StandardProcessScheduler$1.run(StandardProcessScheduler.java:115) > [nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] > This appears to be due to the fact that we are attempting to expire FlowFiles > before we have finished recovering all of the FlowFiles from the > FlowFileRepository. This is not really harmful, as the expiration will just > keep trying until it is successful. However, it does result in some alarming > ERROR-level messages in the logs and needs to be resolved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
