[jira] [Updated] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues
[ https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3509: --- Fix Version/s: 1.2.0 Status: Patch Available (was: Open) > Creating templates of copied and pasted PGs can cause issues > - > > Key: NIFI-3509 > URL: https://issues.apache.org/jira/browse/NIFI-3509 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Joseph Percivall >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.2.0 > > Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot > 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png, > templatesForNIFI-3509.zip > > > Using the following steps I was able to create a template which instantiates > with connections from one input port to another input port of another PG. See > the template here[1]. > See the first attached screenshot for the seed flow. After importing, it > originally gives an error "Cannot add Connection to Process Group because its > destination does not belong to this Process Group" and nothing appears to be > added but after I refresh the canvas parts of the original flow appear > (screenshot 2 and 3). > What is most odd, is that there are connections going from the input port > "input" of pg "test" to the input ports of other PGs. I didn't think that was > possible. > The steps I used to create it: > 1: create a PG "test" with input port "input", UpdateAttribute and output > port "output". Creating connections between them. > 2: Go to Parent group and copy "test". > 3: Go back into "test" and paste. > 4: Copy and paste "test" multiple times, creating the flow. > 5: Create a template of the flow > 6: When using the template, see the issue described. > The root cause of the issues, I believe, is that the id for the parent group > starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into > itself it created the "test" pg with an id starting with the same value. > [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues
[ https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954150#comment-15954150 ] Oleg Zhurakousky commented on NIFI-3509: Actually I believe [~mcgilman] is correct and the issue rooted in copy/paste via template and it does appear that the logic there is different as it results with two components with the same MSB (which should never happen). Working on isolating the root cause. At least it is now fully reproducible. > Creating templates of copied and pasted PGs can cause issues > - > > Key: NIFI-3509 > URL: https://issues.apache.org/jira/browse/NIFI-3509 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Joseph Percivall >Assignee: Oleg Zhurakousky >Priority: Critical > Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot > 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png, > templatesForNIFI-3509.zip > > > Using the following steps I was able to create a template which instantiates > with connections from one input port to another input port of another PG. See > the template here[1]. > See the first attached screenshot for the seed flow. After importing, it > originally gives an error "Cannot add Connection to Process Group because its > destination does not belong to this Process Group" and nothing appears to be > added but after I refresh the canvas parts of the original flow appear > (screenshot 2 and 3). > What is most odd, is that there are connections going from the input port > "input" of pg "test" to the input ports of other PGs. I didn't think that was > possible. > The steps I used to create it: > 1: create a PG "test" with input port "input", UpdateAttribute and output > port "output". Creating connections between them. > 2: Go to Parent group and copy "test". > 3: Go back into "test" and paste. > 4: Copy and paste "test" multiple times, creating the flow. > 5: Create a template of the flow > 6: When using the template, see the issue described. > The root cause of the issues, I believe, is that the id for the parent group > starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into > itself it created the "test" pg with an id starting with the same value. > [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (NIFI-3612) Add support for Parquet to Nifi-Registry-Bundle
[ https://issues.apache.org/jira/browse/NIFI-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-3612: -- Assignee: Oleg Zhurakousky > Add support for Parquet to Nifi-Registry-Bundle > --- > > Key: NIFI-3612 > URL: https://issues.apache.org/jira/browse/NIFI-3612 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Chaffelson >Assignee: Oleg Zhurakousky > > This bundle could potentially be extended to include a Parquet transform by > leveraging the Apache 2.0 licensed parquet-mr/avro libraries: > https://github.com/apache/parquet-mr/tree/master/parquet-avro > This would provide coverage of this popular format to complement the ORC > support in the Hive Bundle and the other schema-dependent formats already in > this bundle. > Existing NiFi Parquet support in the kite bundle can only write to a > non-kerberised Kite Dataset, which prevents usage on secured environments or > writing to a FlowFile. > As the main competitor to ORC, providing more generic Parquet Transform > support will greatly widen the pool of potential NiFi adopters, particularly > in the Spark community. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3523) WriteAheadProvenanceRepository fails to initialize properly if it contains an invalid Provenance Event
[ https://issues.apache.org/jira/browse/NIFI-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3523: --- Resolution: Fixed Status: Resolved (was: Patch Available) > WriteAheadProvenanceRepository fails to initialize properly if it contains an > invalid Provenance Event > -- > > Key: NIFI-3523 > URL: https://issues.apache.org/jira/browse/NIFI-3523 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > With the new WriteAheadProvenanceRepository, when trying to initialize > itself, the repository will throw an Exception preventing NiFi from starting > if it encounters an invalid Provenance Event. While it can properly handle > IOExceptions that are thrown, other Exceptions (such as those thrown by > NIFI-3522) will still cause problems. While such events should not be > generated, the Provenance Repository should be resilient enough to handle > them and continue on. > 2017-02-22 15:47:11,896 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. > at > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at org.eclipse.jetty.server.Server.start(Server.java:411) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at >
[jira] [Updated] (NIFI-3523) WriteAheadProvenanceRepository fails to initialize properly if it contains an invalid Provenance Event
[ https://issues.apache.org/jira/browse/NIFI-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3523: --- Fix Version/s: 1.2.0 > WriteAheadProvenanceRepository fails to initialize properly if it contains an > invalid Provenance Event > -- > > Key: NIFI-3523 > URL: https://issues.apache.org/jira/browse/NIFI-3523 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > With the new WriteAheadProvenanceRepository, when trying to initialize > itself, the repository will throw an Exception preventing NiFi from starting > if it encounters an invalid Provenance Event. While it can properly handle > IOExceptions that are thrown, other Exceptions (such as those thrown by > NIFI-3522) will still cause problems. While such events should not be > generated, the Provenance Repository should be resilient enough to handle > them and continue on. > 2017-02-22 15:47:11,896 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. > at > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) > ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) > ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at org.eclipse.jetty.server.Server.start(Server.java:411) > ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) > ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] > at >
[jira] [Commented] (NIFI-3543) Add support for using EL when defining 'brokerURL' in jms-cf-service
[ https://issues.apache.org/jira/browse/NIFI-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894358#comment-15894358 ] Oleg Zhurakousky commented on NIFI-3543: [~Shelly_LC] I may have missed that but if documentation says it does support EL, then it is indeed a bug. Thank you for pointing this out. > Add support for using EL when defining 'brokerURL' in jms-cf-service > > > Key: NIFI-3543 > URL: https://issues.apache.org/jira/browse/NIFI-3543 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.0, 1.1.1 >Reporter: Shelly Liu >Priority: Minor > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3543) EL is not working for jms-cf-service Broker URL
[ https://issues.apache.org/jira/browse/NIFI-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892385#comment-15892385 ] Oleg Zhurakousky commented on NIFI-3543: Also, would you mind adding a "Description" and modify the Title of this JIRA to something like ""Add support for using EL when defining 'brokerUri" in jms-cf-service". > EL is not working for jms-cf-service Broker URL > --- > > Key: NIFI-3543 > URL: https://issues.apache.org/jira/browse/NIFI-3543 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.0, 1.1.1 >Reporter: Shelly Liu >Priority: Minor > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3543) EL is not working for jms-cf-service Broker URL
[ https://issues.apache.org/jira/browse/NIFI-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892366#comment-15892366 ] Oleg Zhurakousky commented on NIFI-3543: [~Shelly_LC] Thank you so much for your contribution and it is indeed a valuable one, however it is not a BUG. Basically the bug is something that one claims should work but it doesn't. In this case it was more of a missing feature. One may argue IF we should support EL for Broker-URL, but it also means that in its original implementation the developer decided not to and therefore never claimed that EL Should work. Anyway, would you consider re-classifying this JIRA as "Improvement" rather then a "Bug". This will help the greater community and users to separate the two. Meanwhile I'll review and if all is good will merge the PR. Once agin, thank you for contributing. > EL is not working for jms-cf-service Broker URL > --- > > Key: NIFI-3543 > URL: https://issues.apache.org/jira/browse/NIFI-3543 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.0, 1.1.1 >Reporter: Shelly Liu >Priority: Minor > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues
[ https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890837#comment-15890837 ] Oleg Zhurakousky commented on NIFI-3509: [~JPercivall] Yes, when I import your template I do have the issue you describe, but what I can't reproduce is creation of such template which is where the root of the problem. > Creating templates of copied and pasted PGs can cause issues > - > > Key: NIFI-3509 > URL: https://issues.apache.org/jira/browse/NIFI-3509 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Joseph Percivall >Assignee: Oleg Zhurakousky >Priority: Critical > Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot > 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png > > > Using the following steps I was able to create a template which instantiates > with connections from one input port to another input port of another PG. See > the template here[1]. > See the first attached screenshot for the seed flow. After importing, it > originally gives an error "Cannot add Connection to Process Group because its > destination does not belong to this Process Group" and nothing appears to be > added but after I refresh the canvas parts of the original flow appear > (screenshot 2 and 3). > What is most odd, is that there are connections going from the input port > "input" of pg "test" to the input ports of other PGs. I didn't think that was > possible. > The steps I used to create it: > 1: create a PG "test" with input port "input", UpdateAttribute and output > port "output". Creating connections between them. > 2: Go to Parent group and copy "test". > 3: Go back into "test" and paste. > 4: Copy and paste "test" multiple times, creating the flow. > 5: Create a template of the flow > 6: When using the template, see the issue described. > The root cause of the issues, I believe, is that the id for the parent group > starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into > itself it created the "test" pg with an id starting with the same value. > [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues
[ https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890619#comment-15890619 ] Oleg Zhurakousky commented on NIFI-3509: [~JPercivall] I have tried multiple times to reproduce the issue with no success. I have discovered another issue, which may not be an issue at all (still investigating) since it gives a friendly error message, but is completely unrelated to the above. Could you please verify the steps? I must be missing something. > Creating templates of copied and pasted PGs can cause issues > - > > Key: NIFI-3509 > URL: https://issues.apache.org/jira/browse/NIFI-3509 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Joseph Percivall >Assignee: Oleg Zhurakousky >Priority: Critical > Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot > 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png > > > Using the following steps I was able to create a template which instantiates > with connections from one input port to another input port of another PG. See > the template here[1]. > See the first attached screenshot for the seed flow. After importing, it > originally gives an error "Cannot add Connection to Process Group because its > destination does not belong to this Process Group" and nothing appears to be > added but after I refresh the canvas parts of the original flow appear > (screenshot 2 and 3). > What is most odd, is that there are connections going from the input port > "input" of pg "test" to the input ports of other PGs. I didn't think that was > possible. > The steps I used to create it: > 1: create a PG "test" with input port "input", UpdateAttribute and output > port "output". Creating connections between them. > 2: Go to Parent group and copy "test". > 3: Go back into "test" and paste. > 4: Copy and paste "test" multiple times, creating the flow. > 5: Create a template of the flow > 6: When using the template, see the issue described. > The root cause of the issues, I believe, is that the id for the parent group > starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into > itself it created the "test" pg with an id starting with the same value. > [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues
[ https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-3509: -- Assignee: Oleg Zhurakousky > Creating templates of copied and pasted PGs can cause issues > - > > Key: NIFI-3509 > URL: https://issues.apache.org/jira/browse/NIFI-3509 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Joseph Percivall >Assignee: Oleg Zhurakousky >Priority: Critical > Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot > 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png > > > Using the following steps I was able to create a template which instantiates > with connections from one input port to another input port of another PG. See > the template here[1]. > See the first attached screenshot for the seed flow. After importing, it > originally gives an error "Cannot add Connection to Process Group because its > destination does not belong to this Process Group" and nothing appears to be > added but after I refresh the canvas parts of the original flow appear > (screenshot 2 and 3). > What is most odd, is that there are connections going from the input port > "input" of pg "test" to the input ports of other PGs. I didn't think that was > possible. > The steps I used to create it: > 1: create a PG "test" with input port "input", UpdateAttribute and output > port "output". Creating connections between them. > 2: Go to Parent group and copy "test". > 3: Go back into "test" and paste. > 4: Copy and paste "test" multiple times, creating the flow. > 5: Create a template of the flow > 6: When using the template, see the issue described. > The root cause of the issues, I believe, is that the id for the parent group > starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into > itself it created the "test" pg with an id starting with the same value. > [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3522) If a Processor clones a FlowFile then adds attributes to or modifies the clone, a Provenance Event is generated with no FlowFile UUID.
[ https://issues.apache.org/jira/browse/NIFI-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3522: --- Resolution: Fixed Status: Resolved (was: Patch Available) > If a Processor clones a FlowFile then adds attributes to or modifies the > clone, a Provenance Event is generated with no FlowFile UUID. > -- > > Key: NIFI-3522 > URL: https://issues.apache.org/jira/browse/NIFI-3522 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > This can be recreated by adding a SplitText Processor to the canvas and then > feeding it in a FlowFile that does not need to be split. In this case, a > Provenance CLONE event is generated, and so is an ATTRIBUTES_MODIFIED event > because after cloning the incoming FlowFile, we add attributes to the clone. > If we then attempt to query provenance for events generated by this Processor > - or use the Provenance Reporting Task - we will get an error indicating that > the Provenance Event could not be generated because it has no FlowFile UUID. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3356) Provide a newly refactored provenance repository
[ https://issues.apache.org/jira/browse/NIFI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3356: --- Resolution: Fixed Status: Resolved (was: Patch Available) Mark, obviously this one is too large of a change to give it a thorough review it deserves, but it is a much needed change nevertheless and we did have a good discussion in the PR. So it is merged/resolved. Whatever we may have missed could be addressed as a separate JIRA in the future. > Provide a newly refactored provenance repository > > > Key: NIFI-3356 > URL: https://issues.apache.org/jira/browse/NIFI-3356 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0 > > > The Persistent Provenance Repository has been redesigned a few different > times over several years. The original design for the repository was to > provide storage of events and sequential iteration over those events via a > Reporting Task. After that, we added the ability to compress the data so that > it could be held longer. We then introduced the notion of indexing and > searching via Lucene. We've since made several more modifications to try to > boost performance. > At this point, however, the repository is still the bottleneck for many flows > that handle large volumes of small FlowFiles. We need a new implementation > that is based around the current goals for the repository and that can > provide better throughput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3255:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.2.0
>
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (NIFI-3354) Create CSV To Avro transformer
[ https://issues.apache.org/jira/browse/NIFI-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3354: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Create CSV To Avro transformer > -- > > Key: NIFI-3354 > URL: https://issues.apache.org/jira/browse/NIFI-3354 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > Attachments: csv_content.txt, NIFI-3354.xml, > Test_Transform_CSV_to_JSON_to_CSV.xml > > > While we currently have CSV to AVRO transformer it required HDFS/Kite > dependencies which could be easily eliminated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3500) Add support for multi-char delimiter to the new CSV transformers
Oleg Zhurakousky created NIFI-3500: -- Summary: Add support for multi-char delimiter to the new CSV transformers Key: NIFI-3500 URL: https://issues.apache.org/jira/browse/NIFI-3500 Project: Apache NiFi Issue Type: Improvement Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky This is related to the new work done under NIFI-3354 (transformers with external schema registry). More details are here - https://github.com/apache/nifi/pull/1436 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3499) Add support for handling line delimited input by new CSV Transformers
Oleg Zhurakousky created NIFI-3499: -- Summary: Add support for handling line delimited input by new CSV Transformers Key: NIFI-3499 URL: https://issues.apache.org/jira/browse/NIFI-3499 Project: Apache NiFi Issue Type: Bug Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky This is related to the new work done under NIFI-3354 (transformers with external schema registry). More details are here - https://github.com/apache/nifi/pull/1436 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3495) TextLineDemarcator sets the wrong index when read ahead is performed in isEol operation
[ https://issues.apache.org/jira/browse/NIFI-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3495: --- Description: This condition is very rare. It only occurs when read ahead (call to _fill()_) is made inside of the _isEol_ operation which essentially sets the new index which then is reset inside of the main _nextOffsetInfo_ operation. So the fix is to basically monitor if _isEol_ had to perform read ahead and if it did do not reset the index. More details. While this component is modeled after standard Java BufferedReader which simply reads and returns lines (delimited by CR or LF or both), this reader also holds the information about how each line terminated (i.e., EOF, or CR or LF or CR and LF) returning it to the caller as OffsetInfo. So for example if you have a record "foo\r\nbar" and you read it with BuffereReader you will get 'foo' and 'bar'. However you will not know that between the two tokens there was CR and LF and therefore will not be able to restore (if need to) the record to its original state. The TextLineDemarcator will return OffsetInfo which holds the delimiter and other information. So, to accomplish the above every time we see CR (13) we need to peek at the next byte and see if its LF(10). When at the end of the buffer such peek becomes complicated since we need to read more data and so we did, but didn't handle index properly essentially setting it back to the old value when the new one was set inside of the fill(). was: This condition is very rare. It only occurs when read ahead (call to _fill()_) is made inside of the _isEol_ operation which essentially sets the new index which then is reset inside of the main _nextOffsetInfo_ operation. So the fix is to basically monitor if _isEol_ had to perform read ahead and if it did do not reset the index. > TextLineDemarcator sets the wrong index when read ahead is performed in isEol > operation > --- > > Key: NIFI-3495 > URL: https://issues.apache.org/jira/browse/NIFI-3495 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.2.0 > > > This condition is very rare. It only occurs when read ahead (call to > _fill()_) is made inside of the _isEol_ operation which essentially sets the > new index which then is reset inside of the main _nextOffsetInfo_ operation. > So the fix is to basically monitor if _isEol_ had to perform read ahead and > if it did do not reset the index. > More details. > While this component is modeled after standard Java BufferedReader which > simply reads and returns lines (delimited by CR or LF or both), this reader > also holds the information about how each line terminated (i.e., EOF, or CR > or LF or CR and LF) returning it to the caller as OffsetInfo. > So for example if you have a record "foo\r\nbar" and you read it with > BuffereReader you will get 'foo' and 'bar'. However you will not know that > between the two tokens there was CR and LF and therefore will not be able to > restore (if need to) the record to its original state. The TextLineDemarcator > will return OffsetInfo which holds the delimiter and other information. > So, to accomplish the above every time we see CR (13) we need to peek at the > next byte and see if its LF(10). When at the end of the buffer such peek > becomes complicated since we need to read more data and so we did, but didn't > handle index properly essentially setting it back to the old value when the > new one was set inside of the fill(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3495) TextLineDemarcator sets the wrong index when read ahead is performed in isEol operation
[ https://issues.apache.org/jira/browse/NIFI-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3495: --- Description: This condition is very rare. It only occurs when read ahead (call to _fill()_) is made inside of the _isEol_ operation which essentially sets the new index which then is reset inside of the main _nextOffsetInfo_ operation. So the fix is to basically monitor if _isEol_ had to perform read ahead and if it did do not reset the index. was: This condition is very rare. It only occurs when read ahead (call to _fill()_) is made inside of the _isEol_ operation which essentially sets the new index which then is reset inside of the main _nextOffsetInfo_ operation. So the fox is to basically monitor if _isEol_ had tio perform read ahead and if it did do not reset the index. > TextLineDemarcator sets the wrong index when read ahead is performed in isEol > operation > --- > > Key: NIFI-3495 > URL: https://issues.apache.org/jira/browse/NIFI-3495 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.2.0 > > > This condition is very rare. It only occurs when read ahead (call to > _fill()_) is made inside of the _isEol_ operation which essentially sets the > new index which then is reset inside of the main _nextOffsetInfo_ operation. > So the fix is to basically monitor if _isEol_ had to perform read ahead and > if it did do not reset the index. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3495) TextLineDemarcator sets the wrong index when read ahead is performed in isEol operation
Oleg Zhurakousky created NIFI-3495: -- Summary: TextLineDemarcator sets the wrong index when read ahead is performed in isEol operation Key: NIFI-3495 URL: https://issues.apache.org/jira/browse/NIFI-3495 Project: Apache NiFi Issue Type: Bug Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky Priority: Critical Fix For: 1.2.0 This condition is very rare. It only occurs when read ahead (call to _fill()_) is made inside of the _isEol_ operation which essentially sets the new index which then is reset inside of the main _nextOffsetInfo_ operation. So the fox is to basically monitor if _isEol_ had tio perform read ahead and if it did do not reset the index. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3354) Create CSV To Avro transformer
[
https://issues.apache.org/jira/browse/NIFI-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3354:
---
Attachment: csv_content.txt
NIFI-3354.xml
Attaching template (NIFI-3354.xml) and sample content (csv_content.txt) for
testing.
Drop the file (csv_content.txt) into GetFile's directory and watch it come out
the other side in PutFile. The source and output should look identical.
Additionally in the logs you should see the output from the LogAttribute. Keep
an eye on the shown below (they must be there)
{code}
Standard FlowFile Attributes
. . .
FlowFile Attribute Map Content
Key: 'DRIVER_NAME'
Value: 'John Doe'
Key: 'EVENT_TYPE'
Value: 'Normal'
. . .
Key: 'schema.text'
Value:
'{"type":"record","name":"truckgeoevent","namespace":"hortonworks.hdp.refapp.trucking","fields":[{"name":"eventTime","type":"string"},{"name":"eventSource","type":"string"},{"name":"truckId","type":"int"},{"name":"driverId","type":"int"},{"name":"driverName","type":"string"},{"name":"routeId","type":"int"},{"name":"route","type":"string"},{"name":"eventType","type":"string"},{"name":"longitude","type":"double"},{"name":"latitude","type":"double"},{"name":"correlationId","type":"long"}]}'
. . .
{code}
> Create CSV To Avro transformer
> --
>
> Key: NIFI-3354
> URL: https://issues.apache.org/jira/browse/NIFI-3354
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
> Fix For: 1.2.0
>
> Attachments: csv_content.txt, NIFI-3354.xml
>
>
> While we currently have CSV to AVRO transformer it required HDFS/Kite
> dependencies which could be easily eliminated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
[ https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2876: --- Fix Version/s: 1.2.0 > Refactor TextLineDemarcator and StreamDemarcator into a common abstract class > - > > Key: NIFI-2876 > URL: https://issues.apache.org/jira/browse/NIFI-2876 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 1.2.0 > > > Based on the work that has been performed as part of the NIFI-2851 we now > have a new class with a significantly faster logic to perform demarcation of > the InputStream (TextLineDemarcator). This new class's initial starting point > was the existing LineDemarcator. They both now share ~60-70% of common code > which would be important to extract into a common abstract class as well as > incorporate the new (faster) demarcation logic int StreamDemarcator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name
Title: Message Title Oleg Zhurakousky commented on NIFI-3193 Re: Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name Brian thank you, I can work with that. Will let you know. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3290) Reporting task to send bulletins over S2S
Title: Message Title Oleg Zhurakousky updated NIFI-3290 Apache NiFi / NIFI-3290 Reporting task to send bulletins over S2S Change By: Oleg Zhurakousky Resolution: Fixed Status: Patch Available Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3290) Reporting task to send bulletins over S2S
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3290 Reporting task to send bulletins over S2S Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-957) Ability to provide default schedules per type
Title: Message Title Oleg Zhurakousky updated NIFI-957 Apache NiFi / NIFI-957 Ability to provide default schedules per type Change By: Oleg Zhurakousky Resolution: Fixed Status: Patch Available Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-957) Ability to provide default schedules per type
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-957 Ability to provide default schedules per type Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3161) nifi-mock pulls two different versions of ASM libraries
Title: Message Title Oleg Zhurakousky updated NIFI-3161 Apache NiFi / NIFI-3161 nifi-mock pulls two different versions of ASM libraries Change By: Oleg Zhurakousky Status: Open Patch Available Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3161) nifi-mock pulls two different versions of ASM libraries
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3161 nifi-mock pulls two different versions of ASM libraries Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-1760) Validation messages should use PropertyDescriptor's display name (if provided) when building validation message
Title: Message Title Oleg Zhurakousky resolved as Duplicate Apache NiFi / NIFI-1760 Validation messages should use PropertyDescriptor's display name (if provided) when building validation message Change By: Oleg Zhurakousky Resolution: Duplicate Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3179) MergeContent extracts demarcator property value bytes without specifying charset encoding
Title: Message Title Oleg Zhurakousky updated NIFI-3179 Apache NiFi / NIFI-3179 MergeContent extracts demarcator property value bytes without specifying charset encoding Change By: Oleg Zhurakousky Status: Open Patch Available Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3179) MergeContent extracts demarcator property value bytes without specifying charset encoding
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3179 MergeContent extracts demarcator property value bytes without specifying charset encoding Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name
Title: Message Title Oleg Zhurakousky commented on NIFI-3193 Re: Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name Brian do you by any chance have some sample code or link you can point to that provides a Java example of such authentication? Just want to make sure I understand requirements correctly while I am going thru https://www.rabbitmq.com/ssl.html Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3146) Add support for various transformer processors that transform content based on external schema
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3146 Add support for various transformer processors that transform content based on external schema Change By: Oleg Zhurakousky Currently NiFi provides pairs of processors to transform from format A to format B (i.e., Avro to JSON and JSON to Avro etc.) which rely on schema being defined as part of such processor configuration which essentially couples processor instance to a specific schema . We also have CSV Given dynamic nature of schemas and requirements to AVRO maintain schemas in external stores (i.e. , but we don't have AVRO schema registries) a more sophisticated model would be to CSV to complete the pair define a common strategy for interacting with external schema stores and set of processors that integrate with such store . Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3146) Add support for various transformer processors that transform content based on external schema
Title: Message Title Oleg Zhurakousky updated NIFI-3146 Apache NiFi / NIFI-3146 Add support for various transformer processors that transform content based on external schema Change By: Oleg Zhurakousky Status: Open Patch Available Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3146) Add support for various transformer processors that transform content based on external schema
Title: Message Title Oleg Zhurakousky commented on NIFI-3146 Re: Add support for various transformer processors that transform content based on external schema As part of the PR attached in NIFI-3354 the following features have been added Controller Service Strategy for interacting with external schema stores (registries) Simple key/value implementation of the above strategy which allows schemas to be defined as dynamic properties of the service 8 new processors that use the above strategy to request schemas ExtractAvroFieldsViaSchemaRegistry TransformAvroToCSVViaSchemaRegistry TransformAvroToJsonViaSchemaRegistry TransformCSVToAvroViaSchemaRegistry TransformCSVToJsonViaSchemaRegistry TransformJsonToAvroViaSchemaRegistry TransformJsonToCSVViaSchemaRegistry UpdateAttributeWithSchemaViaSchemaRegistry Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3146) Add support for various transformer processors that transform content based on external schema
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3146 Add support for various transformer processors that transform content based on external schema Change By: Oleg Zhurakousky Summary: Create AVRO to CSV Add support for various transformer processors that transform content based on external schema Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-1722) Intermittent deadlocks in Kafka API when when stopping GetKafka
Title: Message Title Oleg Zhurakousky resolved as Won't Fix The issue is being addressed in Kafka Apache NiFi / NIFI-1722 Intermittent deadlocks in Kafka API when when stopping GetKafka Change By: Oleg Zhurakousky Resolution: Won't Fix Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3180) unable to import a flow template to NIFI 1.2 SNAPSHOT with "null" error
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3180 unable to import a flow template to NIFI 1.2 SNAPSHOT with "null" error Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3180) unable to import a flow template to NIFI 1.2 SNAPSHOT with "null" error
Title: Message Title Oleg Zhurakousky updated NIFI-3180 Apache NiFi / NIFI-3180 unable to import a flow template to NIFI 1.2 SNAPSHOT with "null" error Change By: Oleg Zhurakousky Status: Open Patch Available Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3255 SplitText fails with IllegalArgumentException: Destination cannot be within sources Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (NIFI-3354) Create CSV To Avro transformer
Title: Message Title Oleg Zhurakousky updated an issue Apache NiFi / NIFI-3354 Create CSV To Avro transformer Change By: Oleg Zhurakousky Fix Version/s: 1.2.0 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Assigned] (NIFI-3412) GetTCP testSuccessInteraction failing for multi-threaded builds and bare .m2 repo
[
https://issues.apache.org/jira/browse/NIFI-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky reassigned NIFI-3412:
--
Assignee: Oleg Zhurakousky
> GetTCP testSuccessInteraction failing for multi-threaded builds and bare .m2
> repo
> -
>
> Key: NIFI-3412
> URL: https://issues.apache.org/jira/browse/NIFI-3412
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Joey Frazee
>Assignee: Oleg Zhurakousky
>
> GetTCP tests are failing for multi-threaded builds (e.g., mvn -T 2.0C clean
> install ...) when .m2 is completely empty and the build is being run for the
> first time.
> {code}
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
> Maven home: /usr/local/maven
> Java version: 1.8.0_121, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.29.amzn1.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.41-36.55.amzn1.x86_64", arch: "amd64",
> family: "unix"
> {code}
> {code}
> ---
> Test set: org.apache.nifi.processors.gettcp.TestGetTCP
> ---
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.775 sec <<<
> FAILURE! - in org.apache.nifi.processors.gettcp.TestGetTCP
> testSuccessInteraction(org.apache.nifi.processors.gettcp.TestGetTCP) Time
> elapsed: 0.712 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.assertTransferCount(StandardProcessorTestRunner.java:319)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.assertAllFlowFilesTransferred(StandardProcessorTestRunner.java:314)
> at
> org.apache.nifi.processors.gettcp.TestGetTCP.testSuccessInteraction(TestGetTCP.java:82)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3223) Allow PublishAMQP to use NiFi expression language
[ https://issues.apache.org/jira/browse/NIFI-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3223: --- Fix Version/s: 1.2.0 > Allow PublishAMQP to use NiFi expression language > - > > Key: NIFI-3223 > URL: https://issues.apache.org/jira/browse/NIFI-3223 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 0.7.0, 1.1.0, 0.7.1 >Reporter: Brian >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > Enable the use of NiFi expression language for the PublishAMQP processors, > Routing Key value to allow it to be better used within the NiFi workflows. > PublishAMQP fields to enable: > "Routing Key" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3223) Allow PublishAMQP to use NiFi expression language
[ https://issues.apache.org/jira/browse/NIFI-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3223: --- Status: Patch Available (was: Open) > Allow PublishAMQP to use NiFi expression language > - > > Key: NIFI-3223 > URL: https://issues.apache.org/jira/browse/NIFI-3223 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 0.7.1, 1.1.0, 0.7.0 >Reporter: Brian >Assignee: Oleg Zhurakousky > > Enable the use of NiFi expression language for the PublishAMQP processors, > Routing Key value to allow it to be better used within the NiFi workflows. > PublishAMQP fields to enable: > "Routing Key" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3363) PutKafka throws NullPointerException when User-Defined partition strategy is used
[
https://issues.apache.org/jira/browse/NIFI-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3363:
---
Fix Version/s: 1.2.0
0.8.0
> PutKafka throws NullPointerException when User-Defined partition strategy is
> used
> -
>
> Key: NIFI-3363
> URL: https://issues.apache.org/jira/browse/NIFI-3363
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.4.0, 0.5.0, 0.6.0, 0.4.1, 0.5.1, 0.7.0, 0.6.1,
> 1.1.0, 0.7.1, 1.1.1, 1.0.1
> Environment: By looking at the release tags contained in a commit
> that added User-Defined partition strategy (NIFI-1097 22de23b), it seems all
> NiFi versions since 0.4.0 is affected.
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 0.8.0, 1.2.0
>
>
> NullPointerException is thrown because PutKafka tries to put null into
> properties since following if statements doesn't cover
> USER_DEFINED_PARTITIONING.
> {code: title=PutKafka.java buildKafkaConfigProperties}
> String partitionStrategy = context.getProperty(PARTITION_STRATEGY).getValue();
> String partitionerClass = null;
> if (partitionStrategy.equalsIgnoreCase(ROUND_ROBIN_PARTITIONING.getValue())) {
> partitionerClass = Partitioners.RoundRobinPartitioner.class.getName();
> } else if
> (partitionStrategy.equalsIgnoreCase(RANDOM_PARTITIONING.getValue())) {
> partitionerClass = Partitioners.RandomPartitioner.class.getName();
> }
> properties.setProperty("partitioner.class", partitionerClass); // Happens here
> {code}
> A naive fix for this would be adding one more if statement so that it puts
> 'partitioner.class' property only if partitionerClass is set.
> However, while I was testing the fix, I found following facts, that revealed
> this approach wouldn't be the right solution for this issue.
> In short, we don't have to set 'partitioner.class' property with Kafka 0.8.x
> client in the first place. I assume it's there because PutKafka came through
> a long history..
> h2. PutKafka history analysis
> - PutKafka used to cover Kafka 0.8 and 0.9
> - Around the time Kafka 0.9 was released, PutKafka added 'partitioner.class'
> via NIFI-1097: start using new API. There were two client libraries
> kafka-clients and kafka_2.9.1, both 0.8.2.2.
> - Then PublishKafka is added for Kafka 0.9, at this point, we could add
> 'partition' property to PublishKafka, but we didn't do that for some reason.
> PublishKafka doesn't support user defined partition as of this writing.
> NIFI-1296.
> - The code adding 'partitioner.class' has been left in PutKafka.
> - Further, we separated nar into 0.8, 0.9 and 0.10.
> - Now only PutKafka(0.8) uses 'partitioner.class' property, but 0.8 client
> doesn't use that property. So we don't need that code at all.
> h2. Then, how should we fix this?
> Since PutKafka in both master and 0.x branch specifically uses Kafka 0.8.x
> client. We can simply remove the codes adding 'partitioner.class', probably
> PARTITION_STRATEGY processor property, too.
> h2. Expected result after fix
> - Users can specify Kafka partition with PutKafka 'partition' property, but
> no need to specify 'partition strategy', NullPointerException won't be thrown
> - A warning log, that used to be logged in nifi-app.log won't be logged any
> more: {code}2017-01-18 13:53:33,071 WARN [Timer-Driven Process Thread-9]
> o.a.k.clients.producer.ProducerConfig The configuration partitioner.class =
> null was supplied but isn't a known config.{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3363) PutKafka throws NullPointerException when User-Defined partition strategy is used
[
https://issues.apache.org/jira/browse/NIFI-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3363:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> PutKafka throws NullPointerException when User-Defined partition strategy is
> used
> -
>
> Key: NIFI-3363
> URL: https://issues.apache.org/jira/browse/NIFI-3363
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.4.0, 0.5.0, 0.6.0, 0.4.1, 0.5.1, 0.7.0, 0.6.1,
> 1.1.0, 0.7.1, 1.1.1, 1.0.1
> Environment: By looking at the release tags contained in a commit
> that added User-Defined partition strategy (NIFI-1097 22de23b), it seems all
> NiFi versions since 0.4.0 is affected.
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 0.8.0, 1.2.0
>
>
> NullPointerException is thrown because PutKafka tries to put null into
> properties since following if statements doesn't cover
> USER_DEFINED_PARTITIONING.
> {code: title=PutKafka.java buildKafkaConfigProperties}
> String partitionStrategy = context.getProperty(PARTITION_STRATEGY).getValue();
> String partitionerClass = null;
> if (partitionStrategy.equalsIgnoreCase(ROUND_ROBIN_PARTITIONING.getValue())) {
> partitionerClass = Partitioners.RoundRobinPartitioner.class.getName();
> } else if
> (partitionStrategy.equalsIgnoreCase(RANDOM_PARTITIONING.getValue())) {
> partitionerClass = Partitioners.RandomPartitioner.class.getName();
> }
> properties.setProperty("partitioner.class", partitionerClass); // Happens here
> {code}
> A naive fix for this would be adding one more if statement so that it puts
> 'partitioner.class' property only if partitionerClass is set.
> However, while I was testing the fix, I found following facts, that revealed
> this approach wouldn't be the right solution for this issue.
> In short, we don't have to set 'partitioner.class' property with Kafka 0.8.x
> client in the first place. I assume it's there because PutKafka came through
> a long history..
> h2. PutKafka history analysis
> - PutKafka used to cover Kafka 0.8 and 0.9
> - Around the time Kafka 0.9 was released, PutKafka added 'partitioner.class'
> via NIFI-1097: start using new API. There were two client libraries
> kafka-clients and kafka_2.9.1, both 0.8.2.2.
> - Then PublishKafka is added for Kafka 0.9, at this point, we could add
> 'partition' property to PublishKafka, but we didn't do that for some reason.
> PublishKafka doesn't support user defined partition as of this writing.
> NIFI-1296.
> - The code adding 'partitioner.class' has been left in PutKafka.
> - Further, we separated nar into 0.8, 0.9 and 0.10.
> - Now only PutKafka(0.8) uses 'partitioner.class' property, but 0.8 client
> doesn't use that property. So we don't need that code at all.
> h2. Then, how should we fix this?
> Since PutKafka in both master and 0.x branch specifically uses Kafka 0.8.x
> client. We can simply remove the codes adding 'partitioner.class', probably
> PARTITION_STRATEGY processor property, too.
> h2. Expected result after fix
> - Users can specify Kafka partition with PutKafka 'partition' property, but
> no need to specify 'partition strategy', NullPointerException won't be thrown
> - A warning log, that used to be logged in nifi-app.log won't be logged any
> more: {code}2017-01-18 13:53:33,071 WARN [Timer-Driven Process Thread-9]
> o.a.k.clients.producer.ProducerConfig The configuration partitioner.class =
> null was supplied but isn't a known config.{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-3223) Allow PublishAMQP to use NiFi expression language
[ https://issues.apache.org/jira/browse/NIFI-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-3223: -- Assignee: Oleg Zhurakousky > Allow PublishAMQP to use NiFi expression language > - > > Key: NIFI-3223 > URL: https://issues.apache.org/jira/browse/NIFI-3223 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 0.7.0, 1.1.0, 0.7.1 >Reporter: Brian >Assignee: Oleg Zhurakousky > > Enable the use of NiFi expression language for the PublishAMQP processors, > Routing Key value to allow it to be better used within the NiFi workflows. > PublishAMQP fields to enable: > "Routing Key" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-2615) Add support for GetTCP processor
[ https://issues.apache.org/jira/browse/NIFI-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky resolved NIFI-2615. Resolution: Fixed [~apsaltis] thank you so much for your hard work and contribution. I know it took a bit longer then it had to but it is finally merged. > Add support for GetTCP processor > > > Key: NIFI-2615 > URL: https://issues.apache.org/jira/browse/NIFI-2615 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0, 0.6.1 >Reporter: Andrew Psaltis >Assignee: Andrew Psaltis > Fix For: 1.2.0 > > > This processor will allow NiFi to connect to a host via TCP, thus acting as > the client and consume data. This should provide the following properties: > * Endpoint - this should accept a list of addresses in the format of > : - if a user wants to be able to track the ingestion rate per > address then you would want to have one address in this list. However, there > are times when multiple endpoints represent a logical entity and the > aggregate ingestion rate is representative of it. > * Failover Endpoint - An endpoint to fall over to if the list of Endpoints is > exhausted and a connection cannot be made to them or it is disconnected and > cannot reconnect. > * Receive Buffer Size -- The size of the TCP receive buffer to use. This does > not related to the size of content in the resulting flow file. > * Keep Alive -- This enables TCP keep Alive > * Connection Timeout -- How long to wait when trying to establish a connection > * Batch Size - The number of messages to put into a Flow File. This will be > the max number of messages, as there may be cases where the number of > messages received over the wire waiting to be emitted as FF content may be > less then the desired batch. > This processor should also support the following: > 1. If a connection to end endpoint is broken, it should be logged and > reconnections to it should be made. Potentially an exponential backoff > strategy will be used. The strategy if there is more than one should be > documented and potentially exposed as an Attribute. > 2. When there are multiple instances of this processor in a flow and NiFi is > setup in a cluster, this processor needs to ensure that received messages are > not dual processed. For example if this processor is configured to point to > the endpoint (172.31.32.212:1) and the data flow is running on more than > one node then only one node should be processing data. In essence they should > form a group and have similar semantics as a Kafka consumer group does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2615) Add support for GetTCP processor
[ https://issues.apache.org/jira/browse/NIFI-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2615: --- Fix Version/s: 1.2.0 > Add support for GetTCP processor > > > Key: NIFI-2615 > URL: https://issues.apache.org/jira/browse/NIFI-2615 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0, 0.6.1 >Reporter: Andrew Psaltis >Assignee: Andrew Psaltis > Fix For: 1.2.0 > > > This processor will allow NiFi to connect to a host via TCP, thus acting as > the client and consume data. This should provide the following properties: > * Endpoint - this should accept a list of addresses in the format of > : - if a user wants to be able to track the ingestion rate per > address then you would want to have one address in this list. However, there > are times when multiple endpoints represent a logical entity and the > aggregate ingestion rate is representative of it. > * Failover Endpoint - An endpoint to fall over to if the list of Endpoints is > exhausted and a connection cannot be made to them or it is disconnected and > cannot reconnect. > * Receive Buffer Size -- The size of the TCP receive buffer to use. This does > not related to the size of content in the resulting flow file. > * Keep Alive -- This enables TCP keep Alive > * Connection Timeout -- How long to wait when trying to establish a connection > * Batch Size - The number of messages to put into a Flow File. This will be > the max number of messages, as there may be cases where the number of > messages received over the wire waiting to be emitted as FF content may be > less then the desired batch. > This processor should also support the following: > 1. If a connection to end endpoint is broken, it should be logged and > reconnections to it should be made. Potentially an exponential backoff > strategy will be used. The strategy if there is more than one should be > documented and potentially exposed as an Attribute. > 2. When there are multiple instances of this processor in a flow and NiFi is > setup in a cluster, this processor needs to ensure that received messages are > not dual processed. For example if this processor is configured to point to > the endpoint (172.31.32.212:1) and the data flow is running on more than > one node then only one node should be processing data. In essence they should > form a group and have similar semantics as a Kafka consumer group does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3354) Create CSV To Avro transformer
[ https://issues.apache.org/jira/browse/NIFI-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3354: --- Status: Patch Available (was: Open) > Create CSV To Avro transformer > -- > > Key: NIFI-3354 > URL: https://issues.apache.org/jira/browse/NIFI-3354 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky > > While we currently have CSV to AVRO transformer it required HDFS/Kite > dependencies which could be easily eliminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3255:
---
Status: Patch Available (was: Open)
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.1, 1.1.0
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3278) TextLineDemarcator fails when InputStream ends with '\r' and it's length equals buffer length
[ https://issues.apache.org/jira/browse/NIFI-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3278: --- Description: This is really an edge case, but there is a bug in _isEol()_ operation which attempts to read the next byte after call to _fill()_ even though that may be the end of the stream. And it only happens IF the current character is '\r' and the length of the InputStream is the length of the buffer. > TextLineDemarcator fails when InputStream ends with '\r' and it's length > equals buffer length > - > > Key: NIFI-3278 > URL: https://issues.apache.org/jira/browse/NIFI-3278 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.2.0 > > > This is really an edge case, but there is a bug in _isEol()_ operation which > attempts to read the next byte after call to _fill()_ even though that may be > the end of the stream. And it only happens IF the current character is '\r' > and the length of the InputStream is the length of the buffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3278) TextLineDemarcator fails when InputStream ends with '\r' and it's length equals buffer length
[ https://issues.apache.org/jira/browse/NIFI-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3278: --- Status: Patch Available (was: Open) > TextLineDemarcator fails when InputStream ends with '\r' and it's length > equals buffer length > - > > Key: NIFI-3278 > URL: https://issues.apache.org/jira/browse/NIFI-3278 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.2.0 > > > This is really an edge case, but there is a bug in _isEol()_ operation which > attempts to read the next byte after call to _fill()_ even though that may be > the end of the stream. And it only happens IF the current character is '\r' > and the length of the InputStream is the length of the buffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3278) TextLineDemarcator fails when InputStream ends with '\r' and it's length equals buffer length
Oleg Zhurakousky created NIFI-3278: -- Summary: TextLineDemarcator fails when InputStream ends with '\r' and it's length equals buffer length Key: NIFI-3278 URL: https://issues.apache.org/jira/browse/NIFI-3278 Project: Apache NiFi Issue Type: Bug Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky Priority: Critical Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2693) Documentation fails to load if .nar unpacking fails at startup
[
https://issues.apache.org/jira/browse/NIFI-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783062#comment-15783062
]
Oleg Zhurakousky commented on NIFI-2693:
[~d3v3l0] Actually, thank you for reminding. Will do my best to target it for
the next release. Shouldn't be that difficult.
> Documentation fails to load if .nar unpacking fails at startup
> --
>
> Key: NIFI-2693
> URL: https://issues.apache.org/jira/browse/NIFI-2693
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Joey Frazee
>Assignee: Oleg Zhurakousky
>
> The NiFi documentation can fail to load and throws an NPE if some but not all
> of the .nars don't unpack at startup.
> {code}
> HTTP ERROR 500
> Problem accessing /nifi-docs/documentation. Reason:
> Server Error
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.nifi.web.docs.DocumentationController.doGet(DocumentationController.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> A warning does get logged, but it doesn't give you any indication of which
> .nar file is broken [1], so while it's easy to spot that the problem is with
> the .nar unpacking, you have no idea which .nar is causing it.
> The NPE appears to be because when a .nar doesn't unpack, it doesn't return
> the extensionMapping thus far [2], so NiFi is operational with a subset of
> the .nars, but doesn't get access to docs for those since instead of a
> partial extensionMapping it's just null.
> 1.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java#L162
> 2.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java#L169
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780487#comment-15780487
]
Oleg Zhurakousky edited comment on NIFI-3255 at 12/27/16 2:24 PM:
--
I am now wondering if the check in StandardProcessSession:2216 is valid:
{code}
if (sources.contains(destination)) {
throw new IllegalArgumentException("Destination cannot be within
sources");
}
{code}
It doesn't seem to be covered by any tests and I don't see any references to it
except in SplitText.
Anyway, i'll dig in and add tests for it to TestStandardProcessSession and will
fix the MockProcessSession. Will raise new JIRA as subtask of this one, since I
do not believe the issue is with SplitText, rather with ProcessSession
implementation.
was (Author: ozhurakousky):
I am now wondering if the check in StandaprdProcessSession:2216 is valid:
{code}
if (sources.contains(destination)) {
throw new IllegalArgumentException("Destination cannot be within
sources");
}
{code}
It doesn't seem to be covered by any tests and I don't see any references to it
except in SplitText.
Anyway, i'll dig in and add tests for it to TestStandardProcessSession and will
fix the MockProcessSession. Will raise new JIRA as subtask of this one, since I
do not believe the issue is with SplitText, rather with ProcessSession
implementation.
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780487#comment-15780487
]
Oleg Zhurakousky edited comment on NIFI-3255 at 12/27/16 2:24 PM:
--
I am now wondering if the check in StandaprdProcessSession:2216 is valid:
{code}
if (sources.contains(destination)) {
throw new IllegalArgumentException("Destination cannot be within
sources");
}
{code}
It doesn't seem to be covered by any tests and I don't see any references to it
except in SplitText.
Anyway, i'll dig in and add tests for it to TestStandardProcessSession and will
fix the MockProcessSession. Will raise new JIRA as subtask of this one, since I
do not believe the issue is with SplitText, rather with ProcessSession
implementation.
was (Author: ozhurakousky):
I am now wondering if the check in StandaprdProcessSession:2216 is valid:
{code}
if (sources.contains(destination)) {
throw new IllegalArgumentException("Destination cannot be within
sources");
}
{code}
It doesn't seem to be covered by any tests and I don't see any references to it
except fro SplitText.
Anyway, i'll dig in and add tests for it to TestStandardProcessSession and will
fix the MockProcessSession. Will raise new JIRA as subtask of this one, since I
do not believe the issue is with SplitText, rather with ProcessSession
implementation.
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-3256) MockProcessSession and StandardProcessSession have deviated from one another
Oleg Zhurakousky created NIFI-3256: -- Summary: MockProcessSession and StandardProcessSession have deviated from one another Key: NIFI-3256 URL: https://issues.apache.org/jira/browse/NIFI-3256 Project: Apache NiFi Issue Type: Sub-task Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky Fix For: 1.2.0 Certain operations in MockProcessSession and StandardProcessSession exhibit different behavior. One of such operations is _merge_. (see NIFI-3225) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780487#comment-15780487
]
Oleg Zhurakousky commented on NIFI-3255:
I am now wondering if the check in StandaprdProcessSession:2216 is valid:
{code}
if (sources.contains(destination)) {
throw new IllegalArgumentException("Destination cannot be within
sources");
}
{code}
It doesn't seem to be covered by any tests and I don't see any references to it
except fro SplitText.
Anyway, i'll dig in and add tests for it to TestStandardProcessSession and will
fix the MockProcessSession. Will raise new JIRA as subtask of this one, since I
do not believe the issue is with SplitText, rather with ProcessSession
implementation.
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-3255) SplitText fails with IllegalArgumentException: Destination cannot be within sources
[
https://issues.apache.org/jira/browse/NIFI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky reassigned NIFI-3255:
--
Assignee: Oleg Zhurakousky
> SplitText fails with IllegalArgumentException: Destination cannot be within
> sources
> ---
>
> Key: NIFI-3255
> URL: https://issues.apache.org/jira/browse/NIFI-3255
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Koji Kawamura
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> If SplitText is configured with Header Line Count grater than 0, and input
> flow file like below is passed, it fails with IllegalArgumentException:
> {code}
> header
> line1
> line2
> {code}
> Stacktrace:
> {code}
> 2016-12-27 16:41:51,016 WARN [Timer-Driven Process Thread-2]
> o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException:
> Destination cannot be within sources
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2217)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.merge(StandardProcessSession.java:2209)
> ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.nifi.processors.standard.SplitText.generateSplitFlowFiles(SplitText.java:305)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:253)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.2.0-SNAPSHOT.jar:1
> {code}
> Marked as critical since this is a regression and breaks existing flow. With
> 1.0, SplitText produces following output flow files to "split" relationship
> with the same input flow file and processor configuration:
> {code}
> # output flow file 1
> header
> line1
> # output flow file 2
> header
> line 2
> {code}
> The related code block in the processor has been covered by unit tests,
> however, since unit test uses MockProcessSession, it skips check conditions
> in StandardProcessSession. MockProcessSession should be updated, too, in
> order to catch this type of issue with Unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name
[ https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-3193: -- Assignee: Oleg Zhurakousky > Update ConsumeAMQP and PublishAMQP to retrieve username from certificate > common name > > > Key: NIFI-3193 > URL: https://issues.apache.org/jira/browse/NIFI-3193 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.0.0, 1.1.0, 0.7.1 >Reporter: Brian >Assignee: Oleg Zhurakousky > > At the moment the NiFi AMQP processors can establish a SSL connection to > RabbitMQ but still user a user defined username and password to authenticate. > When using certificates RabbitMQ allows you to use to COMMON_NAME from the > certificate to authenticate instead of providing a username and password. > Unfortunately the NiFi processors do not support this so I would like to > request an update to the processors to enable this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3179) MergeContent extracts demarcator property value bytes without specifying charset encoding
Oleg Zhurakousky created NIFI-3179: -- Summary: MergeContent extracts demarcator property value bytes without specifying charset encoding Key: NIFI-3179 URL: https://issues.apache.org/jira/browse/NIFI-3179 Project: Apache NiFi Issue Type: Bug Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky This may cause issues with byte translation with default encodings of the machine -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3172) Provide for a better diagnostics for non-existing exchange vs bad routingKey in PublishAMQP
Oleg Zhurakousky created NIFI-3172: -- Summary: Provide for a better diagnostics for non-existing exchange vs bad routingKey in PublishAMQP Key: NIFI-3172 URL: https://issues.apache.org/jira/browse/NIFI-3172 Project: Apache NiFi Issue Type: Bug Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky A user on the mailing list has reported the issue that demonstrated a condition where the first message sent to AMQP always appears to be sent successfully while never reaching its destination. However, the second message and each subsequent message would fail due to a closed channel exception. This is due to the fact that the channel was closed as soon as client had determined that the provided exchange does not exist. Now, that could be due to a misconfiguration or as reported by the user and in his case seems to be due to an older version of RabbitMQ/Erlang pair. In any event, that was not properly reported to the user of PublishAMQP processor even though the fix for that is relatively simple and that is to check the state of the channel after _basicPublish()_ call (i.e., channel.isOpen()). Further more, there could also be an issue with the bad _routingKey_ for which, as a part of this ticket, we should register a ReturnListener on the channel, which given that we set _mandatory_ flag to always be true, would allow us to know immediately after basic publish if a Message is routable or not (essentially delivered or not) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3161) nifi-mock pulls two different versions of ASM libraries
Oleg Zhurakousky created NIFI-3161:
--
Summary: nifi-mock pulls two different versions of ASM libraries
Key: NIFI-3161
URL: https://issues.apache.org/jira/browse/NIFI-3161
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Oleg Zhurakousky
Assignee: Oleg Zhurakousky
Priority: Minor
With _nifi-mock_ dependency one will observe two ASM libraries
{code}
/Users/you/.m2/repository/net/minidev/asm/1.0.2/asm-1.0.2.jar
/Users/you/.m2/repository/asm/asm/3.3.1/asm-3.3.1.jar
{code}
They are two different libraries, but one (minidev) includes packages from the
other and may result in namespace collision. Basically both JARs include
_org.objectweb.asm.*_
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3141) TailFile failed with ArrayIndexOutOfBoundsException after updating NiFi to 1.1.0
[
https://issues.apache.org/jira/browse/NIFI-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3141:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TailFile failed with ArrayIndexOutOfBoundsException after updating NiFi to
> 1.1.0
>
>
> Key: NIFI-3141
> URL: https://issues.apache.org/jira/browse/NIFI-3141
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: NiFi 1.0.0 TailFile status.png, NiFi 1.1.0 TailFile
> status.png
>
>
> TailFile with NiFi 1.0.0 or earlier only handles single file. So it doesn't
> have index in state key name. But the updated TailFile expects stored state
> key has index in it, tried to extract the index and failed.
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.processors.standard.TailFile.initStates(TailFile.java:365)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.TailFile.recoverState(TailFile.java:348)
> ~[na:na]
> ... 16 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3141) TailFile failed with ArrayIndexOutOfBoundsException after updating NiFi to 1.1.0
[
https://issues.apache.org/jira/browse/NIFI-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3141:
---
Fix Version/s: 1.2.0
> TailFile failed with ArrayIndexOutOfBoundsException after updating NiFi to
> 1.1.0
>
>
> Key: NIFI-3141
> URL: https://issues.apache.org/jira/browse/NIFI-3141
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.2.0
>
> Attachments: NiFi 1.0.0 TailFile status.png, NiFi 1.1.0 TailFile
> status.png
>
>
> TailFile with NiFi 1.0.0 or earlier only handles single file. So it doesn't
> have index in state key name. But the updated TailFile expects stored state
> key has index in it, tried to extract the index and failed.
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.processors.standard.TailFile.initStates(TailFile.java:365)
> ~[na:na]
> at
> org.apache.nifi.processors.standard.TailFile.recoverState(TailFile.java:348)
> ~[na:na]
> ... 16 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2701) Can’t consume JMS Messages from remote broker
[ https://issues.apache.org/jira/browse/NIFI-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706469#comment-15706469 ] Oleg Zhurakousky commented on NIFI-2701: Raymond Have a bit of a conundrum here. I can't seem to find any download links for SonicMQ. Any pointers? Would love to test what we have against it and see if any gaps needs to be filled. > Can’t consume JMS Messages from remote broker > - > > Key: NIFI-2701 > URL: https://issues.apache.org/jira/browse/NIFI-2701 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 > Environment: Windows 7 >Reporter: Raymond >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > I’m using the ConsumeJMS together with JMSConnectionFactoryProvider service > to consume message from a SonicMQ broker. If I’m using a NiFi instance and > the broker on the same machine this works. I can consume the messages from > the queue on the broker (using the broker url: tcp://localhost:2506). > If the broker is on another machine I get the following error: > 08:55:48 CEST > WARNING > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > Processor Administratively Yielded for 1 sec due to processing failure > 08:55:51 CEST > ERROR > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > failed to process due to org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: java.net.ConnectException: Connection refused: > connect: ServerA; rolling back session: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: ServerA > Server A is the machine where the NiFi instance is running. So the connection > is not refused by the machine where the broker is running. The broker is > running correctly and I can connect from the same machine with other JMS > Clients with same credentials. This problem occurred with a NiFi 0.6.1 and > 1.0B instance. > Log: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: SERVERA > at > org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316) > ~[na:na] > at > org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:764) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:738) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:727) > ~[na:na] > at > org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:65) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:79) > ~[na:na] > at > org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:136) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:50) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_102] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_102] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) [na:1.8.0_102] > at >
[jira] [Updated] (NIFI-3109) Refactor TimerDrivenSchedulingAgent
[ https://issues.apache.org/jira/browse/NIFI-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3109: --- Status: Patch Available (was: Open) > Refactor TimerDrivenSchedulingAgent > --- > > Key: NIFI-3109 > URL: https://issues.apache.org/jira/browse/NIFI-3109 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 1.2.0 > > > While working on determining reasons for bug described in NIFI-2886, I've > noticed an opportunity for some refactoring which in a way is significant as > it greatly reduces the code base, eliminates unnecessary synchronization as > well as provides for better reusability of object instances. (i.e., Currently > for each task an instance of ContinuallyRunProcessorTask is created while the > same instance could be reused for all tasks, and more) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2886) Framework doesn't release thread if processor administratively yields
[ https://issues.apache.org/jira/browse/NIFI-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2886: --- Status: Patch Available (was: Open) > Framework doesn't release thread if processor administratively yields > - > > Key: NIFI-2886 > URL: https://issues.apache.org/jira/browse/NIFI-2886 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David A. Wynne >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > If a processor yields due to a exception from onScheduled it doesn't > immediately release the thread back to the pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2886) Framework doesn't release thread if processor administratively yields
[ https://issues.apache.org/jira/browse/NIFI-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702642#comment-15702642 ] Oleg Zhurakousky commented on NIFI-2886: Actually while above code snippet would still have to be revisited, the core issue is in _StandardProcessNode.start(..)_. We basically schedule the next re-try in the _catch_ clause so so the Future is already in the queue > Framework doesn't release thread if processor administratively yields > - > > Key: NIFI-2886 > URL: https://issues.apache.org/jira/browse/NIFI-2886 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David A. Wynne >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > If a processor yields due to a exception from onScheduled it doesn't > immediately release the thread back to the pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3109) Refactor TimerDrivenSchedulingAgent
[ https://issues.apache.org/jira/browse/NIFI-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3109: --- Description: While working on determining reasons for bug described in NIFI-2886, I've noticed an opportunity for some refactoring which in a way is significant as it greatly reduces the code base, eliminates unnecessary synchronization as well as provides for better reusability of object instances. (i.e., Currently for each task an instance of ContinuallyRunProcessorTask is created while the same instance could be reused for all tasks, and more) (was: While working on determining reasons for bug described in NIFI-2886, I've noticed an opportunity for some refactoring opportunity which in a way is significant as it greatly reduces the code base, eliminates unnecessary synchronization as well as provides for better reusability of object instances. (i.e., Currently for each task an instance of ContinuallyRunProcessorTask is created while the same instance could be reused for all tasks, and more)) > Refactor TimerDrivenSchedulingAgent > --- > > Key: NIFI-3109 > URL: https://issues.apache.org/jira/browse/NIFI-3109 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 1.2.0 > > > While working on determining reasons for bug described in NIFI-2886, I've > noticed an opportunity for some refactoring which in a way is significant as > it greatly reduces the code base, eliminates unnecessary synchronization as > well as provides for better reusability of object instances. (i.e., Currently > for each task an instance of ContinuallyRunProcessorTask is created while the > same instance could be reused for all tasks, and more) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3109) Refactor TimerDrivenSchedulingAgent
Oleg Zhurakousky created NIFI-3109: -- Summary: Refactor TimerDrivenSchedulingAgent Key: NIFI-3109 URL: https://issues.apache.org/jira/browse/NIFI-3109 Project: Apache NiFi Issue Type: Sub-task Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky Priority: Minor Fix For: 1.2.0 While working on determining reasons for bug described in NIFI-2886, I've noticed an opportunity for some refactoring opportunity which in a way is significant as it greatly reduces the code base, eliminates unnecessary synchronization as well as provides for better reusability of object instances. (i.e., Currently for each task an instance of ContinuallyRunProcessorTask is created while the same instance could be reused for all tasks, and more) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2701) Can’t consume JMS Messages from remote broker
[ https://issues.apache.org/jira/browse/NIFI-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2701: --- Fix Version/s: 1.2.0 > Can’t consume JMS Messages from remote broker > - > > Key: NIFI-2701 > URL: https://issues.apache.org/jira/browse/NIFI-2701 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 > Environment: Windows 7 >Reporter: Raymond >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > I’m using the ConsumeJMS together with JMSConnectionFactoryProvider service > to consume message from a SonicMQ broker. If I’m using a NiFi instance and > the broker on the same machine this works. I can consume the messages from > the queue on the broker (using the broker url: tcp://localhost:2506). > If the broker is on another machine I get the following error: > 08:55:48 CEST > WARNING > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > Processor Administratively Yielded for 1 sec due to processing failure > 08:55:51 CEST > ERROR > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > failed to process due to org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: java.net.ConnectException: Connection refused: > connect: ServerA; rolling back session: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: ServerA > Server A is the machine where the NiFi instance is running. So the connection > is not refused by the machine where the broker is running. The broker is > running correctly and I can connect from the same machine with other JMS > Clients with same credentials. This problem occurred with a NiFi 0.6.1 and > 1.0B instance. > Log: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: SERVERA > at > org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316) > ~[na:na] > at > org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:764) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:738) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:727) > ~[na:na] > at > org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:65) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:79) > ~[na:na] > at > org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:136) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:50) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_102] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_102] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) [na:1.8.0_102] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) [na:1.8.0_102] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.8.0_102] > at
[jira] [Commented] (NIFI-2701) Can’t consume JMS Messages from remote broker
[ https://issues.apache.org/jira/browse/NIFI-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15700496#comment-15700496 ] Oleg Zhurakousky commented on NIFI-2701: Raymond Thank you so much for providing that input. I will definitely take a look. In all fairness while the intention was to be JMS-provider agnostic what was tested is the likes of IBM, Tibco and ActiveMQ. I've personally never worked of SonicMQ and based on few google searches was under impression that only JNDI CFs are supported. But based on your input I am now eager to find what's missing in current code base to support SonicMQ (thanks to your sample). So I'll keep the title as is for now and hopefully once it's all figured out it can be changed. That said, I believe that there is somethiny minor, so I am going to set target version of 1.2. Thanks again! Oleg > Can’t consume JMS Messages from remote broker > - > > Key: NIFI-2701 > URL: https://issues.apache.org/jira/browse/NIFI-2701 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 > Environment: Windows 7 >Reporter: Raymond >Assignee: Oleg Zhurakousky > Fix For: 1.2.0 > > > I’m using the ConsumeJMS together with JMSConnectionFactoryProvider service > to consume message from a SonicMQ broker. If I’m using a NiFi instance and > the broker on the same machine this works. I can consume the messages from > the queue on the broker (using the broker url: tcp://localhost:2506). > If the broker is on another machine I get the following error: > 08:55:48 CEST > WARNING > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > Processor Administratively Yielded for 1 sec due to processing failure > 08:55:51 CEST > ERROR > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > failed to process due to org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: java.net.ConnectException: Connection refused: > connect: ServerA; rolling back session: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: ServerA > Server A is the machine where the NiFi instance is running. So the connection > is not refused by the machine where the broker is running. The broker is > running correctly and I can connect from the same machine with other JMS > Clients with same credentials. This problem occurred with a NiFi 0.6.1 and > 1.0B instance. > Log: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: SERVERA > at > org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316) > ~[na:na] > at > org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:764) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:738) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:727) > ~[na:na] > at > org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:65) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:79) > ~[na:na] > at > org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:136) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:50) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at >
[jira] [Commented] (NIFI-2886) Framework doesn't release thread if processor administratively yields
[
https://issues.apache.org/jira/browse/NIFI-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699962#comment-15699962
]
Oleg Zhurakousky commented on NIFI-2886:
Looks like it is due (at least partially) to the following code in
_StandardProcessScheduler_
{code}
try {
Thread.sleep(administrativeYieldMillis);
} catch (final InterruptedException ie) {
}
{code}
> Framework doesn't release thread if processor administratively yields
> -
>
> Key: NIFI-2886
> URL: https://issues.apache.org/jira/browse/NIFI-2886
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: David A. Wynne
>Assignee: Oleg Zhurakousky
> Fix For: 1.2.0
>
>
> If a processor yields due to a exception from onScheduled it doesn't
> immediately release the thread back to the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-2693) Documentation fails to load if .nar unpacking fails at startup
[
https://issues.apache.org/jira/browse/NIFI-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky reassigned NIFI-2693:
--
Assignee: Oleg Zhurakousky
> Documentation fails to load if .nar unpacking fails at startup
> --
>
> Key: NIFI-2693
> URL: https://issues.apache.org/jira/browse/NIFI-2693
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Joey Frazee
>Assignee: Oleg Zhurakousky
>
> The NiFi documentation can fail to load and throws an NPE if some but not all
> of the .nars don't unpack at startup.
> {code}
> HTTP ERROR 500
> Problem accessing /nifi-docs/documentation. Reason:
> Server Error
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.nifi.web.docs.DocumentationController.doGet(DocumentationController.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> A warning does get logged, but it doesn't give you any indication of which
> .nar file is broken [1], so while it's easy to spot that the problem is with
> the .nar unpacking, you have no idea which .nar is causing it.
> The NPE appears to be because when a .nar doesn't unpack, it doesn't return
> the extensionMapping thus far [2], so NiFi is operational with a subset of
> the .nars, but doesn't get access to docs for those since instead of a
> partial extensionMapping it's just null.
> 1.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java#L162
> 2.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java#L169
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-1796) Create the 'Hello World' example of custom processor development
[ https://issues.apache.org/jira/browse/NIFI-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-1796: -- Assignee: Oleg Zhurakousky > Create the 'Hello World' example of custom processor development > > > Key: NIFI-1796 > URL: https://issues.apache.org/jira/browse/NIFI-1796 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Joseph Witt >Assignee: Oleg Zhurakousky > > We really need to create a simple guide for how to build custom processor in > nifi. The sort of 'hello world' example. > This was created due to finding it hard to give a nice simple short pointer > for a question on the mailing list. > We do have a solid developer guide here [1] that goes into the key concepts > and processor development a bit. The next best thing is to use the supplied > maven archetype to lay down the fundamental pieces. You can read a bit more > about that here [2]. Finally something that is also quite helpful is looking > at the standard processors to see if anything follows the pattern/problem you > had in mind and which can serve as a good basis [3] > [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html > [2] > https://richardstechnotes.wordpress.com/2015/12/06/setting-up-for-custom-nifi-processor-development/ > [3] > https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2255) ConsumeJMS does not detect dead connections
[ https://issues.apache.org/jira/browse/NIFI-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-2255: -- Assignee: Oleg Zhurakousky > ConsumeJMS does not detect dead connections > --- > > Key: NIFI-2255 > URL: https://issues.apache.org/jira/browse/NIFI-2255 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 >Reporter: Christopher McDermott >Assignee: Oleg Zhurakousky > > Once the connection is dead ConsumeJMS will no longer receive any messages > even after the broker becomes available because it will not open new > connection. > To reproduce: do a ‘reboot –f now’ on the node running the JMS broker and > ConsumeJMS will not notice a problem until the TCP timeout which could be > hours, days, or even never. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2040) JMSConnectionFactoryProvider - "MQ Client Libraries path" property documentation is incorrect
[ https://issues.apache.org/jira/browse/NIFI-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-2040: -- Assignee: Oleg Zhurakousky > JMSConnectionFactoryProvider - "MQ Client Libraries path" property > documentation is incorrect > - > > Key: NIFI-2040 > URL: https://issues.apache.org/jira/browse/NIFI-2040 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 0.6.0 >Reporter: Christopher McDermott >Assignee: Oleg Zhurakousky > > The documentation on JMSConnectionFactoryProvider says that the "MQ Client > Libraries path" path property is optional with > org.apache.activemq.ActiveMQConnectionFactory provider. That is incorrect. I > couple good examples would be helpful. Showing how to configure an AMQ > provider would be most helpful since that is probably the most widely used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2169) Improve RouteText performance with pre-compilation of RegEx in certain cases
[ https://issues.apache.org/jira/browse/NIFI-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-2169: -- Assignee: Oleg Zhurakousky > Improve RouteText performance with pre-compilation of RegEx in certain cases > > > Key: NIFI-2169 > URL: https://issues.apache.org/jira/browse/NIFI-2169 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.6.1 >Reporter: Stephane Maarek >Assignee: Oleg Zhurakousky > Labels: beginner, easy > > When using RegEx matches for the RouteText processor (and possibly other > processors), the RegEx gets recompiled every time the processor works. The > RegEx could be precompiled / cached under certain conditions, in order to > improve the performance of the processor > See email from Mark Payne: > Re #2: The regular expression is compiled every time. This is done, though, > because the Regex allows the Expression > Language to be used, so the Regex could actually be different for each > FlowFile. That being said, it could certainly be > improved by either (a) pre-compiling in the case that no Expression Language > is used and/or (b) cache up to say 10 > Regex'es once they are compiled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-610) PutJMS should allow the Destination Name to support the Expression Language
[ https://issues.apache.org/jira/browse/NIFI-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699726#comment-15699726 ] Oleg Zhurakousky commented on NIFI-610: --- Given that we are deprecating Put/GetJMS, should we just close it? > PutJMS should allow the Destination Name to support the Expression Language > --- > > Key: NIFI-610 > URL: https://issues.apache.org/jira/browse/NIFI-610 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne > > PutJMS does not allow the 'Destination Name' property to contain Expression > Language. This was done so that we can create a single JmsProducer object and > send all messages using this producer. However, this is very limiting, > especially in the case that the data was pulled from GetJMS and has a > 'jms.JMSReplyTo' attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1920) Poor performance for SplitText when its 'splits' relationship is cloned
[
https://issues.apache.org/jira/browse/NIFI-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699722#comment-15699722
]
Oleg Zhurakousky commented on NIFI-1920:
[~markap14] Should this be closed in lieu of recent performance improvements in
SplitText?
> Poor performance for SplitText when its 'splits' relationship is cloned
> ---
>
> Key: NIFI-1920
> URL: https://issues.apache.org/jira/browse/NIFI-1920
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>
> On my laptop, I downloaded the file
> http://standards.ieee.org/develop/regauth/oui/oui.csv and ran it through
> SplitText, splitting into 1-line FlowFiles. It completed in about 1.8
> seconds. I then routed the 'splits' relationship to a second processor so
> that the data would be cloned. Rather than completing in 1.8 seconds, when I
> ran the file through again, it took 2.5 minutes to complete. A stack trace
> during that time shows the time being spent adding the 'CLONE' Provenance
> Event in the Provenance Reporter:
> {code}
> "Timer-Driven Process Thread-3" Id=93 RUNNABLE
> at java.util.HashMap$TreeNode.find(HashMap.java:1865)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.find(HashMap.java:1861)
> at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:1981)
> at java.util.HashMap.putVal(HashMap.java:637)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at
> org.apache.nifi.controller.repository.StandardProcessSession$Checkpoint.checkpoint(StandardProcessSession.java:2572)
> at
> org.apache.nifi.controller.repository.StandardProcessSession$Checkpoint.access$100(StandardProcessSession.java:2539)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:276)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:282)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1072)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3e0d1518
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2124) InputStream should be used in optimized way
[
https://issues.apache.org/jira/browse/NIFI-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699718#comment-15699718
]
Oleg Zhurakousky commented on NIFI-2124:
[~vcharmcaster] I do understand what you are saying but in this particular case
we are not creating an instance of InputStream, rather accessing and existing
one managed by an external class
{code}
InputStream inStream = dataPacket.getData();
{code}
. . . which indirectly derives from _SiteToSiteClient_ which is closed in the
finally block
{code}
try {
client.close();
} catch (final IOException ioe) {
reportError("Failed to close client", ioe);
}
{code}
Anyway, I would recommend closing this issue with "Won't fix".
Let me know if you have any issues with it.
Cheers
Oleg
> InputStream should be used in optimized way
> ---
>
> Key: NIFI-2124
> URL: https://issues.apache.org/jira/browse/NIFI-2124
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: vaibhav khamgavkar
> Labels: performance
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> I was looking into code
> found that
> "nifi-external/nifi-spark-receiver/src/main/java/org/apache/nifi/spark/NiFiReceiver.java"
> class dont use InputStream in optimized way.
> Usually it is good practice to close input stream and reuse variable instead
> of creating one each time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (NIFI-2701) Can’t consume JMS Messages from remote broker
[ https://issues.apache.org/jira/browse/NIFI-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky reassigned NIFI-2701: -- Assignee: Oleg Zhurakousky > Can’t consume JMS Messages from remote broker > - > > Key: NIFI-2701 > URL: https://issues.apache.org/jira/browse/NIFI-2701 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 > Environment: Windows 7 >Reporter: Raymond >Assignee: Oleg Zhurakousky > > I’m using the ConsumeJMS together with JMSConnectionFactoryProvider service > to consume message from a SonicMQ broker. If I’m using a NiFi instance and > the broker on the same machine this works. I can consume the messages from > the queue on the broker (using the broker url: tcp://localhost:2506). > If the broker is on another machine I get the following error: > 08:55:48 CEST > WARNING > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > Processor Administratively Yielded for 1 sec due to processing failure > 08:55:51 CEST > ERROR > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > failed to process due to org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: java.net.ConnectException: Connection refused: > connect: ServerA; rolling back session: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: ServerA > Server A is the machine where the NiFi instance is running. So the connection > is not refused by the machine where the broker is running. The broker is > running correctly and I can connect from the same machine with other JMS > Clients with same credentials. This problem occurred with a NiFi 0.6.1 and > 1.0B instance. > Log: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: SERVERA > at > org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316) > ~[na:na] > at > org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:764) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:738) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:727) > ~[na:na] > at > org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:65) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:79) > ~[na:na] > at > org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:136) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:50) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_102] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_102] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) [na:1.8.0_102] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) [na:1.8.0_102] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.8.0_102] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
[jira] [Commented] (NIFI-2701) Can’t consume JMS Messages from remote broker
[ https://issues.apache.org/jira/browse/NIFI-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699710#comment-15699710 ] Oleg Zhurakousky commented on NIFI-2701: Raymond If you don't mind I would like to change the title of this JIRA to "Add JNDI Factory support for JMS ConnectionFactory service". Current title is a bit misleading since there are no issues with connecting to remote brokers (i.e., IBM, Tibco, ActiveMQ etc), but for those a regular CF is used. Anyway, I'll assign it to myself with hopes of addressing in the near future. > Can’t consume JMS Messages from remote broker > - > > Key: NIFI-2701 > URL: https://issues.apache.org/jira/browse/NIFI-2701 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 > Environment: Windows 7 >Reporter: Raymond > > I’m using the ConsumeJMS together with JMSConnectionFactoryProvider service > to consume message from a SonicMQ broker. If I’m using a NiFi instance and > the broker on the same machine this works. I can consume the messages from > the queue on the broker (using the broker url: tcp://localhost:2506). > If the broker is on another machine I get the following error: > 08:55:48 CEST > WARNING > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > Processor Administratively Yielded for 1 sec due to processing failure > 08:55:51 CEST > ERROR > d6d8ec67-0156-1000-006c-054ada0ce528 > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > ConsumeJMS - JMSConsumer[destination:q.testconnector.inbox; pub-sub:false;] > failed to process due to org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: java.net.ConnectException: Connection refused: > connect: ServerA; rolling back session: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: ServerA > Server A is the machine where the NiFi instance is running. So the connection > is not refused by the machine where the broker is running. The broker is > running correctly and I can connect from the same machine with other JMS > Clients with same credentials. This problem occurred with a NiFi 0.6.1 and > 1.0B instance. > Log: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occured during JMS processing; nested exception is javax.jms.JMSException: > java.net.ConnectException: Connection refused: connect: SERVERA > at > org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316) > ~[na:na] > at > org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:764) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:738) > ~[na:na] > at > org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:727) > ~[na:na] > at > org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:65) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:79) > ~[na:na] > at > org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:136) > ~[na:na] > at > org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:50) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_102] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_102] > at >
[jira] [Commented] (NIFI-3068) NiFi can not reliably support multiple HDFS clusters in the same flow
[ https://issues.apache.org/jira/browse/NIFI-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687983#comment-15687983 ] Oleg Zhurakousky commented on NIFI-3068: [~bende] +1 > NiFi can not reliably support multiple HDFS clusters in the same flow > - > > Key: NIFI-3068 > URL: https://issues.apache.org/jira/browse/NIFI-3068 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Sam Hjelmfelt >Assignee: Bryan Bende > Labels: HDFS > Fix For: 1.1.0 > > Attachments: NIFI-3068.patch > > > The HDFS configurations in PutHDFS are not respected when two (or more) > PutHDFS processors exist with different configurations. The second processor > to run will use the configurations from the first processor. This can cause > data to be written to the wrong cluster. > This appears to be caused by configuration caching in > AbstractHadoopProcessor, which would affect all HDFS processors. > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L144 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3068) NiFi can not reliably support multiple HDFS clusters in the same flow
[ https://issues.apache.org/jira/browse/NIFI-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687984#comment-15687984 ] Oleg Zhurakousky commented on NIFI-3068: [~bende] +1 > NiFi can not reliably support multiple HDFS clusters in the same flow > - > > Key: NIFI-3068 > URL: https://issues.apache.org/jira/browse/NIFI-3068 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Sam Hjelmfelt >Assignee: Bryan Bende > Labels: HDFS > Fix For: 1.1.0 > > Attachments: NIFI-3068.patch > > > The HDFS configurations in PutHDFS are not respected when two (or more) > PutHDFS processors exist with different configurations. The second processor > to run will use the configurations from the first processor. This can cause > data to be written to the wrong cluster. > This appears to be caused by configuration caching in > AbstractHadoopProcessor, which would affect all HDFS processors. > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L144 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
[
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky updated NIFI-3066:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
> -
>
> Key: NIFI-3066
> URL: https://issues.apache.org/jira/browse/NIFI-3066
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.1.0
>
>
> With the following properties in MergeContent, I get this error when I try to
> process a flow file (which is 135MB):
> Merge Strategy = Bin-Packing Algorithm
> Merge Format = ZIP
> Attribute Strategy = Keep Only Common Attributes
> Correlation Attribute Name = bundle.identifier
> Minimum Number of Entries = 200
> Maximum Number of Entries = 200
> Minimum Group Size = 20 MB
> Maximum Group Size = *20 MB*
> Max Bin Age = 1 min
> Maximum number of Bins = 100
> Delimiter Strategy = Filename
> Keep Path = false
> {code}
> 016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83]
> o.a.n.processors.standard.MergeContent
> java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process
> Session to itself
> at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
> ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
> ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> ...
> 2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor
> Administratively Yielded for 1 sec due to processing failure
> {code}
> However, other smaller flow files are able to successfully go through the
> processor. Not sure if it's related to the size of the flow file or not.
> Perhaps there just needs to be a check around this line in Bin.java:142, to
> make sure we don't try to migrate a flow file to its own session:
> {code}
> session.migrate(getSession(), Collections.singleton(flowFile));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-3073) poorly named nifi clones of standard java classes
[
https://issues.apache.org/jira/browse/NIFI-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683893#comment-15683893
]
Oleg Zhurakousky commented on NIFI-3073:
[~devriesb] would you consider changing Fix Version to 1.2 or even remove it
all together untill we go through more formal planning? The reason why I am
suggesting it is that we are trying to get the 1.1 release out the door and are
trying to close the remaining few issues.
Another reason is NIFI-3071, where we may end up deprecating/removing these all
together after re-validating the need for those classes.
Anyway, appreciate your thoughts.
> poorly named nifi clones of standard java classes
> -
>
> Key: NIFI-3073
> URL: https://issues.apache.org/jira/browse/NIFI-3073
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Brandon DeVries
>Priority: Minor
> Fix For: 1.1.0, 0.7.1
>
>
> org.apache.nifi.stream.io.DataOutputStream is a clone of
> java.io.DataOutputStream that *does not* do synchronization. This should
> have a different name, indicating this. Overloading the name invites
> confusion. The comment in the class\[1] should also be corrected to
> accurately reflect its intent:
> {quote}
> This class is different from java.io.DataOutputStream in that it does
> synchronize on its methods.
> {quote}
> The incorrect comment should further illustrate the point on confusion, in
> that both the name and comment imply behavior potentially much different than
> the actual.
> Implementation note... previously there were concerns about backwards
> compatibility should this method be removed. This can be avoided by renaming
> this class, and creating a new org.apache.nifi.stream.io.DataOutputStream
> that is marked as deprecated and simply extends the properly named class
> without modification. org.apache.nifi.stream.io.DataOutputStream can then be
> removed in a future release. This would also have the benefit of
> highlighting for developers any instances where this class was mistakenly
> used.
> It would likely be wise to examine the code base for further instances of
> this (anti-)pattern.
> \[1]
> https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/DataOutputStream.java#L26
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2015) Corrupted flow file leads to a wedged flow
[ https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2015: --- Fix Version/s: 1.1.0 > Corrupted flow file leads to a wedged flow > -- > > Key: NIFI-2015 > URL: https://issues.apache.org/jira/browse/NIFI-2015 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 >Reporter: Christopher McDermott >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.1.0 > > > Encountered the following error. Once encountered the processor in question > cannot make any progress. The incoming connection cannot be emptied or even > listed. Note this issue is not to address the corrupted flow file but rather > the handling of the corrupted file. > 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] > o.a.n.p.attributes.UpdateAttribute > java.lang.IllegalStateException: Cannot create Provenance Event Record > because FlowFile UUID is not set > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) > ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-2015) Corrupted flow file leads to a wedged flow
[ https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15681469#comment-15681469 ] Oleg Zhurakousky edited comment on NIFI-2015 at 11/20/16 5:03 PM: -- This JIRA was reopened due to the issue raise in the Apache Mailing list: https://lists.apache.org/thread.html/12aafa3697664624b97212013731128ec702ffabe106bb313ed1423b@%3Cusers.nifi.apache.org%3E Resolving it due to the fact that: 1. The fix that went into this specific JIRA was unrelated to the problem reported on mailing list 2. The problem reported on the mailing list has been identified and fixed - NIFI-3040 was (Author: ozhurakousky): This JIRA was reopened due to the issue raise in the Apache Mailing list: https://lists.apache.org/thread.html/12aafa3697664624b97212013731128ec702ffabe106bb313ed1423b@%3Cusers.nifi.apache.org%3E Resolving it due to the fact that: 1. The fix that went into this specific JIRA was unrelated to the problem reported on mailing list 2. The problem reported on the mailing list has been identified and fixed - https://issues.apache.org/jira/browse/NIFI-3040 > Corrupted flow file leads to a wedged flow > -- > > Key: NIFI-2015 > URL: https://issues.apache.org/jira/browse/NIFI-2015 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 >Reporter: Christopher McDermott >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.1.0 > > > Encountered the following error. Once encountered the processor in question > cannot make any progress. The incoming connection cannot be emptied or even > listed. Note this issue is not to address the corrupted flow file but rather > the handling of the corrupted file. > 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] > o.a.n.p.attributes.UpdateAttribute > java.lang.IllegalStateException: Cannot create Provenance Event Record > because FlowFile UUID is not set > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) > ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-2015) Corrupted flow file leads to a wedged flow
[ https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky resolved NIFI-2015. Resolution: Fixed This JIRA was reopened due to the issue raise in the Apache Mailing list: https://lists.apache.org/thread.html/12aafa3697664624b97212013731128ec702ffabe106bb313ed1423b@%3Cusers.nifi.apache.org%3E Resolving it due to the fact that: 1. The fix that went into this specific JIRA was unrelated to the problem reported on mailing list 2. The problem reported on the mailing list has been identified and fixed - https://issues.apache.org/jira/browse/NIFI-3040 > Corrupted flow file leads to a wedged flow > -- > > Key: NIFI-2015 > URL: https://issues.apache.org/jira/browse/NIFI-2015 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 >Reporter: Christopher McDermott >Assignee: Oleg Zhurakousky >Priority: Critical > > Encountered the following error. Once encountered the processor in question > cannot make any progress. The incoming connection cannot be emptied or even > listed. Note this issue is not to address the corrupted flow file but rather > the handling of the corrupted file. > 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] > o.a.n.p.attributes.UpdateAttribute > java.lang.IllegalStateException: Cannot create Provenance Event Record > because FlowFile UUID is not set > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412) > ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) > ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_45] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3071) Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor of jaba.io.ByteArrayOutputStream
Oleg Zhurakousky created NIFI-3071: -- Summary: Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor of jaba.io.ByteArrayOutputStream Key: NIFI-3071 URL: https://issues.apache.org/jira/browse/NIFI-3071 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.0.0 Reporter: Oleg Zhurakousky Assignee: Oleg Zhurakousky Priority: Minor I believe the "efficiency" statements need to be revisited. For example, preliminary testing shows that there are no performance difference between using the one provided with java.io. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
[
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15681278#comment-15681278
]
Oleg Zhurakousky commented on NIFI-3066:
[~markap14] It's marked "Patch Available" but no patch is attached/linked.
> MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
> -
>
> Key: NIFI-3066
> URL: https://issues.apache.org/jira/browse/NIFI-3066
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.1.0
>
>
> With the following properties in MergeContent, I get this error when I try to
> process a flow file (which is 135MB):
> Merge Strategy = Bin-Packing Algorithm
> Merge Format = ZIP
> Attribute Strategy = Keep Only Common Attributes
> Correlation Attribute Name = bundle.identifier
> Minimum Number of Entries = 200
> Maximum Number of Entries = 200
> Minimum Group Size = 20 MB
> Maximum Group Size = *20 MB*
> Max Bin Age = 1 min
> Maximum number of Bins = 100
> Delimiter Strategy = Filename
> Keep Path = false
> {code}
> 016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83]
> o.a.n.processors.standard.MergeContent
> java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process
> Session to itself
> at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
> ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
> ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> ...
> 2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor
> Administratively Yielded for 1 sec due to processing failure
> {code}
> However, other smaller flow files are able to successfully go through the
> processor. Not sure if it's related to the size of the flow file or not.
> Perhaps there just needs to be a check around this line in Bin.java:142, to
> make sure we don't try to migrate a flow file to its own session:
> {code}
> session.migrate(getSession(), Collections.singleton(flowFile));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3020) LDAP - Support configurable user identity
[ https://issues.apache.org/jira/browse/NIFI-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-3020: --- Resolution: Fixed Status: Resolved (was: Patch Available) > LDAP - Support configurable user identity > - > > Key: NIFI-3020 > URL: https://issues.apache.org/jira/browse/NIFI-3020 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Gilman >Assignee: Matt Gilman > Fix For: 1.1.0 > > > The current LDAP provider supports a configurable search filter that will > allow the user specified login name to be matched against any LDAP entry > attribute. We should offer a configuration option that will indicate if we > should use the LDAP entry DN or if we should use the login name that was used > in the search filter. For instance, this would allow an admin to configure a > user to login with their sAMAccountName and subsequently use that name as > their user's identity. > Note: we should default this option to be the user DN in order to ensure > backwards compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-32) Swap File format should include summary at end for faster NiFi system startup
[ https://issues.apache.org/jira/browse/NIFI-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky resolved NIFI-32. -- Resolution: Resolved > Swap File format should include summary at end for faster NiFi system startup > - > > Key: NIFI-32 > URL: https://issues.apache.org/jira/browse/NIFI-32 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 1.1.0 > > > When NiFi is restarted with many hundreds or thousands of Swap Files, it can > take a very long time to startup. > If when we write out the Swap Files, we append a summary of the Swap File > (queue identifier, number of FlowFiles, size of FlowFiles, and ContentClaim > info/count map), we can then read far less data for restart. This means we > would serialize the summary to a ByteArrayOutputStream, then get the length, > and then write out the summary, followed by length. On restore, we read just > the last 8 bytes, and that allows us to generate the offset into the file to > start reading the summary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios
[ https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2854: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Enable repositories to support upgrades and rollback in well defined scenarios > -- > > Key: NIFI-2854 > URL: https://issues.apache.org/jira/browse/NIFI-2854 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.1.0 > > > The flowfile, swapfile, provenance, and content repositories play a very > important roll in NiFi's ability to be safely upgraded and rolled back. We > need to have well documented behaviors, designs, and version adherence so > that users can safely rely on these mechanisms. > Once this is formalized and in place we should update our versioning guidance > to reflect this as well. > The following would be true from NiFi 1.2.0 onward > * No changes to how the repositories are persisted to disk can be made which > will break forward/backward compatibility and specifically this means that > things like the way each is serialized to disk cannot change. > * If changes are made which impact forward or backward compatibility they > should be reserved for major releases only and should include a utility to > help users with pre-existing data convert from some older format to the newer > format. It may not be feasible to have rollback on major releases. > * The content repository should not be changed within a major release cycle > in any way that will harm forward or backward compatibility. > * The flow file repository can change in that new fields can be added to > existing write ahead log record types but no fields can be removed nor can > any new types be added. Once a field is considered required it must remain > required. Changes may only be made across minor version changes - not > incremental. > * Swap File storage should follow very similar rules to the flow file > repository. Adding a schema to the swap file header may allow some variation > there but the variation should only be hints to optimize how they're > processed and not change their behavior otherwise. Changes are only permitted > during minor version releases. > * Provenance repository changes are only permitted during minor version > releases. These changes may include adding or removing fields from existing > event types. If a field is considered required it must always be considered > required. If a field is removed then it must not be a required field and > there must be a sensible default an older version could use if that value is > not found in new data once rolled back. New event types may be added. > Fields or event types not known to older version, if seen after a rollback, > will simply be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-3059) ZooKeeper Migrator needs an option to ignore source check
[ https://issues.apache.org/jira/browse/NIFI-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky resolved NIFI-3059. Resolution: Fixed > ZooKeeper Migrator needs an option to ignore source check > - > > Key: NIFI-3059 > URL: https://issues.apache.org/jira/browse/NIFI-3059 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 1.1.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.1.0 > > > The ZooKeeper Migrator has a check that will prevent the user from writing to > the same ZK that the data came from. However, when upgrading from NiFi 0.x > which uses an external ZooKeeper to NiFi 1.x, and the same ZK will be used, > the nodes need to be migrated from user/password to a Kerberos principal. A > command-line option should be provided to ignore the source check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2950) Allow literal interpretation of hex numbers as whole numbers in EL
[ https://issues.apache.org/jira/browse/NIFI-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2950: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Allow literal interpretation of hex numbers as whole numbers in EL > -- > > Key: NIFI-2950 > URL: https://issues.apache.org/jira/browse/NIFI-2950 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Minor > Fix For: 1.1.0 > > > Currently in EL you cannot pass hex numbers as a whole number and instead > only decimals support them. Whole numbers should support being passed in as > hex too. > This was brought up by this SO[1] question. > [1] > http://stackoverflow.com/questions/40265791/applying-string-manipulations-mathematical-operations-to-the-contents-of-a-flow/40267507#40267507 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
