[jira] [Commented] (NIFI-5857) Non deterministic behaviour in Kubernetes by trying to inject custom properties
[ https://issues.apache.org/jira/browse/NIFI-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762441#comment-16762441 ] dirkjkb commented on NIFI-5857: --- Hi [~achristianson], by injecting the conf that way the problem seems to be solved when I tried it. Furthermore by calling scripts on scripts on scripts at run time we bring more complexity into a docker container. By just injecting the config we will improve the readability > Non deterministic behaviour in Kubernetes by trying to inject custom > properties > --- > > Key: NIFI-5857 > URL: https://issues.apache.org/jira/browse/NIFI-5857 > Project: Apache NiFi > Issue Type: Bug > Components: Docker >Affects Versions: 1.8.0 > Environment: Kubernetes, Docker >Reporter: dirkjkb >Priority: Critical > Attachments: > 0001-NIFI-5857-Non-deterministic-behaviour-in-Kubernetes-.patch > > > I want to override some config files in Nifi via Kubernetes. In order to do > so I am trying to replace the files after the start. It appears that the > docker file is started through a start.sh script which calls several other > scripts. This implementation Leeds to a non deterministic state, since the > replacement time can differ from the start.sh runtime. Furthermore, after > restarting a pod, the replacing command will be run each time again what also > leeds to a fuzzy state. > My proposal would be instead of injecting and running some sh files who will > set some variables the customized config files should just be copy replaced > in the building step. The run command can then be replaced through the > ENTRYPOINT ["bin/nifi.sh", "run"] Command. > In order to get the logging output to the console, a logback-test.xml file > should be created and configured so that all the meaningful information will > be piped to stdout. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5990) Move Developer docs section above Processors
[ https://issues.apache.org/jira/browse/NIFI-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5990: Resolution: Fixed Status: Resolved (was: Patch Available) > Move Developer docs section above Processors > > > Key: NIFI-5990 > URL: https://issues.apache.org/jira/browse/NIFI-5990 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 1.9.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The Developer section of the docs is hidden below Processors, we should move > it up to make it easier to find. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6002) Fix formatting on docs links
Andy LoPresto created NIFI-6002: --- Summary: Fix formatting on docs links Key: NIFI-6002 URL: https://issues.apache.org/jira/browse/NIFI-6002 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.8.0 Reporter: Andy LoPresto When reviewing NIFI-5990, I noticed some link CSS classes were probably duplicated via copy/paste, and the display value for "Rest Api" should be "REST API" as those are both acronyms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] alopresto commented on issue #3283: NIFI-5990 Moving Developer docs section above Processors
alopresto commented on issue #3283: NIFI-5990 Moving Developer docs section above Processors URL: https://github.com/apache/nifi/pull/3283#issuecomment-461301073 Looks like @joewitt took care of this. I noticed a couple (existing) issues with formatting, so I'll open a new Jira for that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5990) Move Developer docs section above Processors
[ https://issues.apache.org/jira/browse/NIFI-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762299#comment-16762299 ] ASF subversion and git services commented on NIFI-5990: --- Commit 8c58d518578c12fa335b6ecb092e50e355e7d0da in nifi's branch refs/heads/master from Bryan Bende [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8c58d51 ] NIFI-5990 This closes #3283. Moving Developer docs section above Processors Signed-off-by: joewitt > Move Developer docs section above Processors > > > Key: NIFI-5990 > URL: https://issues.apache.org/jira/browse/NIFI-5990 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The Developer section of the docs is hidden below Processors, we should move > it up to make it easier to find. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3283: NIFI-5990 Moving Developer docs section above Processors
asfgit closed pull request #3283: NIFI-5990 Moving Developer docs section above Processors URL: https://github.com/apache/nifi/pull/3283 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5968) Add standard HTTP security headers
[ https://issues.apache.org/jira/browse/NIFI-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762256#comment-16762256 ] ASF subversion and git services commented on NIFI-5968: --- Commit f81d6bd63b50c27dc62aabb85a6d864db991c9dd in nifi's branch refs/heads/master from thenatog [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f81d6bd ] NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security HTTP headers using Jetty Filters. Added some tests. Removed bad test. Refactored filter creation method. Ensure HSTS header is only applied if NiFi is secured with HTTPS Small changes to header array list. Fixed checkstyle errors. This closes #3273. Signed-off-by: Andy LoPresto > Add standard HTTP security headers > -- > > Key: NIFI-5968 > URL: https://issues.apache.org/jira/browse/NIFI-5968 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Some HTTP security headers could be added to improve NiFi security stance. > These include: Strict-Transport-Security (HSTS), X-XSS-Protection, and > Content-Security-Policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5968) Add standard HTTP security headers
[ https://issues.apache.org/jira/browse/NIFI-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5968: Component/s: Core Framework > Add standard HTTP security headers > -- > > Key: NIFI-5968 > URL: https://issues.apache.org/jira/browse/NIFI-5968 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Labels: headers, http, security > Fix For: 1.9.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Some HTTP security headers could be added to improve NiFi security stance. > These include: Strict-Transport-Security (HSTS), X-XSS-Protection, and > Content-Security-Policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5968) Add standard HTTP security headers
[ https://issues.apache.org/jira/browse/NIFI-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5968: Resolution: Fixed Status: Resolved (was: Patch Available) > Add standard HTTP security headers > -- > > Key: NIFI-5968 > URL: https://issues.apache.org/jira/browse/NIFI-5968 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Labels: headers, http, security > Fix For: 1.9.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Some HTTP security headers could be added to improve NiFi security stance. > These include: Strict-Transport-Security (HSTS), X-XSS-Protection, and > Content-Security-Policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5968) Add standard HTTP security headers
[ https://issues.apache.org/jira/browse/NIFI-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5968: Labels: headers http security (was: ) Fix Version/s: 1.9.0 Affects Version/s: 1.8.0 Status: Patch Available (was: Open) > Add standard HTTP security headers > -- > > Key: NIFI-5968 > URL: https://issues.apache.org/jira/browse/NIFI-5968 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Labels: security, http, headers > Fix For: 1.9.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Some HTTP security headers could be added to improve NiFi security stance. > These include: Strict-Transport-Security (HSTS), X-XSS-Protection, and > Content-Security-Policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] alopresto commented on issue #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security …
alopresto commented on issue #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security … URL: https://github.com/apache/nifi/pull/3273#issuecomment-461240942 There were some checkstyle problems on unused imports, but I fixed those. Ran `contrib-check` and all tests pass. +1, merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security …
asfgit closed pull request #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security … URL: https://github.com/apache/nifi/pull/3273 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-5983) recordReader parse problems in PutDatabaseRecord: flowfiles not transferred to failure relationship
[ https://issues.apache.org/jira/browse/NIFI-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5983: --- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > recordReader parse problems in PutDatabaseRecord: flowfiles not transferred > to failure relationship > --- > > Key: NIFI-5983 > URL: https://issues.apache.org/jira/browse/NIFI-5983 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Endre Kovacs >Assignee: Endre Kovacs >Priority: Minor > Fix For: 1.9.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > When using PutDatabaseRecord, parse problems in record reader (Avro, CSV, but > possibly others too) should cause the flowfiles to transfer to failure > relationship, however, they are instead session rollbacked. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5983) recordReader parse problems in PutDatabaseRecord: flowfiles not transferred to failure relationship
[ https://issues.apache.org/jira/browse/NIFI-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762137#comment-16762137 ] ASF subversion and git services commented on NIFI-5983: --- Commit 24a7d480c8fde18a7dd64d7de80812d18eb2c5a4 in nifi's branch refs/heads/master from Endre Zoltan Kovacs [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=24a7d48 ] NIFI-5983: handling parse problems in recordReader implementations Fixed Checkstyle violation Signed-off-by: Matthew Burgess This closes #3282 > recordReader parse problems in PutDatabaseRecord: flowfiles not transferred > to failure relationship > --- > > Key: NIFI-5983 > URL: https://issues.apache.org/jira/browse/NIFI-5983 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Endre Kovacs >Assignee: Endre Kovacs >Priority: Minor > Time Spent: 3.5h > Remaining Estimate: 0h > > When using PutDatabaseRecord, parse problems in record reader (Avro, CSV, but > possibly others too) should cause the flowfiles to transfer to failure > relationship, however, they are instead session rollbacked. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3282: NIFI-5983: handling parse problems in recordReader implementations
asfgit closed pull request #3282: NIFI-5983: handling parse problems in recordReader implementations URL: https://github.com/apache/nifi/pull/3282 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mattyb149 commented on issue #3282: NIFI-5983: handling parse problems in recordReader implementations
mattyb149 commented on issue #3282: NIFI-5983: handling parse problems in recordReader implementations URL: https://github.com/apache/nifi/pull/3282#issuecomment-461184016 +1 LGTM, ran full unit and integration tests, and tried on a live NiFi instance with Avro and CSV data. I also tried with the JsonTreeReader and invalid JSON, they appear to throw a checked exception so the error was already handled correctly, no need to add these fixes to that reader ATM. Thanks for the fix! Merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5435) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/NIFI-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761982#comment-16761982 ] Joseph Witt commented on NIFI-5435: --- removed fix version for now. once we're making progress on review/merge we can pin the fix version. that said - this is awesome and I really hope we get traction on it soon! > Prometheus /metrics http endpoint for monitoring integration > > > Key: NIFI-5435 > URL: https://issues.apache.org/jira/browse/NIFI-5435 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration, Core Framework, Extensions, Tools and > Build >Affects Versions: 1.5.0 >Reporter: Hari Sekhon >Assignee: Satwik Bhandiwad >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5774) Refresh available component types on the front-end
[ https://issues.apache.org/jira/browse/NIFI-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5774: -- Fix Version/s: (was: 1.9.0) > Refresh available component types on the front-end > -- > > Key: NIFI-5774 > URL: https://issues.apache.org/jira/browse/NIFI-5774 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Priority: Minor > > Currently the application retrieves the available processors, controller > services, and reporting tasks during initial page load and caches them on the > client. This was fine because the types could never change without restarting > the application, but now NIFI-5673 introduces the ability to dynamically load > new NARs without restarting, so we'll need a way to reload the types on the > front end. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5774) Refresh available component types on the front-end
[ https://issues.apache.org/jira/browse/NIFI-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761984#comment-16761984 ] Joseph Witt commented on NIFI-5774: --- i have removed fix version for now. once we have pr/review/merge traction lets resolve. > Refresh available component types on the front-end > -- > > Key: NIFI-5774 > URL: https://issues.apache.org/jira/browse/NIFI-5774 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Priority: Minor > > Currently the application retrieves the available processors, controller > services, and reporting tasks during initial page load and caches them on the > client. This was fine because the types could never change without restarting > the application, but now NIFI-5673 introduces the ability to dynamically load > new NARs without restarting, so we'll need a way to reload the types on the > front end. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5435) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/NIFI-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5435: -- Fix Version/s: (was: 1.9.0) > Prometheus /metrics http endpoint for monitoring integration > > > Key: NIFI-5435 > URL: https://issues.apache.org/jira/browse/NIFI-5435 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration, Core Framework, Extensions, Tools and > Build >Affects Versions: 1.5.0 >Reporter: Hari Sekhon >Assignee: Satwik Bhandiwad >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5940) Cluster Node Offload Hangs if any RPG on flow is Disabled
[ https://issues.apache.org/jira/browse/NIFI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5940: - Status: Patch Available (was: Open) > Cluster Node Offload Hangs if any RPG on flow is Disabled > - > > Key: NIFI-5940 > URL: https://issues.apache.org/jira/browse/NIFI-5940 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Major > Fix For: 1.9.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If any Remote Process Group on the flow is disabled when a user starts a node > Offload, then offload fails. > This is because the Offload process tries to turn off all Remote Process > Group's, even if they are already disabled. > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Offload Flow > Files from Node,5,main]: java.lang.IllegalStateException: > 33a4935b-5800-360d-9250-2179e3ef5efe is not transmitting > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi > java.lang.IllegalStateException: 33a4935b-5800-360d-9250-2179e3ef5efe is not > transmitting > at > org.apache.nifi.remote.StandardRemoteProcessGroup.verifyCanStopTransmitting(StandardRemoteProcessGroup.java:1333) > at > org.apache.nifi.remote.StandardRemoteProcessGroup.stopTransmitting(StandardRemoteProcessGroup.java:1036) > at java.util.ArrayList.forEach(ArrayList.java:1249) > at > org.apache.nifi.controller.StandardFlowService.offload(StandardFlowService.java:706) > at > org.apache.nifi.controller.StandardFlowService.handleOffloadRequest(StandardFlowService.java:688) > at > org.apache.nifi.controller.StandardFlowService.access$400(StandardFlowService.java:105) > at > org.apache.nifi.controller.StandardFlowService$3.run(StandardFlowService.java:428) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5940) Cluster Node Offload Hangs if any RPG on flow is Disabled
[ https://issues.apache.org/jira/browse/NIFI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5940: - Resolution: Fixed Status: Resolved (was: Patch Available) > Cluster Node Offload Hangs if any RPG on flow is Disabled > - > > Key: NIFI-5940 > URL: https://issues.apache.org/jira/browse/NIFI-5940 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Major > Fix For: 1.9.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If any Remote Process Group on the flow is disabled when a user starts a node > Offload, then offload fails. > This is because the Offload process tries to turn off all Remote Process > Group's, even if they are already disabled. > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Offload Flow > Files from Node,5,main]: java.lang.IllegalStateException: > 33a4935b-5800-360d-9250-2179e3ef5efe is not transmitting > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi > java.lang.IllegalStateException: 33a4935b-5800-360d-9250-2179e3ef5efe is not > transmitting > at > org.apache.nifi.remote.StandardRemoteProcessGroup.verifyCanStopTransmitting(StandardRemoteProcessGroup.java:1333) > at > org.apache.nifi.remote.StandardRemoteProcessGroup.stopTransmitting(StandardRemoteProcessGroup.java:1036) > at java.util.ArrayList.forEach(ArrayList.java:1249) > at > org.apache.nifi.controller.StandardFlowService.offload(StandardFlowService.java:706) > at > org.apache.nifi.controller.StandardFlowService.handleOffloadRequest(StandardFlowService.java:688) > at > org.apache.nifi.controller.StandardFlowService.access$400(StandardFlowService.java:105) > at > org.apache.nifi.controller.StandardFlowService$3.run(StandardFlowService.java:428) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] thenatog commented on issue #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security …
thenatog commented on issue #3273: NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security … URL: https://github.com/apache/nifi/pull/3273#issuecomment-461117451 @alopresto How are we looking for this one now? Are there any remaining changes I need to make? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5940) Cluster Node Offload Hangs if any RPG on flow is Disabled
[ https://issues.apache.org/jira/browse/NIFI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761971#comment-16761971 ] ASF subversion and git services commented on NIFI-5940: --- Commit 35147a620f2ae4e2164ba342e77976667837d5f3 in nifi's branch refs/heads/master from Peter Wicks [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=35147a6 ] NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled This closes #3255 Signed-off-by: Mark Payne > Cluster Node Offload Hangs if any RPG on flow is Disabled > - > > Key: NIFI-5940 > URL: https://issues.apache.org/jira/browse/NIFI-5940 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Major > Fix For: 1.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > If any Remote Process Group on the flow is disabled when a user starts a node > Offload, then offload fails. > This is because the Offload process tries to turn off all Remote Process > Group's, even if they are already disabled. > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Offload Flow > Files from Node,5,main]: java.lang.IllegalStateException: > 33a4935b-5800-360d-9250-2179e3ef5efe is not transmitting > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi > java.lang.IllegalStateException: 33a4935b-5800-360d-9250-2179e3ef5efe is not > transmitting > at > org.apache.nifi.remote.StandardRemoteProcessGroup.verifyCanStopTransmitting(StandardRemoteProcessGroup.java:1333) > at > org.apache.nifi.remote.StandardRemoteProcessGroup.stopTransmitting(StandardRemoteProcessGroup.java:1036) > at java.util.ArrayList.forEach(ArrayList.java:1249) > at > org.apache.nifi.controller.StandardFlowService.offload(StandardFlowService.java:706) > at > org.apache.nifi.controller.StandardFlowService.handleOffloadRequest(StandardFlowService.java:688) > at > org.apache.nifi.controller.StandardFlowService.access$400(StandardFlowService.java:105) > at > org.apache.nifi.controller.StandardFlowService$3.run(StandardFlowService.java:428) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] markap14 commented on issue #3255: NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled
markap14 commented on issue #3255: NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled URL: https://github.com/apache/nifi/pull/3255#issuecomment-461117215 Thanks @patricker! Definitely a good catch and happy to see it make it in for 1.9.0. I have merged this to master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3255: NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled
asfgit closed pull request #3255: NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled URL: https://github.com/apache/nifi/pull/3255 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kevdoran commented on issue #3291: NIFI-5859 Add nifi-api dependency to nifi-jetty-bundle pom so the NAR…
kevdoran commented on issue #3291: NIFI-5859 Add nifi-api dependency to nifi-jetty-bundle pom so the NAR… URL: https://github.com/apache/nifi/pull/3291#issuecomment-461112856 Will review... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bbende commented on issue #7: NIFI-5859: Build NAR Extension Definitions/docs at build time
bbende commented on issue #7: NIFI-5859: Build NAR Extension Definitions/docs at build time URL: https://github.com/apache/nifi-maven/pull/7#issuecomment-461092692 @markap14 I have created a new PR that includes your work + some additional improvements and clean up: https://github.com/apache/nifi-maven/pull/8 If you are good with those changes then I think we can merge this. Let me know. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bbende opened a new pull request #8: NIFI-5859 - Build NAR Extension Definitions/docs at build time
bbende opened a new pull request #8: NIFI-5859 - Build NAR Extension Definitions/docs at build time URL: https://github.com/apache/nifi-maven/pull/8 This PR includes the work from https://github.com/apache/nifi-maven/pull/7 with some additional improvements. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5857) Non deterministic behaviour in Kubernetes by trying to inject custom properties
[ https://issues.apache.org/jira/browse/NIFI-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761880#comment-16761880 ] Andrew Christianson commented on NIFI-5857: --- Hi [~dirkjkb], I took a look at this ticket & your patch. It seems the nondeterminism/fuzzy state stems from conf/ having a mix of stateless configs and stateful. I believe that nifi properties (all the prop_replace) is not leading to the nondeterminism and should probably stay. It may not be perfect, but I don't think it's the root cause of this issue. Could you update your patch to keep the prop_replace and instead put the actually dynamic configs into volumes? Some things (e.g. authorized users) do change dynamically at runtime, so a volume is appropriate. One thing we need to look into as part of this patch is whether there is a clean separation between the stateless config and that which is updated at runtime. Overall, it does seem there is room for improvement in the Dockerfile/config management here. I'm just not sure it warrants fully taking out the prop_replace code. > Non deterministic behaviour in Kubernetes by trying to inject custom > properties > --- > > Key: NIFI-5857 > URL: https://issues.apache.org/jira/browse/NIFI-5857 > Project: Apache NiFi > Issue Type: Bug > Components: Docker >Affects Versions: 1.8.0 > Environment: Kubernetes, Docker >Reporter: dirkjkb >Priority: Critical > Attachments: > 0001-NIFI-5857-Non-deterministic-behaviour-in-Kubernetes-.patch > > > I want to override some config files in Nifi via Kubernetes. In order to do > so I am trying to replace the files after the start. It appears that the > docker file is started through a start.sh script which calls several other > scripts. This implementation Leeds to a non deterministic state, since the > replacement time can differ from the start.sh runtime. Furthermore, after > restarting a pod, the replacing command will be run each time again what also > leeds to a fuzzy state. > My proposal would be instead of injecting and running some sh files who will > set some variables the customized config files should just be copy replaced > in the building step. The run command can then be replaced through the > ENTRYPOINT ["bin/nifi.sh", "run"] Command. > In order to get the logging output to the console, a logback-test.xml file > should be created and configured so that all the meaningful information will > be piped to stdout. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6001) Nested versioned PGs can cause CS references to be incorrect on import
Bryan Bende created NIFI-6001: - Summary: Nested versioned PGs can cause CS references to be incorrect on import Key: NIFI-6001 URL: https://issues.apache.org/jira/browse/NIFI-6001 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.7.1, 1.8.0, 1.6.0, 1.5.0 Reporter: Bryan Bende Steps to reproduce... 1) Create a PG named PG1 2) Inside PG1 create another PG named PG2 3) Inside PG2 create two controller services where one service references the other, example: CsvReader referencing AvroSchemaRegistry 4) Start version control on PG2 5) Start version control on PG1 6) On the root canvas, create a new PG and import PG1 At this point if you go into the second instance of PG1 and look at the services inside the second instance of PG2, the CsvReader no longer has the correct reference to the AvroSchemaRegistry and says "Incompatible Controller Service Configured". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5786) Add nifi-hazelcast-bundle processor to communicate with Hazlecast
[ https://issues.apache.org/jira/browse/NIFI-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5786: -- Fix Version/s: (was: 1.9.0) > Add nifi-hazelcast-bundle processor to communicate with Hazlecast > -- > > Key: NIFI-5786 > URL: https://issues.apache.org/jira/browse/NIFI-5786 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.0 >Reporter: Milan Das >Priority: Major > > I have developed nifi-hazelcast-bundle processor to talk to hazelcast server. > Would like to contribute to NIFI. > It contains following: > # One controller service : This creates a client connection to Hazelcast > server . > # PutHazlecastMap --> Put data to Hazlecast Map (IMap, ReplicatedMap or > MultipleMap. Drop down selection). Dependent on > HazelcastClientControllerService > ## It expects hazle.key as attribute for key > ## Flow file is considered as Value for the Map. > # GetHazlecastMap --> Get data from Hazlecast Map (IMap, ReplicatedMap or > MultipleMap. Drop down selection). It accepts Map name > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5940) Cluster Node Offload Hangs if any RPG on flow is Disabled
[ https://issues.apache.org/jira/browse/NIFI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5940: -- Fix Version/s: 1.9.0 > Cluster Node Offload Hangs if any RPG on flow is Disabled > - > > Key: NIFI-5940 > URL: https://issues.apache.org/jira/browse/NIFI-5940 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Major > Fix For: 1.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > If any Remote Process Group on the flow is disabled when a user starts a node > Offload, then offload fails. > This is because the Offload process tries to turn off all Remote Process > Group's, even if they are already disabled. > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Offload Flow > Files from Node,5,main]: java.lang.IllegalStateException: > 33a4935b-5800-360d-9250-2179e3ef5efe is not transmitting > 2019-01-09 17:22:00,823 ERROR [Offload Flow Files from Node] > org.apache.nifi.NiFi > java.lang.IllegalStateException: 33a4935b-5800-360d-9250-2179e3ef5efe is not > transmitting > at > org.apache.nifi.remote.StandardRemoteProcessGroup.verifyCanStopTransmitting(StandardRemoteProcessGroup.java:1333) > at > org.apache.nifi.remote.StandardRemoteProcessGroup.stopTransmitting(StandardRemoteProcessGroup.java:1036) > at java.util.ArrayList.forEach(ArrayList.java:1249) > at > org.apache.nifi.controller.StandardFlowService.offload(StandardFlowService.java:706) > at > org.apache.nifi.controller.StandardFlowService.handleOffloadRequest(StandardFlowService.java:688) > at > org.apache.nifi.controller.StandardFlowService.access$400(StandardFlowService.java:105) > at > org.apache.nifi.controller.StandardFlowService$3.run(StandardFlowService.java:428) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5857) Non deterministic behaviour in Kubernetes by trying to inject custom properties
[ https://issues.apache.org/jira/browse/NIFI-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5857: -- Fix Version/s: (was: 1.9.0) > Non deterministic behaviour in Kubernetes by trying to inject custom > properties > --- > > Key: NIFI-5857 > URL: https://issues.apache.org/jira/browse/NIFI-5857 > Project: Apache NiFi > Issue Type: Bug > Components: Docker >Affects Versions: 1.8.0 > Environment: Kubernetes, Docker >Reporter: dirkjkb >Priority: Critical > Attachments: > 0001-NIFI-5857-Non-deterministic-behaviour-in-Kubernetes-.patch > > > I want to override some config files in Nifi via Kubernetes. In order to do > so I am trying to replace the files after the start. It appears that the > docker file is started through a start.sh script which calls several other > scripts. This implementation Leeds to a non deterministic state, since the > replacement time can differ from the start.sh runtime. Furthermore, after > restarting a pod, the replacing command will be run each time again what also > leeds to a fuzzy state. > My proposal would be instead of injecting and running some sh files who will > set some variables the customized config files should just be copy replaced > in the building step. The run command can then be replaced through the > ENTRYPOINT ["bin/nifi.sh", "run"] Command. > In order to get the logging output to the console, a logback-test.xml file > should be created and configured so that all the meaningful information will > be piped to stdout. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5938) Allow Record Readers to Infer Schema on Read
[ https://issues.apache.org/jira/browse/NIFI-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761851#comment-16761851 ] Joseph Witt commented on NIFI-5938: --- [~markap14] [~mattyb149]is this close to merging? > Allow Record Readers to Infer Schema on Read > > > Key: NIFI-5938 > URL: https://issues.apache.org/jira/browse/NIFI-5938 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The introduction of record-oriented processors was a huge improvement for > NiFi in terms of usability. However, they only improve usability if you have > a schema for your data. There have been several comments along the lines of > "I would really love to use the record-oriented processors, but I don't have > a schema for my data." > Sometimes users have no schema because they don't want to bother with > creating the schemas. The schema becomes a usability issue. This is > especially true for very large documents that contain a lot of nested > Records. Other times, users cannot create a schema because they retrieve > arbitrary data from some source, and they have no idea what the data will > look like. > We do not want to remove the notion of a schema, however. Schemas provide for > a very powerful construct for many use cases, and it provides Processors a > much easier-to-use API. If we provide the ability to Infer the Schema on > Read, though, we can provide the best of both worlds. While we do have > processors for inferring schemas for JSON and CSV data, those are not always > sufficient. They cannot be used, for instance, by ConsumeKafkaRecord, > ExecuteSQL, etc. because those Processors need the schema before that. > Additionally, we have no ability to infer a schema for XML, logs, etc. > Finally, we need to consider processors that are designed to manipulate the > data. For example, UpdateRecord, JoltTransformRecord, LookupRecord (when used > for enrichment), and QueryRecord. These Processors follow a typical pattern > of "get reader's schema, then provide it to the writer in order to get > writer's schema." This means that if the Record Writer inherits the record's > schema, and we infer that schema, then any newly added fields will simply be > dropped by the writer because the writer's schema doesn't know about those > fields. As a result, we need to ensure that we first transform the first > record, get the schema for the transformed record, and then pass that > transformed record's schema to the Writer, so that the Writer inherits the > schema describing data after transformation. > Design/Implementation Goals should include: > - High performance: users should be impacted as little as is feasible. > - Usability: users should be able to infer schemas with as little > configuration as is reasonable. > - Ease of Development: code should be written in a way that makes it easy for > new Record Readers to provide schema inference that is fast, efficient, > correct, and consistent with how the other readers infer schemas. > - Implementations: At a minimum, we should provide the ability to infer > schemas for JSON, XML, and CSV data. > - Backward Compatibility: The new feature should not break backward > compatibility for any Record Reader. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5929) Support for IBM MQ multi-instance queue managers (brokers)
[ https://issues.apache.org/jira/browse/NIFI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761849#comment-16761849 ] Joseph Witt commented on NIFI-5929: --- looks like a great contrib and review is going on but not clear it is close to merge based on latest testing required comment. The fix version can be applied once it is about to land. > Support for IBM MQ multi-instance queue managers (brokers) > -- > > Key: NIFI-5929 > URL: https://issues.apache.org/jira/browse/NIFI-5929 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.6.0, 0.7.0, 0.6.1, 1.1.0, 0.7.1, 1.2.0, 1.1.1, > 1.0.1, 1.3.0, 1.4.0, 0.7.4, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1 >Reporter: Veli Kerim Celik >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently connections provided by JMSConnectionFactoryProvider controller > service can connect to just a single IBM MQ queue manager. This is > problematic when the queue manager is part of a [multi-instance queue > manager|https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.con.doc/q018140_.html] > setup and goes from active to standby. > The goal of this issue is to support multiple queue managers, detect > standby/broken instance and switch to active instance. This behavior is > already implemented in [official Java > library|https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.1.0/com.ibm.mq.javadoc.doc/WMQJMSClasses/com/ibm/mq/jms/MQConnectionFactory.html#setConnectionNameList_java.lang.String_] > and should be leveraged. > Syntax used to specify multiple queue managers: myhost01(1414),myhost02(1414) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5929) Support for IBM MQ multi-instance queue managers (brokers)
[ https://issues.apache.org/jira/browse/NIFI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5929: -- Fix Version/s: (was: 1.9.0) > Support for IBM MQ multi-instance queue managers (brokers) > -- > > Key: NIFI-5929 > URL: https://issues.apache.org/jira/browse/NIFI-5929 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.6.0, 0.7.0, 0.6.1, 1.1.0, 0.7.1, 1.2.0, 1.1.1, > 1.0.1, 1.3.0, 1.4.0, 0.7.4, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1 >Reporter: Veli Kerim Celik >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently connections provided by JMSConnectionFactoryProvider controller > service can connect to just a single IBM MQ queue manager. This is > problematic when the queue manager is part of a [multi-instance queue > manager|https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.con.doc/q018140_.html] > setup and goes from active to standby. > The goal of this issue is to support multiple queue managers, detect > standby/broken instance and switch to active instance. This behavior is > already implemented in [official Java > library|https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.1.0/com.ibm.mq.javadoc.doc/WMQJMSClasses/com/ibm/mq/jms/MQConnectionFactory.html#setConnectionNameList_java.lang.String_] > and should be leveraged. > Syntax used to specify multiple queue managers: myhost01(1414),myhost02(1414) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5994) NiFi keyboard shortcuts
[ https://issues.apache.org/jira/browse/NIFI-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5994: -- Fix Version/s: (was: 1.9.0) > NiFi keyboard shortcuts > --- > > Key: NIFI-5994 > URL: https://issues.apache.org/jira/browse/NIFI-5994 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.8.0 >Reporter: Nimrod Avni >Priority: Minor > Labels: keyboard, nifi, shortcuts, ui > > NiFi is a very easy tool to use with a mouse, but I think for a lot of > developers who use NiFI, we have a better control with the keyboard and i > think it would be great for a lot of developers to have keyboard shortcuts to > use when developing over the canvas > * CTRL-Z and ALT-CTRL-Z feature, i.e to reverse actions we did (like moving > a component, creating a component, deleting a component, configuring a > component and starting/stopping a component) > * To have an option to start typing the name of the component we want to > hover and we will hover over the first component that its name matches what > we typed (if i have a GenerateFlowfile processor in my current process group, > i could start typing "Gene" and i will hover over it). this can work with > processors/process groups or basically anything with a name > * Easier traversal through the canvas, have shortcuts to go back to the > father process group, the root process group and so on > * shortcuts to create components (for example CTRL-P to open the add > processor tab and so on) > These are some of the features that will make NiFi even smoother and easier > to use, and i hope they can be implemented -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5993) NiFi text search improvment
[ https://issues.apache.org/jira/browse/NIFI-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5993: -- Fix Version/s: (was: 1.9.0) > NiFi text search improvment > --- > > Key: NIFI-5993 > URL: https://issues.apache.org/jira/browse/NIFI-5993 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.8.0 >Reporter: Nimrod Avni >Priority: Minor > Labels: nifi, search, ui > > The NiFi textual search on all the components in the canvas has some big > issues > The searching refreshes every time you type in a new letter, since the NiFi > canvas can get very big quickly, the searching takes up a lot of time and > resources. I had more then a few times where i typed in 2 letters and made my > chrome tab crash. I had to resort to typing what i want to search in a > notepad and copying it to to search box to avoid it searching every time i > type in a new letter. > I think it can be resolved by better textual indexing or potentially making > the search box do an "on demand" search, where the user clicks the search > button or the enter key to start the search. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5989) Improve PutKudu BatchSize handling
[ https://issues.apache.org/jira/browse/NIFI-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5989: --- Affects Version/s: (was: 1.8.0) Status: Patch Available (was: Open) > Improve PutKudu BatchSize handling > -- > > Key: NIFI-5989 > URL: https://issues.apache.org/jira/browse/NIFI-5989 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Alex Goos >Priority: Major > Labels: kudu, nifi > Fix For: 1.9.0 > > Attachments: > 0001-NIFI-5989-PutKudu-Additional-FF-Queue-length-setting.patch > > > Current "Batch size" property of PutKudu affects both: the number of > Flowfiles pulled per OnTrigger and the size of the Kudu client modification > buffer. > If the Flowfiles contain a considerable amount of records, then a > disproportionate amount of data is pulled in and deserialized into memory, > when in AUTO_FLUSH_BACKGROUND mode. > We propose introducing a separate setting for the batch size of FlowFiles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-6000) ConvertAvroToORC fails to process Avro type null and rollback instead of transferring the flowfile to failure.
[ https://issues.apache.org/jira/browse/NIFI-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761638#comment-16761638 ] Aleksandr Salatich commented on NIFI-6000: -- Hi, I would like to take it > ConvertAvroToORC fails to process Avro type null and rollback instead of > transferring the flowfile to failure. > -- > > Key: NIFI-6000 > URL: https://issues.apache.org/jira/browse/NIFI-6000 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sujesh Menon >Priority: Major > Labels: easyfix > Attachments: AvroToORC_test.xml > > > The ConvertAvroToORC processor throws an IllegalArgumentException when the > input avro data has null types or empty arrays. > ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055] > ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055] failed to process > session due to java.lang.IllegalArgumentException: Did not recognize Avro > type null; Processor Administratively Yielded for 1 sec: > java.lang.IllegalArgumentException: Did not recognize Avro type null > java.lang.IllegalArgumentException: Did not recognize Avro type null > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:295) > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$getOrcField$11(NiFiOrcUtils.java:284) > at java.util.ArrayList.forEach(Unknown Source) > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:281) > at > org.apache.nifi.processors.hive.ConvertAvroToORC.lambda$onTrigger$0(ConvertAvroToORC.java:217) > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2910) > at > org.apache.nifi.processors.hive.ConvertAvroToORC.onTrigger(ConvertAvroToORC.java:209) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > > > The flowfile is not transfered to failure as the > https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L286 > only catches ProcessException but > https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L217 > Throws IllegalArgumentException when the fieldSchema is anything other than > hive primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
