[GitHub] nifi issue #3055: NIFI-5600: Fixing columns in queue listing and component s...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3055 LGTM, built the PR and tested the change, working as expected, merging to master. Thanks @mcgilman ---
[jira] [Commented] (NIFI-5600) Display the node on which flowfiles are located in queues
[ https://issues.apache.org/jira/browse/NIFI-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644567#comment-16644567 ] ASF GitHub Bot commented on NIFI-5600: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3055 LGTM, built the PR and tested the change, working as expected, merging to master. Thanks @mcgilman > Display the node on which flowfiles are located in queues > - > > Key: NIFI-5600 > URL: https://issues.apache.org/jira/browse/NIFI-5600 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Affects Versions: 1.7.0, 1.7.1 >Reporter: Jeff Storck >Assignee: Pierre Villard >Priority: Major > > When displaying a flowfile queue in the UI, the user should be able to see on > which node a flowfile is queued. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3055: NIFI-5600: Fixing columns in queue listing and comp...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3055 ---
[jira] [Commented] (NIFI-5600) Display the node on which flowfiles are located in queues
[ https://issues.apache.org/jira/browse/NIFI-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644572#comment-16644572 ] ASF GitHub Bot commented on NIFI-5600: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3055 > Display the node on which flowfiles are located in queues > - > > Key: NIFI-5600 > URL: https://issues.apache.org/jira/browse/NIFI-5600 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Affects Versions: 1.7.0, 1.7.1 >Reporter: Jeff Storck >Assignee: Matt Gilman >Priority: Major > Fix For: 1.8.0 > > > When displaying a flowfile queue in the UI, the user should be able to see on > which node a flowfile is queued. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5600) Display the node on which flowfiles are located in queues
[ https://issues.apache.org/jira/browse/NIFI-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5600: - Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Display the node on which flowfiles are located in queues > - > > Key: NIFI-5600 > URL: https://issues.apache.org/jira/browse/NIFI-5600 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Affects Versions: 1.7.0, 1.7.1 >Reporter: Jeff Storck >Assignee: Matt Gilman >Priority: Major > Fix For: 1.8.0 > > > When displaying a flowfile queue in the UI, the user should be able to see on > which node a flowfile is queued. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5600) Display the node on which flowfiles are located in queues
[ https://issues.apache.org/jira/browse/NIFI-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard reassigned NIFI-5600: Assignee: Matt Gilman (was: Pierre Villard) > Display the node on which flowfiles are located in queues > - > > Key: NIFI-5600 > URL: https://issues.apache.org/jira/browse/NIFI-5600 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Affects Versions: 1.7.0, 1.7.1 >Reporter: Jeff Storck >Assignee: Matt Gilman >Priority: Major > Fix For: 1.8.0 > > > When displaying a flowfile queue in the UI, the user should be able to see on > which node a flowfile is queued. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3059: NIFI-5676: Fix a timezone-dependent test in PutORCTest
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3059 +1, merging to master, thanks @kotarot ---
[jira] [Commented] (NIFI-5676) A test in PutORCTest fails because it depends on timezone
[
https://issues.apache.org/jira/browse/NIFI-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644575#comment-16644575
]
ASF GitHub Bot commented on NIFI-5676:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3059
+1, merging to master, thanks @kotarot
> A test in PutORCTest fails because it depends on timezone
> -
>
> Key: NIFI-5676
> URL: https://issues.apache.org/jira/browse/NIFI-5676
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Priority: Major
>
> Test {{testWriteORCWithAvroLogicalTypes}} fails because it depends on
> timezone. It seems that, in this test, the converted date in ORC is "1 day"
> before the input date when the timezone is ahead of GMT. For example, when
> the timezone is "GMT+9", this test fails as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.orc.PutORCTest
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.694
> s <<< FAILURE! - in org.apache.nifi.processors.orc.PutORCTest
> [ERROR]
> testWriteORCWithAvroLogicalTypes(org.apache.nifi.processors.orc.PutORCTest)
> Time elapsed: 0.146 s <<< FAILURE!
> java.lang.AssertionError: expected:<17813> but was:<17812>
> at
> org.apache.nifi.processors.orc.PutORCTest.lambda$testWriteORCWithAvroLogicalTypes$1(PutORCTest.java:250)
> at
> org.apache.nifi.processors.orc.PutORCTest.verifyORCUsers(PutORCTest.java:408)
> at
> org.apache.nifi.processors.orc.PutORCTest.testWriteORCWithAvroLogicalTypes(PutORCTest.java:246)
> {noformat}
> To fix this, we can set the timezone information in the assertion as we did
> in [NIFI-3718|https://issues.apache.org/jira/browse/NIFI-3718].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3059: NIFI-5676: Fix a timezone-dependent test in PutORCT...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3059 ---
[jira] [Commented] (NIFI-5676) A test in PutORCTest fails because it depends on timezone
[
https://issues.apache.org/jira/browse/NIFI-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644577#comment-16644577
]
ASF GitHub Bot commented on NIFI-5676:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3059
> A test in PutORCTest fails because it depends on timezone
> -
>
> Key: NIFI-5676
> URL: https://issues.apache.org/jira/browse/NIFI-5676
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Priority: Major
>
> Test {{testWriteORCWithAvroLogicalTypes}} fails because it depends on
> timezone. It seems that, in this test, the converted date in ORC is "1 day"
> before the input date when the timezone is ahead of GMT. For example, when
> the timezone is "GMT+9", this test fails as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.orc.PutORCTest
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.694
> s <<< FAILURE! - in org.apache.nifi.processors.orc.PutORCTest
> [ERROR]
> testWriteORCWithAvroLogicalTypes(org.apache.nifi.processors.orc.PutORCTest)
> Time elapsed: 0.146 s <<< FAILURE!
> java.lang.AssertionError: expected:<17813> but was:<17812>
> at
> org.apache.nifi.processors.orc.PutORCTest.lambda$testWriteORCWithAvroLogicalTypes$1(PutORCTest.java:250)
> at
> org.apache.nifi.processors.orc.PutORCTest.verifyORCUsers(PutORCTest.java:408)
> at
> org.apache.nifi.processors.orc.PutORCTest.testWriteORCWithAvroLogicalTypes(PutORCTest.java:246)
> {noformat}
> To fix this, we can set the timezone information in the assertion as we did
> in [NIFI-3718|https://issues.apache.org/jira/browse/NIFI-3718].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5676) A test in PutORCTest fails because it depends on timezone
[
https://issues.apache.org/jira/browse/NIFI-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5676:
-
Component/s: Extensions
> A test in PutORCTest fails because it depends on timezone
> -
>
> Key: NIFI-5676
> URL: https://issues.apache.org/jira/browse/NIFI-5676
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Test {{testWriteORCWithAvroLogicalTypes}} fails because it depends on
> timezone. It seems that, in this test, the converted date in ORC is "1 day"
> before the input date when the timezone is ahead of GMT. For example, when
> the timezone is "GMT+9", this test fails as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.orc.PutORCTest
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.694
> s <<< FAILURE! - in org.apache.nifi.processors.orc.PutORCTest
> [ERROR]
> testWriteORCWithAvroLogicalTypes(org.apache.nifi.processors.orc.PutORCTest)
> Time elapsed: 0.146 s <<< FAILURE!
> java.lang.AssertionError: expected:<17813> but was:<17812>
> at
> org.apache.nifi.processors.orc.PutORCTest.lambda$testWriteORCWithAvroLogicalTypes$1(PutORCTest.java:250)
> at
> org.apache.nifi.processors.orc.PutORCTest.verifyORCUsers(PutORCTest.java:408)
> at
> org.apache.nifi.processors.orc.PutORCTest.testWriteORCWithAvroLogicalTypes(PutORCTest.java:246)
> {noformat}
> To fix this, we can set the timezone information in the assertion as we did
> in [NIFI-3718|https://issues.apache.org/jira/browse/NIFI-3718].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5676) A test in PutORCTest fails because it depends on timezone
[
https://issues.apache.org/jira/browse/NIFI-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-5676.
--
Resolution: Fixed
Fix Version/s: 1.8.0
> A test in PutORCTest fails because it depends on timezone
> -
>
> Key: NIFI-5676
> URL: https://issues.apache.org/jira/browse/NIFI-5676
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Test {{testWriteORCWithAvroLogicalTypes}} fails because it depends on
> timezone. It seems that, in this test, the converted date in ORC is "1 day"
> before the input date when the timezone is ahead of GMT. For example, when
> the timezone is "GMT+9", this test fails as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.orc.PutORCTest
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.694
> s <<< FAILURE! - in org.apache.nifi.processors.orc.PutORCTest
> [ERROR]
> testWriteORCWithAvroLogicalTypes(org.apache.nifi.processors.orc.PutORCTest)
> Time elapsed: 0.146 s <<< FAILURE!
> java.lang.AssertionError: expected:<17813> but was:<17812>
> at
> org.apache.nifi.processors.orc.PutORCTest.lambda$testWriteORCWithAvroLogicalTypes$1(PutORCTest.java:250)
> at
> org.apache.nifi.processors.orc.PutORCTest.verifyORCUsers(PutORCTest.java:408)
> at
> org.apache.nifi.processors.orc.PutORCTest.testWriteORCWithAvroLogicalTypes(PutORCTest.java:246)
> {noformat}
> To fix this, we can set the timezone information in the assertion as we did
> in [NIFI-3718|https://issues.apache.org/jira/browse/NIFI-3718].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3058: NIFI-5675: Fix some locale-dependent tests in ConvertExcel...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3058 +1, merging to master, thanks @kotarot ---
[jira] [Commented] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644581#comment-16644581
]
ASF GitHub Bot commented on NIFI-5675:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3058
+1, merging to master, thanks @kotarot
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Priority: Major
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644590#comment-16644590
]
ASF GitHub Bot commented on NIFI-5675:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3058
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Priority: Major
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3058: NIFI-5675: Fix some locale-dependent tests in Conve...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3058 ---
[jira] [Updated] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5675:
-
Component/s: Extensions
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5675:
-
Fix Version/s: 1.8.0
Status: Patch Available (was: Open)
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5675:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (MINIFICPP-632) Testing CAPI
[ https://issues.apache.org/jira/browse/MINIFICPP-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda reassigned MINIFICPP-632: Assignee: Arpad Boda > Testing CAPI > > > Key: MINIFICPP-632 > URL: https://issues.apache.org/jira/browse/MINIFICPP-632 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Major > > Ideas for Testing: > > Testing Tailing Log File > > 1) Use TestController to create temp files and directories ( see > PutFileTests as an example) > 2) Test that you add appropriate attributes and that content exists > 3) Test negative cases for removing log files > 4) Negative test cases for invalid/null arguments -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MINIFICPP-632) Testing CAPI
[ https://issues.apache.org/jira/browse/MINIFICPP-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda reassigned MINIFICPP-632: Assignee: (was: Arpad Boda) > Testing CAPI > > > Key: MINIFICPP-632 > URL: https://issues.apache.org/jira/browse/MINIFICPP-632 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Mr TheSegfault >Priority: Major > > Ideas for Testing: > > Testing Tailing Log File > > 1) Use TestController to create temp files and directories ( see > PutFileTests as an example) > 2) Test that you add appropriate attributes and that content exists > 3) Test negative cases for removing log files > 4) Negative test cases for invalid/null arguments -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-636) C API: allow creation of empty flow, remove some code duplication
[ https://issues.apache.org/jira/browse/MINIFICPP-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda resolved MINIFICPP-636. -- Resolution: Fixed > C API: allow creation of empty flow, remove some code duplication > - > > Key: MINIFICPP-636 > URL: https://issues.apache.org/jira/browse/MINIFICPP-636 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > The current C API doesn't support creation of emty flows. Add that to improve > usability of the API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 Hi, I have been verifying the pull request and I came across this error for one of my Avro messages, I am currently working to figure out which part of the message caused this. ``` 2018-10-10 14:14:42,489 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.orc.PutORC PutORC[id=0430e7ab-99f1-3e25-c491-935245567fa3] Failed to write due to java.lang.ClassCa stException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo: java.lang.ClassCastException: org.apache.hadoop .hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:127) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:177) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$convertToORCObject$0(NiFiOrcUtils.java:129) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:130) at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:73) at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:94) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ---
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[ https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644914#comment-16644914 ] ASF GitHub Bot commented on NIFI-5667: -- Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 Hi, I have been verifying the pull request and I came across this error for one of my Avro messages, I am currently working to figure out which part of the message caused this. ``` 2018-10-10 14:14:42,489 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.orc.PutORC PutORC[id=0430e7ab-99f1-3e25-c491-935245567fa3] Failed to write due to java.lang.ClassCa stException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo: java.lang.ClassCastException: org.apache.hadoop .hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:127) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:177) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$convertToORCObject$0(NiFiOrcUtils.java:129) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:130) at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:73) at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:94) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662) at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` > Hive3 PutOrc processors, error when using nestled Avro Record types > --- > > Key: NIFI-5667 > URL: https://issues.apache.org/jira/browse/NIFI-5667 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0, 1.7.1 > Environment: Centos 7 and Docker Image from Hortonworks >Reporter: Viking Karstorp >Assignee: Matt Burgess >Priority: Major > > I have been test
[GitHub] nifi issue #3046: NIFI-5661: Adding Load Balance config UI
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3046 Thanks for the update. We do not want to reference another module directly via `nf.`. This would only be working currently because we have not enabled a module loader yet. However, the codebase currently supports one. This can be seen at the top of any of the JS files. If a module loader was turned on, this would fail because the `nf.` namespace would no longer be in global scope. If you want to reference something in another module, we'll need to import it. Alternatively, you could import it in a module that is already imported in both. In the policy case, the combo options are in `nfCommon` and a utility function was added to access them. ---
[GitHub] nifi issue #3037: NIFI-5645: Auto reconnect ConsumeWindowsEventLog
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3037 Will review... ---
[jira] [Commented] (NIFI-5661) Add UI support for load balancing connections
[ https://issues.apache.org/jira/browse/NIFI-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644942#comment-16644942 ] ASF GitHub Bot commented on NIFI-5661: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3046 Thanks for the update. We do not want to reference another module directly via `nf.`. This would only be working currently because we have not enabled a module loader yet. However, the codebase currently supports one. This can be seen at the top of any of the JS files. If a module loader was turned on, this would fail because the `nf.` namespace would no longer be in global scope. If you want to reference something in another module, we'll need to import it. Alternatively, you could import it in a module that is already imported in both. In the policy case, the combo options are in `nfCommon` and a utility function was added to access them. > Add UI support for load balancing connections > - > > Key: NIFI-5661 > URL: https://issues.apache.org/jira/browse/NIFI-5661 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > NIFI-5516 adds FlowFile load-balancing feature at connections. We need UI > controls to: > - Configure if a connection is load balanced and what strategy to use. > - Indicator that load balancing is in progress -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5645) Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured time period
[ https://issues.apache.org/jira/browse/NIFI-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644944#comment-16644944 ] ASF GitHub Bot commented on NIFI-5645: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3037 Will review... > Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured > time period > -- > > Key: NIFI-5645 > URL: https://issues.apache.org/jira/browse/NIFI-5645 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > While ConsumeWindowsEventLog is running, if Windows Event Log service is > restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned, the processor > keeps running, but will not receive further log events. > Current work-around is restarting the processor manually. > We could implement auto-reconnect logic like below, so that the processor can > recover from such situation automatically. > * Add a new processor property, ''Inactive duration to reconnect", e.g. "3 > mins" > * At onTrigger, check how long has it passed since the last message was > received. If it exceeds the configured duration, reset the consumer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5677) Add/clarify why modifying/creating variables are not considered local changes in versioned flows
Andrew Lim created NIFI-5677: Summary: Add/clarify why modifying/creating variables are not considered local changes in versioned flows Key: NIFI-5677 URL: https://issues.apache.org/jira/browse/NIFI-5677 Project: Apache NiFi Issue Type: Improvement Components: Documentation & Website Reporter: Andrew Lim There has been some confusion over why creating or modifying variables in a versioned flow do not trigger local changes in the flow. Will improve the relevant section in the User Guide (https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#managing_local_changes) with the following clarifications: Modifying doesn’t trigger local changes because variable values are intended to be different in each environment. When a flow is imported to an environment, it is assumed there is a one-time operation required to set those variables specific for the given environment. Creating a variable doesn’t trigger a local change because just creating a variable on its own has not changed anything about what the flow processes. A component will have to be created/modified that uses the new variable, which will trigger a local change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224099820 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). + + Offload Nodes + +Flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading. In the Cluster Management dialog, select the "Offload" icon (image:iconOffload.png["Offload Icon"]) for a Disconnected node. This will stop all processors, terminate all processors, stop transmitting on all remote process groups and rebalance flowfiles to the other connected nodes in the cluster. + +image::offloading-node-cluster-mgt.png["Offloading Node in Cluster Management UI"] + +Nodes that remain in "Offloading" state due to errors encountered (out of memory, no network connection, etc.) can be reconnected to the cluster by restarting NiFi on the node. Offloaded nodes can be either reconnected to the cluster (by selecting Connect or restarting NiFi on the node) or deleted from the cluster. + +image::offloaded-node-cluster-mgt.png["Offloaded Node in Cluster Management UI"] + + Delete Nodes + +There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster. In this case, the DFM may elect to delete the node from the cluster entirely. In the Cluster Management dialog, select the "Delete" icon (image:iconDelete.png["Delete Icon"]) for a Disconnected or Offloaded node. Once deleted, the node cannot be rejoined to the cluster until it has been restarted. + + Decommission Nodes + +The steps to decommission a node and remove it from a cluster are as follows: + +1. Disconnect the node. +2. Once disconnect completes, offload the node. +3. Once offload completes, delete the node. +4. Once the delete request has finished, stop/remove the NiFi service on the host. + + NiFi Toolkit Node Commands -A DFM may manually disconnect a node from the cluster. But if a node becomes disconnected for any other reason (such as due to lack of heartbeat), -the Cluster Coordinator will show a bulletin on the User Interface. The DFM will not be able to make any changes to the dataflow until the issue -of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any -new changes may be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working; -this may happen for a few reasons, including that the node is unable to communicate with the Cluster Coordinator due to network problems. +As an alternative to the UI, the following NiFi Toolkit CLI commands can be used for retrieving a single node, retrieving a list of nodes, and connecting/disconnecting/offloading/deleting nodes: -There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster. -In this case, they DFM may elect to remove the node from the cluster entirely through the Cluster Management dialog. Once removed, -the node cannot be rejoined to the cluster until it has been restarted. +* `nifi get-node` --- End diff -- Sounds good. Will definitely link commands covered
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645042#comment-16645042 ] ASF GitHub Bot commented on NIFI-5659: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224099820 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). + + Offload Nodes + +Flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading. In the Cluster Management dialog, select the "Offload" icon (image:iconOffload.png["Offload Icon"]) for a Disconnected node. This will stop all processors, terminate all processors, stop transmitting on all remote process groups and rebalance flowfiles to the other connected nodes in the cluster. + +image::offloading-node-cluster-mgt.png["Offloading Node in Cluster Management UI"] + +Nodes that remain in "Offloading" state due to errors encountered (out of memory, no network connection, etc.) can be reconnected to the cluster by restarting NiFi on the node. Offloaded nodes can be either reconnected to the cluster (by selecting Connect or restarting NiFi on the node) or deleted from the cluster. + +image::offloaded-node-cluster-mgt.png["Offloaded Node in Cluster Management UI"] + + Delete Nodes + +There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster. In this case, the DFM may elect to delete the node from the cluster entirely. In the Cluster Management dialog, select the "Delete" icon (image:iconDelete.png["Delete Icon"]) for a Disconnected or Offloaded node. Once deleted, the node cannot be rejoined to the cluster until it has been restarted. + + Decommission Nodes + +The steps to decommission a node and remove it from a cluster are as follows: + +1. Disconnect the node. +2. Once disconnect completes, offload the node. +3. Once offload completes, delete the node. +4. Once the delete request has finished, stop/remove the NiFi service on the host. + + NiFi Toolkit Node Commands -A DFM may manually disconnect a node from the cluster. But if a node becomes disconnected for any other reason (such as due to lack of heartbeat), -the Cluster Coordinator will show a bulletin on the User Interface. The DFM will not be able to make any changes to the dataflow until the issue -of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any -new changes may be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working; -this may happen for a few reasons, including that the node is unable to communicate with the Cluster Coordinator due to network problems. +As an alternative to the UI, the following NiFi Toolkit CLI commands can be used for retrieving a single node, retrieving a list of nodes, and connecting/disconnecting/offloading/deleting nodes: -There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster. -In this case, they DFM may elect to remove the node from th
[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3057 Can you share the schema of the data, and possibly a JSON export of your Avro file? I couldn't reproduce this with an array of ints, and @bbende ran successfully with an array of records. ---
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645097#comment-16645097
]
ASF GitHub Bot commented on NIFI-5667:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3057
Can you share the schema of the data, and possibly a JSON export of your
Avro file? I couldn't reproduce this with an array of ints, and @bbende ran
successfully with an array of records.
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.8.0, 1.7.1
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Execu
[jira] [Updated] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5667:
---
Affects Version/s: (was: 1.7.1)
(was: 1.8.0)
Status: Patch Available (was: In Progress)
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
[jira] [Commented] (NIFI-5674) Add referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645120#comment-16645120 ] Scott Wilburn commented on NIFI-5674: - [~bende] , Thanks for the comment. I haven't tried NiFi registry yet, mostly because I read that there was no way to make the mass changes to new versions of process groups. It's possible I miss-read that. I'll just have to test it out for myself. Thanks for the tip. > Add referenceable templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > > Thanks, > Scott -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5678) ValidateRecord does not handle Map type correctly
Matt Burgess created NIFI-5678:
--
Summary: ValidateRecord does not handle Map type correctly
Key: NIFI-5678
URL: https://issues.apache.org/jira/browse/NIFI-5678
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Matt Burgess
Consider the following Avro Schema:
{{{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}}}
and corresponding JSON data adhering to the schema:
{{[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]}}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5678:
---
Description:
Consider the following Avro Schema:
{code}
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
{code}
and corresponding JSON data adhering to the schema:
{code}
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
{code}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
was:
Consider the following Avro Schema:
{{{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}}}
and corresponding JSON data adhering to the schema:
{{[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]}}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5678:
---
Description:
Consider the following Avro Schema:
{code}
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
{code}
and corresponding JSON data adhering to the schema:
{code}
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
{code}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
was:
Consider the following Avro Schema:
{code}
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
{code}
and corresponding JSON data adhering to the schema:
{code}
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
{code}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[\{toto=v1, titi=v2\}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5678:
---
Description:
Consider the following Avro Schema:
{code}
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
{code}
and corresponding JSON data adhering to the schema:
{code}
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
{code}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[\{toto=v1, titi=v2\}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
was:
Consider the following Avro Schema:
{code}
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
{code}
and corresponding JSON data adhering to the schema:
{code}
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
{code}
ValidateRecord marks the record as invalid though it should be valid. The
message in the provenance event is "Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be of
type MAP[STRING]".
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[\{toto=v1, titi=v2\}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645129#comment-16645129
]
Matt Burgess commented on NIFI-5678:
ValidateRecord checks only for the field object being a Map. Additionally it
should support MapRecord, as this is how objects often are exposed by the
Record API (whether they are of Map or Record type).
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5674) Add referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645140#comment-16645140 ] Bryan Bende edited comment on NIFI-5674 at 10/10/18 3:20 PM: - If you publish a new version of a process group, and then you had 10 instances of it imported, currently you would have to go to each one and upgrade each one individually. I think we could offer ability to set a versioned process group to auto-upgrade, but this is definitely something the user needs to opt in to as in many cases it may be scary to have upgrades happening without realizing. We could also offer better ways to bulk upgrade. In my opinion, investing effort in improvements to versioned flows would make more sense than doing a lot of work around templates. was (Author: bende): If you publish a new version of a process group, and then you had 10 instances of it imported, currently you would have to go to each one upgrade each one individually. I think we could offer ability to set a versioned process group to auto-upgrade, but this is definitely something the user needs to opt in to as in many cases it may be scary to have upgrades happening without realizing. We could also offer better ways to bulk upgrade. In my opinion, investing effort in improvements to versioned flows would make more sense than doing a lot of work around templates. > Add referenceable templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > > Thanks, > Scott -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5674) Add referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645140#comment-16645140 ] Bryan Bende commented on NIFI-5674: --- If you publish a new version of a process group, and then you had 10 instances of it imported, currently you would have to go to each one upgrade each one individually. I think we could offer ability to set a versioned process group to auto-upgrade, but this is definitely something the user needs to opt in to as in many cases it may be scary to have upgrades happening without realizing. We could also offer better ways to bulk upgrade. In my opinion, investing effort in improvements to versioned flows would make more sense than doing a lot of work around templates. > Add referenceable templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > > Thanks, > Scott -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3057: NIFI-5667: Add nested record support for PutORC
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3057#discussion_r224126176
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/orc/PutORC.java
---
@@ -157,19 +155,17 @@ public String getDefaultCompressionType(final
ProcessorInitializationContext con
public HDFSRecordWriter createHDFSRecordWriter(final ProcessContext
context, final FlowFile flowFile, final Configuration conf, final Path path,
final RecordSchema schema)
throws IOException, SchemaNotFoundException {
-final Schema avroSchema = AvroTypeUtil.extractAvroSchema(schema);
-
final long stripeSize =
context.getProperty(STRIPE_SIZE).asDataSize(DataUnit.B).longValue();
final int bufferSize =
context.getProperty(BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
final CompressionKind compressionType =
CompressionKind.valueOf(context.getProperty(COMPRESSION_TYPE).getValue());
final boolean normalizeForHive =
context.getProperty(HIVE_FIELD_NAMES).asBoolean();
-TypeInfo orcSchema = NiFiOrcUtils.getOrcField(avroSchema,
normalizeForHive);
+TypeInfo orcSchema = NiFiOrcUtils.getOrcSchema(schema,
normalizeForHive);
final Writer orcWriter = NiFiOrcUtils.createWriter(path, conf,
orcSchema, stripeSize, compressionType, bufferSize);
final String hiveTableName =
context.getProperty(HIVE_TABLE_NAME).isSet()
?
context.getProperty(HIVE_TABLE_NAME).evaluateAttributeExpressions(flowFile).getValue()
-:
NiFiOrcUtils.normalizeHiveTableName(avroSchema.getFullName());
+:
NiFiOrcUtils.normalizeHiveTableName(schema.toString());// TODO
--- End diff --
Yep that's not the right thing to put there :) Will investigate getting a
name from the record somehow, or defaulting to a hardcoded table name if none
is provided.
---
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645145#comment-16645145
]
ASF GitHub Bot commented on NIFI-5667:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3057#discussion_r224126176
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/orc/PutORC.java
---
@@ -157,19 +155,17 @@ public String getDefaultCompressionType(final
ProcessorInitializationContext con
public HDFSRecordWriter createHDFSRecordWriter(final ProcessContext
context, final FlowFile flowFile, final Configuration conf, final Path path,
final RecordSchema schema)
throws IOException, SchemaNotFoundException {
-final Schema avroSchema = AvroTypeUtil.extractAvroSchema(schema);
-
final long stripeSize =
context.getProperty(STRIPE_SIZE).asDataSize(DataUnit.B).longValue();
final int bufferSize =
context.getProperty(BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
final CompressionKind compressionType =
CompressionKind.valueOf(context.getProperty(COMPRESSION_TYPE).getValue());
final boolean normalizeForHive =
context.getProperty(HIVE_FIELD_NAMES).asBoolean();
-TypeInfo orcSchema = NiFiOrcUtils.getOrcField(avroSchema,
normalizeForHive);
+TypeInfo orcSchema = NiFiOrcUtils.getOrcSchema(schema,
normalizeForHive);
final Writer orcWriter = NiFiOrcUtils.createWriter(path, conf,
orcSchema, stripeSize, compressionType, bufferSize);
final String hiveTableName =
context.getProperty(HIVE_TABLE_NAME).isSet()
?
context.getProperty(HIVE_TABLE_NAME).evaluateAttributeExpressions(flowFile).getValue()
-:
NiFiOrcUtils.normalizeHiveTableName(avroSchema.getFullName());
+:
NiFiOrcUtils.normalizeHiveTableName(schema.toString());// TODO
--- End diff --
Yep that's not the right thing to put there :) Will investigate getting a
name from the record somehow, or defaulting to a hardcoded table name if none
is provided.
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645167#comment-16645167 ] ASF GitHub Bot commented on NIFI-5659: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224134708 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- Good points. My intent was to present the available choices provided by the UI for a disconnected node. Perhaps we can add a note to the end of this section that provides more of this error handling/troubleshooting information. NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected because the node died/lack of heartbeat, the offload operation will be unable to reach the node to start the offloading. Additionally, offloading may also be interrupted or prevented due to firewall issues (e.g., if the load balancing port is blocked). > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224134708 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- Good points. My intent was to present the available choices provided by the UI for a disconnected node. Perhaps we can add a note to the end of this section that provides more of this error handling/troubleshooting information. NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected because the node died/lack of heartbeat, the offload operation will be unable to reach the node to start the offloading. Additionally, offloading may also be interrupted or prevented due to firewall issues (e.g., if the load balancing port is blocked). ---
[jira] [Assigned] (NIFI-5653) Add NiFi ports section to the Administration Guide
[ https://issues.apache.org/jira/browse/NIFI-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Lim reassigned NIFI-5653: Assignee: Andrew Lim > Add NiFi ports section to the Administration Guide > -- > > Key: NIFI-5653 > URL: https://issues.apache.org/jira/browse/NIFI-5653 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Trivial > > It would be useful to document/list which ports NiFi opens. Can display what > the defaults are as well as how to configure if applicable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5679) ControllerService validation with dynamic properties
Brandon DeVries created NIFI-5679: - Summary: ControllerService validation with dynamic properties Key: NIFI-5679 URL: https://issues.apache.org/jira/browse/NIFI-5679 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.7.1 Reporter: Brandon DeVries If a controller service (service A) has a dynamic property referencing another controller service (service B), service A can be started even if service B is not enabled. In other words, dynamic properties are not being considered in validation of service A. As a side note... when attempting to stop service B, it does have the correct linkage to know that is must stop service A as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224150506 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- I misspoke on the load balance port being blocked. That would prevent other nodes from sending flowfiles through load balancing to that particular node. If the disconnected node is unreachable, it would not be able to receive the offload request. I agree with moving the error scenarios to the end of the section, or to a separate error handling/troubleshooting section. ---
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645216#comment-16645216 ] ASF GitHub Bot commented on NIFI-5659: -- Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224150506 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- I misspoke on the load balance port being blocked. That would prevent other nodes from sending flowfiles through load balancing to that particular node. If the disconnected node is unreachable, it would not be able to receive the offload request. I agree with moving the error scenarios to the end of the section, or to a separate error handling/troubleshooting section. > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5674) Add referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645214#comment-16645214 ] Scott Wilburn commented on NIFI-5674: - [~bende] Thanks for the clarification. I agree that focusing on auto-upgrade or some kind of bulk upgrade of versioned PGs would be best. I'll update this ticket to reflect that. > Add referenceable templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > > Thanks, > Scott -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5679) ControllerService validation with dynamic properties
[ https://issues.apache.org/jira/browse/NIFI-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon DeVries updated NIFI-5679: -- Description: Dynamic properties are not being considered in validation of controller services. In other words, if a controller service (service A) has a dynamic property referencing another controller service (service B), service A can be started even if service B is not enabled. As a side note... when attempting to stop service B, it does have the correct linkage to know that is must stop service A as well. was: If a controller service (service A) has a dynamic property referencing another controller service (service B), service A can be started even if service B is not enabled. In other words, dynamic properties are not being considered in validation of service A. As a side note... when attempting to stop service B, it does have the correct linkage to know that is must stop service A as well. > ControllerService validation with dynamic properties > > > Key: NIFI-5679 > URL: https://issues.apache.org/jira/browse/NIFI-5679 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Brandon DeVries >Priority: Minor > > Dynamic properties are not being considered in validation of controller > services. In other words, if a controller service (service A) has a dynamic > property referencing another controller service (service B), service A can be > started even if service B is not enabled. > As a side note... when attempting to stop service B, it does have the correct > linkage to know that is must stop service A as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224156171 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- OK. How about this then for the note: NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected because the node died/lack of heartbeat, the offload operation will be unable to reach the node to start the offloading. Additionally, offloading may also be interrupted or prevented due to firewall issues. ---
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645239#comment-16645239 ] ASF GitHub Bot commented on NIFI-5659: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224156171 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- OK. How about this then for the note: NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected because the node died/lack of heartbeat, the offload operation will be unable to reach the node to start the offloading. Additionally, offloading may also be interrupted or prevented due to firewall issues. > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224160990 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- Maybe it would be better to be less explicit: NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected and unreachable, the offload request can not be received by the node to start the offloading. Additionally, offloading may be interrupted or prevented due to firewall rules. ---
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645257#comment-16645257 ] ASF GitHub Bot commented on NIFI-5659: -- Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224160990 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- Maybe it would be better to be less explicit: NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected and unreachable, the offload request can not be received by the node to start the offloading. Additionally, offloading may be interrupted or prevented due to firewall rules. > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224161590 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- I like your version. Let's run with that. ---
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645263#comment-16645263 ] ASF GitHub Bot commented on NIFI-5659: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224161590 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- I like your version. Let's run with that. > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645277#comment-16645277 ] ASF GitHub Bot commented on NIFI-5659: -- Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224164285 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- +1 > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r224164285 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t nodes and waits for each node to respond, indicating that it has made the change on its local flow. -*Dealing with Disconnected Nodes* + +=== Managing Nodes + + Disconnect Nodes + +A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems. + +To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row. + +image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"] + +A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]). --- End diff -- +1 ---
[jira] [Updated] (NIFI-5674) Add bulk upgrade option to versioned process groups or referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Wilburn updated NIFI-5674: Summary: Add bulk upgrade option to versioned process groups or referenceable templates (was: Add referenceable templates) > Add bulk upgrade option to versioned process groups or referenceable > templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > > Thanks, > Scott -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5674) Add bulk upgrade option to versioned process groups or referenceable templates
[ https://issues.apache.org/jira/browse/NIFI-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Wilburn updated NIFI-5674: Description: Everyone knows how important code re-use is. NiFi's templates work great for this if you only have a handful of flows to use them in. Templates act more like copy-paste code re-use, because each time you apply a template, you are making a separate copy of the XML code and applying it to your canvas. What's missing is the ability to maintain one copy of the code, so that you can make changes to logic embedded in a template and have those changes applied to all instances of that code. It should work more like a library of classes in an OO language, or at least have that as an option. I propose having a type of template that can be applied in a read-only state, linked somehow to the saved template. If the source template is updated, changes to all instances of the template should be applied. I would imagine it would act similar to how variables are changed, where it has to stop all processors, apply the changes, then restart them. I think the same general idea could be applied to templates. Referenceable templates would make creating and maintaining large NiFi deployments much easier and less error prone. I know it's been talked about in the groups before, but I think it's time to put some action behind it now. ***EDIT*: Per Bryan's suggestion, this is best solved by adding functionality to the NiFi registry to be able to push out changes to all instances of a PG when the version changes. Not suggesting it be automatic, but it should be a single click to push out to all instances. Thanks, Scott was: Everyone knows how important code re-use is. NiFi's templates work great for this if you only have a handful of flows to use them in. Templates act more like copy-paste code re-use, because each time you apply a template, you are making a separate copy of the XML code and applying it to your canvas. What's missing is the ability to maintain one copy of the code, so that you can make changes to logic embedded in a template and have those changes applied to all instances of that code. It should work more like a library of classes in an OO language, or at least have that as an option. I propose having a type of template that can be applied in a read-only state, linked somehow to the saved template. If the source template is updated, changes to all instances of the template should be applied. I would imagine it would act similar to how variables are changed, where it has to stop all processors, apply the changes, then restart them. I think the same general idea could be applied to templates. Referenceable templates would make creating and maintaining large NiFi deployments much easier and less error prone. I know it's been talked about in the groups before, but I think it's time to put some action behind it now. Thanks, Scott > Add bulk upgrade option to versioned process groups or referenceable > templates > --- > > Key: NIFI-5674 > URL: https://issues.apache.org/jira/browse/NIFI-5674 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Scott Wilburn >Priority: Major > Labels: class, templates > > > Everyone knows how important code re-use is. NiFi's templates work great for > this if you only have a handful of flows to use them in. Templates act more > like copy-paste code re-use, because each time you apply a template, you are > making a separate copy of the XML code and applying it to your canvas. What's > missing is the ability to maintain one copy of the code, so that you can make > changes to logic embedded in a template and have those changes applied to all > instances of that code. It should work more like a library of classes in an > OO language, or at least have that as an option. I propose having a type of > template that can be applied in a read-only state, linked somehow to the > saved template. If the source template is updated, changes to all instances > of the template should be applied. I would imagine it would act similar to > how variables are changed, where it has to stop all processors, apply the > changes, then restart them. I think the same general idea could be applied to > templates. > Referenceable templates would make creating and maintaining large NiFi > deployments much easier and less error prone. I know it's been talked about > in the groups before, but I think it's time to put some action behind it now. > ***EDIT*: Per Bryan's suggestion, this is best solved by adding > functionality to the NiFi registry to be able to push out changes to all > instances o
[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC
Github user VikingK commented on the issue:
https://github.com/apache/nifi/pull/3057
Sorry for the late answer got bogged down.
Avro:
```
{
"name": "IOlist",
"namespace": "analytics.models.its",
"type": "record",
"fields": [
{
"name": "OItems",
"type": [
"null",
{
"type": "array",
"items": {
"name": "ISC",
"namespace": "analytics.models.its.iolist.oitems",
"type": "record",
"fields": [
{
"name": "Od",
"type": [
"null",
"long"
]
},
{
"name": "HS",
"type": [
"null",
"bytes"
]
},
{
"name": "AS",
"type": [
"null",
"bytes"
]
},
{
"name": "NS",
"type": [
"null",
"string"
]
}
]
}
}
]
}
]
}
```
JSON
```
{
"OItems" : {
"array" : [ {
"Od" : {
"long" : 9
},
"HS" : {
"bytes" : "/w=="
},
"AS" : {
"bytes" : "AA=="
},
"NS" : {
"string" : "0"
}
} ]
}
}
```
---
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645356#comment-16645356
]
ASF GitHub Bot commented on NIFI-5667:
--
Github user VikingK commented on the issue:
https://github.com/apache/nifi/pull/3057
Sorry for the late answer got bogged down.
Avro:
```
{
"name": "IOlist",
"namespace": "analytics.models.its",
"type": "record",
"fields": [
{
"name": "OItems",
"type": [
"null",
{
"type": "array",
"items": {
"name": "ISC",
"namespace": "analytics.models.its.iolist.oitems",
"type": "record",
"fields": [
{
"name": "Od",
"type": [
"null",
"long"
]
},
{
"name": "HS",
"type": [
"null",
"bytes"
]
},
{
"name": "AS",
"type": [
"null",
"bytes"
]
},
{
"name": "NS",
"type": [
"null",
"string"
]
}
]
}
}
]
}
]
}
```
JSON
```
{
"OItems" : {
"array" : [ {
"Od" : {
"long" : 9
},
"HS" : {
"bytes" : "/w=="
},
"AS" : {
"bytes" : "AA=="
},
"NS" : {
"string" : "0"
}
} ]
}
}
```
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.N
[jira] [Assigned] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-5678:
--
Assignee: Matt Burgess
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3060: NIFI-5678: Fixed MAP type support of MapRecord obje...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3060 NIFI-5678: Fixed MAP type support of MapRecord objects in StandardSchemaValidator Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5678 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3060.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3060 commit c66de368d07f37a8652c09e4b116b232972821ac Author: Matthew Burgess Date: 2018-10-10T19:01:40Z NIFI-5678: Fixed MAP type support of MapRecord objects in StandardSchemaValidator ---
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5678:
---
Status: Patch Available (was: In Progress)
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645423#comment-16645423
]
ASF GitHub Bot commented on NIFI-5678:
--
GitHub user mattyb149 opened a pull request:
https://github.com/apache/nifi/pull/3060
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mattyb149/nifi NIFI-5678
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3060.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3060
commit c66de368d07f37a8652c09e4b116b232972821ac
Author: Matthew Burgess
Date: 2018-10-10T19:01:40Z
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3037: NIFI-5645: Auto reconnect ConsumeWindowsEventLog
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3037#discussion_r224205359
--- Diff:
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
---
@@ -199,6 +230,8 @@ private String subscribe(ProcessContext context) throws
URISyntaxException {
subscriptionHandle = wEvtApi.EvtSubscribe(null, null, channel,
query, null, null,
evtSubscribeCallback,
WEvtApi.EvtSubscribeFlags.SUBSCRIBE_TO_FUTURE |
WEvtApi.EvtSubscribeFlags.EVT_SUBSCRIBE_STRICT);
+lastActivityTimestamp = System.currentTimeMillis();
--- End diff --
Just wanted to clarify the logic here... Do we want to update the
`lastActivityTimestamp` when the client was unable to subscribe? Could that
lead to premature reconnection later?
---
[jira] [Commented] (NIFI-5645) Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured time period
[
https://issues.apache.org/jira/browse/NIFI-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645427#comment-16645427
]
ASF GitHub Bot commented on NIFI-5645:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3037#discussion_r224205359
--- Diff:
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
---
@@ -199,6 +230,8 @@ private String subscribe(ProcessContext context) throws
URISyntaxException {
subscriptionHandle = wEvtApi.EvtSubscribe(null, null, channel,
query, null, null,
evtSubscribeCallback,
WEvtApi.EvtSubscribeFlags.SUBSCRIBE_TO_FUTURE |
WEvtApi.EvtSubscribeFlags.EVT_SUBSCRIBE_STRICT);
+lastActivityTimestamp = System.currentTimeMillis();
--- End diff --
Just wanted to clarify the logic here... Do we want to update the
`lastActivityTimestamp` when the client was unable to subscribe? Could that
lead to premature reconnection later?
> Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured
> time period
> --
>
> Key: NIFI-5645
> URL: https://issues.apache.org/jira/browse/NIFI-5645
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> While ConsumeWindowsEventLog is running, if Windows Event Log service is
> restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned, the processor
> keeps running, but will not receive further log events.
> Current work-around is restarting the processor manually.
> We could implement auto-reconnect logic like below, so that the processor can
> recover from such situation automatically.
> * Add a new processor property, ''Inactive duration to reconnect", e.g. "3
> mins"
> * At onTrigger, check how long has it passed since the last message was
> received. If it exceeds the configured duration, reset the consumer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3060: NIFI-5678: Fixed MAP type support of MapRecord obje...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224205938
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Would this be the same if it was a MapRecord?
---
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645433#comment-16645433
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224205938
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Would this be the same if it was a MapRecord?
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3060: NIFI-5678: Fixed MAP type support of MapRecord obje...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224208847
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Sorry, I looked through the code and understand. excuse me
---
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645442#comment-16645442
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224208847
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Sorry, I looked through the code and understand. excuse me
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5593) ConsumeKafka_0_11 topic name variable is not evaluated correctly
[
https://issues.apache.org/jira/browse/NIFI-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-5593.
--
Resolution: Information Provided
> ConsumeKafka_0_11 topic name variable is not evaluated correctly
>
>
> Key: NIFI-5593
> URL: https://issues.apache.org/jira/browse/NIFI-5593
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.5.0
> Environment: RHEL 7.3
> JAVA Oracle 1.8.0_144
> NiFi 1.5.0
> Processor name "ConsumeKafka_0_11"
>Reporter: Georgy
>Priority: Major
> Attachments: proc_prop.PNG, proc_sched.PNG, proc_sett.PNG
>
>
> Hi guys,
> Found that "Topic Name" specified as expression "dmp_order_${now()
> :minus(360) :format('HH') }" is not evaluated correctly in
> "ConsumeKafka_0_11" processor after next hour begins.
> It is evaluated as expected after processor start/restart only.
> Say, the processor is started at 15:05. Value of "Topic name" variable will
> be "dmp_order_14". It's correct. It will consume data from the topic and
> everything is ok.
> After next hour begins, value of "Topic name" is not evaluated as
> "dmp_order_15". I see that consumer still tries to consume data from topic
> with name "dmp_order_14".
> I tried both Scheduling strategies "Timer driven" and "CRON driven" but have
> no success. Seems, I have to restart processor every time I want to
> re-evalueate "Topic name" variable.
> When I filled CustomText field with now() expression in "GenerateFlowFile"
> processor I didn't have such issues. It worked as expected with "CRON driven"
> strategy.
> Can you say is it a bug or normal behaviour for ConsumeKafka processor?
> And if it is a normal behaviour can you say is there any opportunity to
> change "Topic name" dynamically in running processor?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5680) Using inconsistent location URL for Registry Client on different NiFi instances will break ability to import version controlled flows that contain nested version controlle
Matthew Clarke created NIFI-5680: Summary: Using inconsistent location URL for Registry Client on different NiFi instances will break ability to import version controlled flows that contain nested version controlled flows. Key: NIFI-5680 URL: https://issues.apache.org/jira/browse/NIFI-5680 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.7.0, 1.6.0, 1.5.0 Reporter: Matthew Clarke Using inconsistent location URL for Registry Client on different NiFi instances will break ability to import version controlled flows that contain nested version controlled flows. For example: on NiFi instance 1, configure "Registry Client" location URL with https://:/ on NiFi instance 2, configure "Registry Client" location URL with https://: notice that one contains a trailing "/" On one of your NiFi instances, create a Process Group (PG). Inside that PG, create another "sub-PG". Start version control of sub-PG. Go back to parent PG and also start version control on that "PG". Go to your other NiFi instance and try to "import" the "PG" version controlled flow. Import will fail and throw following error in nifi-user.log: o.a.n.w.a.c.IllegalArgumentExceptionMapper java.lang.IllegalArgumentException: The Flow Registry with ID reports that no Flow exists with Bucket , Flow
[jira] [Assigned] (NIFI-5680) Using inconsistent location URL for Registry Client on different NiFi instances will break ability to import version controlled flows that contain nested version controll
[ https://issues.apache.org/jira/browse/NIFI-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende reassigned NIFI-5680: - Assignee: Bryan Bende > Using inconsistent location URL for Registry Client on different NiFi > instances will break ability to import version controlled flows that contain > nested version controlled flows. > --- > > Key: NIFI-5680 > URL: https://issues.apache.org/jira/browse/NIFI-5680 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Matthew Clarke >Assignee: Bryan Bende >Priority: Major > > Using inconsistent location URL for Registry Client on different NiFi > instances will break ability to import version controlled flows that contain > nested version controlled flows. > For example: > on NiFi instance 1, configure "Registry Client" location URL with > https://:/ > on NiFi instance 2, configure "Registry Client" location URL with > https://: > notice that one contains a trailing "/" > On one of your NiFi instances, create a Process Group (PG). > Inside that PG, create another "sub-PG". > Start version control of sub-PG. > Go back to parent PG and also start version control on that "PG". > Go to your other NiFi instance and try to "import" the "PG" version > controlled flow. > Import will fail and throw following error in nifi-user.log: > o.a.n.w.a.c.IllegalArgumentExceptionMapper > java.lang.IllegalArgumentException: The Flow Registry with ID reports > that no Flow exists with Bucket , Flow Request response. > Even though registry does include the referenced UUIDs in that log output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2991: NIFI-3469: multipart request support added to Handl...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2991#discussion_r224003310
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpRequest.java
---
@@ -521,161 +565,223 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final long start = System.nanoTime();
final HttpServletRequest request = container.getRequest();
-FlowFile flowFile = session.create();
-try (OutputStream flowFileOut = session.write(flowFile)) {
-StreamUtils.copy(request.getInputStream(), flowFileOut);
-} catch (final IOException e) {
-// There may be many reasons which can produce an IOException
on the HTTP stream and in some of them, eg.
-// bad requests, the connection to the client is not closed.
In order to address also these cases, we try
-// and answer with a BAD_REQUEST, which lets the client know
that the request has not been correctly
-// processed and makes it aware that the connection can be
closed.
-getLogger().error("Failed to receive content from HTTP Request
from {} due to {}",
-new Object[]{request.getRemoteAddr(), e});
-session.remove(flowFile);
-try {
-HttpServletResponse response = container.getResponse();
-response.sendError(Status.BAD_REQUEST.getStatusCode());
-response.flushBuffer();
-container.getContext().complete();
-} catch (final IOException ioe) {
-getLogger().warn("Failed to send HTTP response to {} due
to {}",
-new Object[]{request.getRemoteAddr(), ioe});
+if (!Strings.isNullOrEmpty(request.getContentType()) &&
request.getContentType().contains(MIME_TYPE__MULTIPART_FORM_DATA)) {
+ final long requestMaxSize =
context.getProperty(MULTIPART_REQUEST_MAX_SIZE).asDataSize(DataUnit.B).longValue();
+ final int readBufferSize =
context.getProperty(MULTIPART_READ_BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
+ String tempDir = System.getProperty("java.io.tmpdir");
+ request.setAttribute(Request.__MULTIPART_CONFIG_ELEMENT, new
MultipartConfigElement(tempDir, requestMaxSize, requestMaxSize,
readBufferSize));
+ try {
+List parts = ImmutableList.copyOf(request.getParts());
+int allPartsCount = parts.size();
+final String contextIdentifier = UUID.randomUUID().toString();
+for (int i = 0; i < allPartsCount; i++) {
+ Part part = parts.get(i);
+ FlowFile flowFile = session.create();
+ try (OutputStream flowFileOut = session.write(flowFile)) {
+StreamUtils.copy(part.getInputStream(), flowFileOut);
+ } catch (IOException e) {
+handleFlowContentStreamingError(session, container,
request, Optional.of(flowFile), e);
+return;
+ }
+ flowFile = savePartAttributes(context, session, part,
flowFile, i, allPartsCount);
+ flowFile = saveRequestAttributes(context, session, request,
flowFile, contextIdentifier);
+ forwardFlowFile(context, session, container, start, request,
flowFile, i == 0);
--- End diff --
What if `contextMap.register()` method returns `false`? I haven't test
that, but current code seems to keep processing remaining multi-part. The
register part should be a separated method and return if the request is
successfully registered, so that this `allPartsCount` loop can break.
Adding a unit test case to confirm that situation would be helpful.
---
[jira] [Commented] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645846#comment-16645846
]
ASF GitHub Bot commented on NIFI-3469:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2991#discussion_r224003310
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpRequest.java
---
@@ -521,161 +565,223 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final long start = System.nanoTime();
final HttpServletRequest request = container.getRequest();
-FlowFile flowFile = session.create();
-try (OutputStream flowFileOut = session.write(flowFile)) {
-StreamUtils.copy(request.getInputStream(), flowFileOut);
-} catch (final IOException e) {
-// There may be many reasons which can produce an IOException
on the HTTP stream and in some of them, eg.
-// bad requests, the connection to the client is not closed.
In order to address also these cases, we try
-// and answer with a BAD_REQUEST, which lets the client know
that the request has not been correctly
-// processed and makes it aware that the connection can be
closed.
-getLogger().error("Failed to receive content from HTTP Request
from {} due to {}",
-new Object[]{request.getRemoteAddr(), e});
-session.remove(flowFile);
-try {
-HttpServletResponse response = container.getResponse();
-response.sendError(Status.BAD_REQUEST.getStatusCode());
-response.flushBuffer();
-container.getContext().complete();
-} catch (final IOException ioe) {
-getLogger().warn("Failed to send HTTP response to {} due
to {}",
-new Object[]{request.getRemoteAddr(), ioe});
+if (!Strings.isNullOrEmpty(request.getContentType()) &&
request.getContentType().contains(MIME_TYPE__MULTIPART_FORM_DATA)) {
+ final long requestMaxSize =
context.getProperty(MULTIPART_REQUEST_MAX_SIZE).asDataSize(DataUnit.B).longValue();
+ final int readBufferSize =
context.getProperty(MULTIPART_READ_BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
+ String tempDir = System.getProperty("java.io.tmpdir");
+ request.setAttribute(Request.__MULTIPART_CONFIG_ELEMENT, new
MultipartConfigElement(tempDir, requestMaxSize, requestMaxSize,
readBufferSize));
+ try {
+List parts = ImmutableList.copyOf(request.getParts());
+int allPartsCount = parts.size();
+final String contextIdentifier = UUID.randomUUID().toString();
+for (int i = 0; i < allPartsCount; i++) {
+ Part part = parts.get(i);
+ FlowFile flowFile = session.create();
+ try (OutputStream flowFileOut = session.write(flowFile)) {
+StreamUtils.copy(part.getInputStream(), flowFileOut);
+ } catch (IOException e) {
+handleFlowContentStreamingError(session, container,
request, Optional.of(flowFile), e);
+return;
+ }
+ flowFile = savePartAttributes(context, session, part,
flowFile, i, allPartsCount);
+ flowFile = saveRequestAttributes(context, session, request,
flowFile, contextIdentifier);
+ forwardFlowFile(context, session, container, start, request,
flowFile, i == 0);
--- End diff --
What if `contextMap.register()` method returns `false`? I haven't test
that, but current code seems to keep processing remaining multi-part. The
register part should be a separated method and return if the request is
successfully registered, so that this `allPartsCount` loop can break.
Adding a unit test case to confirm that situation would be helpful.
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="
[GitHub] nifi issue #3046: NIFI-5661: Adding Load Balance config UI
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3046 @mcgilman I understand that. I moved the combo options to `nfCommon`. Hope this PR looks good now. Thanks! ---
[jira] [Commented] (NIFI-5661) Add UI support for load balancing connections
[ https://issues.apache.org/jira/browse/NIFI-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645861#comment-16645861 ] ASF GitHub Bot commented on NIFI-5661: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3046 @mcgilman I understand that. I moved the combo options to `nfCommon`. Hope this PR looks good now. Thanks! > Add UI support for load balancing connections > - > > Key: NIFI-5661 > URL: https://issues.apache.org/jira/browse/NIFI-5661 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > NIFI-5516 adds FlowFile load-balancing feature at connections. We need UI > controls to: > - Configure if a connection is load balanced and what strategy to use. > - Indicator that load balancing is in progress -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3037: NIFI-5645: Auto reconnect ConsumeWindowsEventLog
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3037#discussion_r224302591
--- Diff:
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
---
@@ -199,6 +230,8 @@ private String subscribe(ProcessContext context) throws
URISyntaxException {
subscriptionHandle = wEvtApi.EvtSubscribe(null, null, channel,
query, null, null,
evtSubscribeCallback,
WEvtApi.EvtSubscribeFlags.SUBSCRIBE_TO_FUTURE |
WEvtApi.EvtSubscribeFlags.EVT_SUBSCRIBE_STRICT);
+lastActivityTimestamp = System.currentTimeMillis();
--- End diff --
I think if the client failed to subscribe, it keeps trying reconnecting at
subsequent onTrigger. So, updating lastActivityTimestamp in case of
subscription failure will not lead to premature I believe. But, I updated the
code anyway. It looks more correct in that way, thanks!
---
[jira] [Commented] (NIFI-5645) Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured time period
[
https://issues.apache.org/jira/browse/NIFI-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645894#comment-16645894
]
ASF GitHub Bot commented on NIFI-5645:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3037#discussion_r224302591
--- Diff:
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
---
@@ -199,6 +230,8 @@ private String subscribe(ProcessContext context) throws
URISyntaxException {
subscriptionHandle = wEvtApi.EvtSubscribe(null, null, channel,
query, null, null,
evtSubscribeCallback,
WEvtApi.EvtSubscribeFlags.SUBSCRIBE_TO_FUTURE |
WEvtApi.EvtSubscribeFlags.EVT_SUBSCRIBE_STRICT);
+lastActivityTimestamp = System.currentTimeMillis();
--- End diff --
I think if the client failed to subscribe, it keeps trying reconnecting at
subsequent onTrigger. So, updating lastActivityTimestamp in case of
subscription failure will not lead to premature I believe. But, I updated the
code anyway. It looks more correct in that way, thanks!
> Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured
> time period
> --
>
> Key: NIFI-5645
> URL: https://issues.apache.org/jira/browse/NIFI-5645
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> While ConsumeWindowsEventLog is running, if Windows Event Log service is
> restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned, the processor
> keeps running, but will not receive further log events.
> Current work-around is restarting the processor manually.
> We could implement auto-reconnect logic like below, so that the processor can
> recover from such situation automatically.
> * Add a new processor property, ''Inactive duration to reconnect", e.g. "3
> mins"
> * At onTrigger, check how long has it passed since the last message was
> received. If it exceeds the configured duration, reset the consumer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-170) nar-maven-plugin does not have any kind of tests (unit- nor integration tests)
[ https://issues.apache.org/jira/browse/NIFI-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manivannan reassigned NIFI-170: Assignee: (was: Arun Manivannan) > nar-maven-plugin does not have any kind of tests (unit- nor integration tests) > -- > > Key: NIFI-170 > URL: https://issues.apache.org/jira/browse/NIFI-170 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 0.0.1 >Reporter: Karl Heinz Marbaise >Priority: Major > Labels: beginner > > There should be some kind of unit and/or integration tests for the plugin to > check if the plugin does what it should and of course make changes easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3037: NIFI-5645: Auto reconnect ConsumeWindowsEventLog
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3037 @mcgilman I've also tried different JNA versions to see if the issue has been solved at the library. The latest 5.0.0 produced following test failure. I haven't researched on this but it may be caused by some breaking-change: ``` [ERROR] org.apache.nifi.processors.windows.event.log.jna.ErrorLookupTest Time elapsed: 0.154 s <<< ERROR! java.lang.UnsatisfiedLinkError: Unable to load library 'kernel32': Native library (darwin/libkernel32.dylib) not found in resource path ([file:/Users/koji/dev/nifi/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/target/surefire/surefirebooter3705181589607153497.jar]) ``` The latest 4.x version 4.5.2 worked fine. Although, just updating JNA version didn't addressed the issue, there has been number of bug-fixes, so I updated the JNA version from 4.2.2 to 4.5.2. ---
[jira] [Commented] (NIFI-5645) Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured time period
[ https://issues.apache.org/jira/browse/NIFI-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645905#comment-16645905 ] ASF GitHub Bot commented on NIFI-5645: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3037 @mcgilman I've also tried different JNA versions to see if the issue has been solved at the library. The latest 5.0.0 produced following test failure. I haven't researched on this but it may be caused by some breaking-change: ``` [ERROR] org.apache.nifi.processors.windows.event.log.jna.ErrorLookupTest Time elapsed: 0.154 s <<< ERROR! java.lang.UnsatisfiedLinkError: Unable to load library 'kernel32': Native library (darwin/libkernel32.dylib) not found in resource path ([file:/Users/koji/dev/nifi/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/target/surefire/surefirebooter3705181589607153497.jar]) ``` The latest 4.x version 4.5.2 worked fine. Although, just updating JNA version didn't addressed the issue, there has been number of bug-fixes, so I updated the JNA version from 4.2.2 to 4.5.2. > Auto reconnect ConsumeWindowsEventLog if no log is consumed for configured > time period > -- > > Key: NIFI-5645 > URL: https://issues.apache.org/jira/browse/NIFI-5645 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > While ConsumeWindowsEventLog is running, if Windows Event Log service is > restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned, the processor > keeps running, but will not receive further log events. > Current work-around is restarting the processor manually. > We could implement auto-reconnect logic like below, so that the processor can > recover from such situation automatically. > * Add a new processor property, ''Inactive duration to reconnect", e.g. "3 > mins" > * At onTrigger, check how long has it passed since the last message was > received. If it exceeds the configured duration, reset the consumer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
