[jira] [Commented] (NIFI-13992) Excessive bootstrap logging
[
https://issues.apache.org/jira/browse/NIFI-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896729#comment-17896729
]
David Handermann commented on NIFI-13992:
-
To provide some background context, as commented in the Apache NiFI Slack:
For containerized deployments, the bootstrap process is really unnecessary, so
the {{nifi.sh run}} command starts NiFi and does not continue running the
bootstrap monitor.
When launching with {{nifi.sh start}} the bootstrap process continues running.
The status logging raises the question of "who watches the watchers?" as it
relates to the health of the bootstrap process itself. Defaulting the status
message to the info level, and logging every five seconds gives a positive
indication of the bootstrap status, and a historical track if the main process
dies. However, I appreciate that this could be considered excessive in some
scenarios. For such situations, simply changing the Logback configuration for
that logger to WARN is an option.
The current default is a bit a of middle-ground approach, although at a high
level, using nifi.sh run with some other robust monitoring strategy is the
better option.
> Excessive bootstrap logging
> ---
>
> Key: NIFI-13992
> URL: https://issues.apache.org/jira/browse/NIFI-13992
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mark Bean
>Assignee: Mark Bean
>Priority: Major
>
> The bootstrap application logger is writing a status message to
> `nifi-bootstrap.log` every 5 seconds. This is an excessive amount of logging.
> Recommend reducing the frequency and/or changing the logging to DEBUG level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13989) Refactor JsonTreeRowRecordReader constructor to take a parameter object
[ https://issues.apache.org/jira/browse/NIFI-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896728#comment-17896728 ] David Handermann commented on NIFI-13989: - I still lean against introducing a parameter object for its own sake in this scenario, as it seems that reducing the number of constructors would address most of the maintenance concerns. > Refactor JsonTreeRowRecordReader constructor to take a parameter object > --- > > Key: NIFI-13989 > URL: https://issues.apache.org/jira/browse/NIFI-13989 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > > When attempting a fix for NIFI-13963 I found it very hard to add another > argument as there are already 13 arguments for the current constructor. This > is a code smell Martin Fowler identifies as a long parameter list > [https://luzkan.github.io/smells/long-parameter-list.|https://luzkan.github.io/smells/long-parameter-list] > One of the solutions is to create a parameter object to encompass all the > arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13989) Refactor JsonTreeRowRecordReader constructor to take a parameter object
[ https://issues.apache.org/jira/browse/NIFI-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896725#comment-17896725 ] David Handermann commented on NIFI-13989: - As a first pass, I would consider reducing the number of constructors to 1, since the others appear to be basically for convenience in testing. Having a shared test method would obviate the need for multiple constructors, making it easier to maintain across the board. > Refactor JsonTreeRowRecordReader constructor to take a parameter object > --- > > Key: NIFI-13989 > URL: https://issues.apache.org/jira/browse/NIFI-13989 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > > When attempting a fix for NIFI-13963 I found it very hard to add another > argument as there are already 13 arguments for the current constructor. This > is a code smell Martin Fowler identifies as a long parameter list > [https://luzkan.github.io/smells/long-parameter-list.|https://luzkan.github.io/smells/long-parameter-list] > One of the solutions is to create a parameter object to encompass all the > arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13989) Refactor JsonTreeRowRecordReader constructor to take a parameter object
[ https://issues.apache.org/jira/browse/NIFI-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896720#comment-17896720 ] David Handermann commented on NIFI-13989: - Thanks for the reply [~dstiegli1]. It is worth noting that several of the overloaded constructors have usage limited to test classes, so there may be some value in refactoring test references to streamline the number of constructors. However, although the number of constructor arguments has grown over time, this is not a public-facing class. As such, I would not expect it to change very often. > Refactor JsonTreeRowRecordReader constructor to take a parameter object > --- > > Key: NIFI-13989 > URL: https://issues.apache.org/jira/browse/NIFI-13989 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > > When attempting a fix for NIFI-13963 I found it very hard to add another > argument as there are already 13 arguments for the current constructor. This > is a code smell Martin Fowler identifies as a long parameter list > [https://luzkan.github.io/smells/long-parameter-list.|https://luzkan.github.io/smells/long-parameter-list] > One of the solutions is to create a parameter object to encompass all the > arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13989) Refactor JsonTreeRowRecordReader constructor to take a parameter object
[ https://issues.apache.org/jira/browse/NIFI-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896715#comment-17896715 ] David Handermann commented on NIFI-13989: - [~dstiegli1] I do not recommend making this change. Although a large number of arguments can be a problem in some cases, simply refactoring that for its own sake does not seem worth the effort right now. > Refactor JsonTreeRowRecordReader constructor to take a parameter object > --- > > Key: NIFI-13989 > URL: https://issues.apache.org/jira/browse/NIFI-13989 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > > When attempting a fix for NIFI-13963 I found it very hard to add another > argument as there already 13 arguments for the current constructor. This is a > code smell Martin Fowler identifies as a long parameter list > [https://luzkan.github.io/smells/long-parameter-list.|https://luzkan.github.io/smells/long-parameter-list] > One of the solutions is to create a parameter object to encompass all the > arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13989) Refactor JsonTreeRowRecordReader constructor to take a parameter object
[ https://issues.apache.org/jira/browse/NIFI-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13989: Affects Version/s: (was: 2.0.0) > Refactor JsonTreeRowRecordReader constructor to take a parameter object > --- > > Key: NIFI-13989 > URL: https://issues.apache.org/jira/browse/NIFI-13989 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > > When attempting a fix for NIFI-13963 I found it very hard to add another > argument as there already 13 arguments for the current constructor. This is a > code smell Martin Fowler identifies as a long parameter list > [https://luzkan.github.io/smells/long-parameter-list.|https://luzkan.github.io/smells/long-parameter-list] > One of the solutions is to create a parameter object to encompass all the > arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13985) StandardSSLContextService error for when keystore file is not provided
[
https://issues.apache.org/jira/browse/NIFI-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-13985.
-
Fix Version/s: 2.1.0
Resolution: Fixed
> StandardSSLContextService error for when keystore file is not provided
> --
>
> Key: NIFI-13985
> URL: https://issues.apache.org/jira/browse/NIFI-13985
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Mark Bean
>Assignee: Mark Bean
>Priority: Major
> Fix For: 2.1.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, when the Keystore file is not provided in the
> StandardSSLContextService configuration, a TrustManager not be created.
> {{252 final TrustManager trustManager;}}
> {{253 final String trustStoreFile = {*}getKeyStoreFile{*}();}}
> {{254 if (trustStoreFile == null || trustStoreFile.isBlank()) {}}
> {{255 getLogger().debug("Trust Store File not configured");}}
> {{256 } else {}}
> {{257 trustManager = createTrustManager();}}
> {{258 sslContextBuilder.trustManager(trustManager);}}
> {{259 }}}
> Line 254 should be:
> {{ final String trustStoreFile = {*}getTrustStoreFile{*}();}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13988) ExcelReader - Use Starting Row schema strategy and string empty values
[
https://issues.apache.org/jira/browse/NIFI-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896712#comment-17896712
]
David Handermann commented on NIFI-13988:
-
[~dstiegli1] The main reason for limiting the number of rows is for efficiency,
avoiding the need to read the file twice.
That's a good point [~iiojj2], given the uncertainty, a more flexible solution
could be to always have {{STRING}} as one of several type choices.
Alternatively, based on the stack trace, it seems like number handling should
account for an empty string and treat it as a {{null}} when the inferred type
is a number.
> ExcelReader - Use Starting Row schema strategy and string empty values
> --
>
> Key: NIFI-13988
> URL: https://issues.apache.org/jira/browse/NIFI-13988
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 2.0.0
>Reporter: Philipp Korniets
>Assignee: Daniel Stieglitz
>Priority: Major
> Attachments: Test workbook NiFi2_0.xlsx
>
>
> When Use Starting Row as schema strategy in ExcelReader it analyses first 10
> row. Problem appears with empty cells of *Numerical* type which can appear
> anywhere after 10 rows. The cells *looks like* NULL, but actually is an empty
> string.
> File with example data attached.
> Field Exercise Price.
> Use Starting Row throws an error:
> {code:java}
> Caused by: java.lang.NumberFormatException: For input string: ""
> at
> java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
> at java.base/java.lang.Long.parseLong(Long.java:719)
> at java.base/java.lang.Long.parseLong(Long.java:832)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.toLong(DataTypeUtils.java:1391)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:213)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:174)
> at
> org.apache.nifi.excel.ExcelRecordReader.convert(ExcelRecordReader.java:170)
> at
> org.apache.nifi.excel.ExcelRecordReader.lambda$getCurrentRowValues$0(ExcelRecordReader.java:127)
> at
> java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
> at
> java.base/java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:617)
> at
> org.apache.nifi.excel.ExcelRecordReader.getCurrentRowValues(ExcelRecordReader.java:114)
> at
> org.apache.nifi.excel.ExcelRecordReader.nextRecord(ExcelRecordReader.java:84)
> ... 28 common frames omitted{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13986) Upgrade Jetty to 12.0.15 and Jackson to 2.18.1 along with others
[ https://issues.apache.org/jira/browse/NIFI-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13986: Fix Version/s: 2.1.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Jetty to 12.0.15 and Jackson to 2.18.1 along with others > > > Key: NIFI-13986 > URL: https://issues.apache.org/jira/browse/NIFI-13986 > Project: Apache NiFi > Issue Type: Task >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.1.0 > > Time Spent: 20m > Remaining Estimate: 0h > > * kafka-clients from 3.8.0 to 3.9.0 > * jetty from 12.0.14 to 12.0.15 > * jackson from 2.18.0 to 2.18.1 > * exec-maven-plugin from 3.4.1 to 3.5.0 > * docker-maven-plugin from 0.43.4 to 0.45.1 > * maven-checkstyle from 3.5.0 to 3.6.0 > * checkstyle from 10.18.2 to 10.20.0 > * swagger-codegen from 3.0.63 to 3.0.64 > * dependency-check from 10.0.4 to 11.1.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13986) Upgrade Jetty to 12.0.15 and Jackson to 2.18.1 along with others
[ https://issues.apache.org/jira/browse/NIFI-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13986: Summary: Upgrade Jetty to 12.0.15 and Jackson to 2.18.1 along with others (was: Upgrade dep. jackson, jetty, kafka and plugins) > Upgrade Jetty to 12.0.15 and Jackson to 2.18.1 along with others > > > Key: NIFI-13986 > URL: https://issues.apache.org/jira/browse/NIFI-13986 > Project: Apache NiFi > Issue Type: Task >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > * kafka-clients from 3.8.0 to 3.9.0 > * jetty from 12.0.14 to 12.0.15 > * jackson from 2.18.0 to 2.18.1 > * exec-maven-plugin from 3.4.1 to 3.5.0 > * docker-maven-plugin from 0.43.4 to 0.45.1 > * maven-checkstyle from 3.5.0 to 3.6.0 > * checkstyle from 10.18.2 to 10.20.0 > * swagger-codegen from 3.0.63 to 3.0.64 > * dependency-check from 10.0.4 to 11.1.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIP-1) Add VerifiableRegistryClient interface to verify Registry Client implementations
[ https://issues.apache.org/jira/browse/NIP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896422#comment-17896422 ] David Handermann commented on NIP-1: It sounds like the last piece remaining is for concurrence from a frontend development perspective. If there is a commitment to do the frontend work, then it sounds like everything is covered. > Add VerifiableRegistryClient interface to verify Registry Client > implementations > > > Key: NIP-1 > URL: https://issues.apache.org/jira/browse/NIP-1 > Project: NiFi Improvement Proposal > Issue Type: New Feature >Reporter: Matt Burgess >Priority: Normal > Labels: nifi-registry > > Motivation > To verify basic connectivity from a NiFiRegistryClient to a Registry instance > Scope > Add VerifiableRegistryClient interface to verify NiFiRegistryClient > implementations > > Compatibility > Existing implementations in Apache NiFi will need to implement the verify() > method, but a default no-op implementation will be offered so as not to > interrupt compatibility. The no-op implementation should log/report that no > actual verification was applied. > Description > Registry Clients didn't exist when the Verifiable (ControllerService, > Processor, ReportingTask, ParameterProvider, etc.) interfaces were added. > Verification > Implement the verify method for JerseyNiFiRegistryClient (contributed as a > separate Jira) and run with a live instance, validating that incorrect > configurations return error message(s) and a correct configuration returns > success. Fetching the available buckets should suffice for verification, > whether buckets exist or not. > Alternatives > None identified, current behavior does not allow verification of > NiFiRegistryClient instances -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIP-1) Add VerifiableRegistryClient interface to verify Registry Client implementations
[ https://issues.apache.org/jira/browse/NIP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIP-1: --- Labels: nifi-api nifi-registry (was: nifi-registry) > Add VerifiableRegistryClient interface to verify Registry Client > implementations > > > Key: NIP-1 > URL: https://issues.apache.org/jira/browse/NIP-1 > Project: NiFi Improvement Proposal > Issue Type: New Feature >Reporter: Matt Burgess >Priority: Normal > Labels: nifi-api, nifi-registry > > Motivation > To verify basic connectivity from a NiFiRegistryClient to a Registry instance > Scope > Add VerifiableRegistryClient interface to verify NiFiRegistryClient > implementations > > Compatibility > Existing implementations in Apache NiFi will need to implement the verify() > method, but a default no-op implementation will be offered so as not to > interrupt compatibility. The no-op implementation should log/report that no > actual verification was applied. > Description > Registry Clients didn't exist when the Verifiable (ControllerService, > Processor, ReportingTask, ParameterProvider, etc.) interfaces were added. > Verification > Implement the verify method for JerseyNiFiRegistryClient (contributed as a > separate Jira) and run with a live instance, validating that incorrect > configurations return error message(s) and a correct configuration returns > success. Fetching the available buckets should suffice for verification, > whether buckets exist or not. > Alternatives > None identified, current behavior does not allow verification of > NiFiRegistryClient instances -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13983) Some java artifcats still missing for 2.0.0 release
[ https://issues.apache.org/jira/browse/NIFI-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896331#comment-17896331 ] David Handermann commented on NIFI-13983: - [~rubenvw] NIFI-13947 lists a number of modules removed from the 2.0.0 version, including the legacy Kerberos Credentials Service API and the Graphite Metric Reporter Service. It looks like these should also be added to the main list of Deprecated Components and Features for completeness. https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features The Kerberos User Service API is the current supported solution for Kerberos integration. The NiFi REST API includes Prometheus and JSON-based metrics, without the need for a separate reporting task. > Some java artifcats still missing for 2.0.0 release > --- > > Key: NIFI-13983 > URL: https://issues.apache.org/jira/browse/NIFI-13983 > Project: Apache NiFi > Issue Type: Task >Reporter: Ruben Van Wanzeele >Priority: Minor > > I cannot fine the following artifacts with the 2.0.0 release: > * nifi-kerberos-credentials-service-api > ** latest one I find is the 2.0.0-M4 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-kerberos-credentials-service-api] > * nifi-metrics-reporter-service-api > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporter-service-api] > * nifi-metrics-reporting-task > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporting-task] > Is this functionality dropped / replaced with something else -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13983) Some java artifcats still missing for 2.0.0 release
[ https://issues.apache.org/jira/browse/NIFI-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13983: Priority: Minor (was: Critical) > Some java artifcats still missing for 2.0.0 release > --- > > Key: NIFI-13983 > URL: https://issues.apache.org/jira/browse/NIFI-13983 > Project: Apache NiFi > Issue Type: Bug >Reporter: Ruben Van Wanzeele >Priority: Minor > > I cannot fine the following artifacts with the 2.0.0 release: > * nifi-kerberos-credentials-service-api > ** latest one I find is the 2.0.0-M4 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-kerberos-credentials-service-api] > * nifi-metrics-reporter-service-api > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporter-service-api] > * nifi-metrics-reporting-task > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporting-task] > Is this functionality dropped / replaced with something else -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13983) Some java artifcats still missing for 2.0.0 release
[ https://issues.apache.org/jira/browse/NIFI-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13983: Issue Type: Task (was: Bug) > Some java artifcats still missing for 2.0.0 release > --- > > Key: NIFI-13983 > URL: https://issues.apache.org/jira/browse/NIFI-13983 > Project: Apache NiFi > Issue Type: Task >Reporter: Ruben Van Wanzeele >Priority: Minor > > I cannot fine the following artifacts with the 2.0.0 release: > * nifi-kerberos-credentials-service-api > ** latest one I find is the 2.0.0-M4 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-kerberos-credentials-service-api] > * nifi-metrics-reporter-service-api > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporter-service-api] > * nifi-metrics-reporting-task > ** latest one I find is 1.28.0 > ** > [https://mvnrepository.com/artifact/org.apache.nifi/nifi-metrics-reporting-task] > Is this functionality dropped / replaced with something else -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13985) StandardSSLContextService error for when keystore file is not provided
[
https://issues.apache.org/jira/browse/NIFI-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896364#comment-17896364
]
David Handermann commented on NIFI-13985:
-
Thanks for flagging this issue [~markbean]. If you are planning on putting
together a pull request, glad to collaborate on a review.
> StandardSSLContextService error for when keystore file is not provided
> --
>
> Key: NIFI-13985
> URL: https://issues.apache.org/jira/browse/NIFI-13985
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Mark Bean
>Assignee: Mark Bean
>Priority: Major
>
> Currently, when the Keystore file is not provided in the
> StandardSSLContextService configuration, a TrustManager not be created.
> {{252 final TrustManager trustManager;}}
> {{253 final String trustStoreFile = {*}getKeyStoreFile{*}();}}
> {{254 if (trustStoreFile == null || trustStoreFile.isBlank()) {}}
> {{255 getLogger().debug("Trust Store File not configured");}}
> {{256 } else {}}
> {{257 trustManager = createTrustManager();}}
> {{258 sslContextBuilder.trustManager(trustManager);}}
> {{259 }}}
> Line 254 should be:
> {{ final String trustStoreFile = {*}getTrustStoreFile{*}();}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13922) SplitExcel appears to drop the values of a column which are from a calculated formula
[
https://issues.apache.org/jira/browse/NIFI-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13922:
Fix Version/s: 1.29.0
1.28.1
> SplitExcel appears to drop the values of a column which are from a calculated
> formula
> --
>
> Key: NIFI-13922
> URL: https://issues.apache.org/jira/browse/NIFI-13922
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.27.0, 2.0.0-M4
>Reporter: Daniel Stieglitz
>Assignee: Daniel Stieglitz
>Priority: Major
> Fix For: 2.0.0, 1.29.0, 1.28.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When reading the SalesData.xlsx referred to in NIFI-13418 with a flow which
> consists of
> GetFile -> SplitExcel -> ConvertRecord where the ConvertRecord is configured
> with an ExcelReader and a CsvrecordSetWriter it appears the values for column
> 'Sale_amt' which is a formula (column F * column G) is completely dropped as
> seen in the sample below with column headers and the first row
> {code:java}
> OrderDate,Region,Manager,SalesMan,Item,Units,Unit_price,Sale_amt
> 43106,East,Martha,Alexander,Television,95,1198,
> {code}
> Although when reading one of the tabs with a flow which consists of GetFile
> -> ConvertRecord the column is not dropped e.g. of the column names and the
> first row
> {code:java}
> OrderDate,Region,Manager,SalesMan,Item,Units,Unit_price,Sale_amt
> 151519680,East,Martha,Alexander,Television,95,1198,113810.0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-3364) Sort Bootstrap java.arg properties in order
[
https://issues.apache.org/jira/browse/NIFI-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-3364.
Fix Version/s: 2.1.0
Resolution: Fixed
> Sort Bootstrap java.arg properties in order
> ---
>
> Key: NIFI-3364
> URL: https://issues.apache.org/jira/browse/NIFI-3364
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 0.8.0, 0.7.1, 1.2.0, 1.1.1, 1.0.1
> Environment: Any and all platforms and environments.
>Reporter: Russell Bateman
>Assignee: Jim Steinebrey
>Priority: Minor
> Labels: easyfix, patch
> Fix For: 2.1.0
>
> Attachments: RunNiFi.java
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In albeit rare cases, JVM argument ordering is important. The present
> implementation of the sense of java.arg.N leaves N ignored in the final list
> when presented to the JVM at launch.
> Two examples of insurmountably critical ordering are the use the Java Flight
> Recorder and the JMX remote arguments in support of using Java Mission
> Control remotely to connect to and profile (what's being done by) NiFi.
> Attached is a coded solution for replacing this block of code in RunNiFi.java:
> for (final Map.Entry entry : props.entrySet()) {
> final String key = entry.getKey();
> final String value = entry.getValue();
>
> if (key.startsWith("java.arg")) {
> javaAdditionalArgs.add(value);
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-3364) Sort Bootstrap java.arg properties in order
[
https://issues.apache.org/jira/browse/NIFI-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-3364:
---
Summary: Sort Bootstrap java.arg properties in order (was: NiFi start-up
fails to honor N in definitions java.arg.N as made in conf/bootstrap.conf)
> Sort Bootstrap java.arg properties in order
> ---
>
> Key: NIFI-3364
> URL: https://issues.apache.org/jira/browse/NIFI-3364
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 0.8.0, 0.7.1, 1.2.0, 1.1.1, 1.0.1
> Environment: Any and all platforms and environments.
>Reporter: Russell Bateman
>Assignee: Jim Steinebrey
>Priority: Minor
> Labels: easyfix, patch
> Attachments: RunNiFi.java
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In albeit rare cases, JVM argument ordering is important. The present
> implementation of the sense of java.arg.N leaves N ignored in the final list
> when presented to the JVM at launch.
> Two examples of insurmountably critical ordering are the use the Java Flight
> Recorder and the JMX remote arguments in support of using Java Mission
> Control remotely to connect to and profile (what's being done by) NiFi.
> Attached is a coded solution for replacing this block of code in RunNiFi.java:
> for (final Map.Entry entry : props.entrySet()) {
> final String key = entry.getKey();
> final String value = entry.getValue();
>
> if (key.startsWith("java.arg")) {
> javaAdditionalArgs.add(value);
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[ https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896084#comment-17896084 ] David Handermann commented on NIFI-13972: - Thanks for the review and verification [~turcsanyip]! > ObjectLocalDateTimeFieldConverter.convertField ignores time zone > > > Key: NIFI-13972 > URL: https://issues.apache.org/jira/browse/NIFI-13972 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When the incoming value is a string that contains timezone/offset information > the returned LocalDateTime simply ignores that part and returns a value as if > it was in the local timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[
https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896054#comment-17896054
]
David Handermann commented on NIFI-13972:
-
For reference, the following simple method runs the string to LocalDateTime
conversion:
{noformat}
public static void main(final String[] args) {
final ObjectLocalDateTimeFieldConverter converter = new
ObjectLocalDateTimeFieldConverter();
final String string = "2020-01-01 12:00:00.123456789";
final Optional pattern = Optional.of("-MM-dd
HH:mm:ss.S");
final long started = System.currentTimeMillis();
final long iterations = 10_000_000L;
for (long i = 0; i < iterations; i++) {
converter.convertField(string, pattern, "fieldName");
}
final long elapsed = System.currentTimeMillis() - started;
System.out.printf("Elapsed: %d ms%n", elapsed);
}
{noformat}
Running that on the current main branch takes around 11000 ms on an Apple M3
Max. Running with a time zone takes around 4500 ms.
Running that with the pull request for NIFI-13978 using the TemporalQuery
approach cuts the above time to around 2500 ms, and around 4000 ms with a time
zone.
> ObjectLocalDateTimeFieldConverter.convertField ignores time zone
>
>
> Key: NIFI-13972
> URL: https://issues.apache.org/jira/browse/NIFI-13972
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
>
> When the incoming value is a string that contains timezone/offset information
> the returned LocalDateTime simply ignores that part and returns a value as if
> it was in the local timezone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13979) Upgrade happy-dom to 15.10.1
[ https://issues.apache.org/jira/browse/NIFI-13979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13979: Status: Patch Available (was: Open) > Upgrade happy-dom to 15.10.1 > > > Key: NIFI-13979 > URL: https://issues.apache.org/jira/browse/NIFI-13979 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: David Handermann >Priority: Minor > > The happy-dom library is a development dependency of the web user interface, > not shipped in any runntime components. The library should be upgraded to > 15.10.1 to resolve an security vulnerability finding in earlier versions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13979) Upgrade happy-dom to 15.10.1
David Handermann created NIFI-13979: --- Summary: Upgrade happy-dom to 15.10.1 Key: NIFI-13979 URL: https://issues.apache.org/jira/browse/NIFI-13979 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: David Handermann The happy-dom library is a development dependency of the web user interface, not shipped in any runntime components. The library should be upgraded to 15.10.1 to resolve an security vulnerability finding in earlier versions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[ https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896038#comment-17896038 ] David Handermann commented on NIFI-13972: - Thanks [~turcsanyip]. I created a separate issue NIFI-13978 and submitted a pull request with proposed changes. After initial evaluation, the parseUnresolved method approach did not prove to be sufficient, as the object resolution process involves more steps within the DateTimeFormatter. Instead of that approach, I implemented a custom TemporalQuery that looks for the presence of a ZoneId in the parsed TemporalAccessor. Using this TemporalQuery avoids the try-catch loop of parseBest() and enables resolution based on the ZonedDateTime or LocalDateTime as appropriate to the parsed set of fields. Glad to compare approaches with that background. > ObjectLocalDateTimeFieldConverter.convertField ignores time zone > > > Key: NIFI-13972 > URL: https://issues.apache.org/jira/browse/NIFI-13972 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When the incoming value is a string that contains timezone/offset information > the returned LocalDateTime simply ignores that part and returns a value as if > it was in the local timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13978) Improve Performance of Record Date Time Parsing
[
https://issues.apache.org/jira/browse/NIFI-13978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13978:
Status: Patch Available (was: Open)
> Improve Performance of Record Date Time Parsing
> ---
>
> Key: NIFI-13978
> URL: https://issues.apache.org/jira/browse/NIFI-13978
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recent changes to handle time zones when parsing strings to LocalDateTime
> objects for Record processing resulted in sub-optimal performance.
> Specifically, parsing strings with timestamp patterns that did not include a
> time zone require two object resolution attempts due to the use of
> {{DateTimeFormatter.parseBest()}} with {{ZonedDateTime::from}} and then
> {{LocalDateTime::from}}.
> Instead of relying on multiple passes try-catch handling in {{parseBest()}},
> using the standard {{parse()}} method with a custom {{TemporalQuery}} avoids
> multiple passes. The {{parse()}} method handles parsing a string to a map of
> fields, and then calls the provided {{TemporalQuery}} to resolve the
> requested object type. The {{TemporalAccessor}} can be queried for a
> {{ZoneId}} to determine whether to resolve the object as a {{LocalDateTime}}
> or a {{ZonedDateTime}} for subsequent conversion to a {{LocalDateTime}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13978) Improve Performance of Record Date Time Parsing
David Handermann created NIFI-13978:
---
Summary: Improve Performance of Record Date Time Parsing
Key: NIFI-13978
URL: https://issues.apache.org/jira/browse/NIFI-13978
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: David Handermann
Assignee: David Handermann
Recent changes to handle time zones when parsing strings to LocalDateTime
objects for Record processing resulted in sub-optimal performance.
Specifically, parsing strings with timestamp patterns that did not include a
time zone require two object resolution attempts due to the use of
{{DateTimeFormatter.parseBest()}} with {{ZonedDateTime::from}} and then
{{LocalDateTime::from}}.
Instead of relying on multiple passes try-catch handling in {{parseBest()}},
using the standard {{parse()}} method with a custom {{TemporalQuery}} avoids
multiple passes. The {{parse()}} method handles parsing a string to a map of
fields, and then calls the provided {{TemporalQuery}} to resolve the requested
object type. The {{TemporalAccessor}} can be queried for a {{ZoneId}} to
determine whether to resolve the object as a {{LocalDateTime}} or a
{{ZonedDateTime}} for subsequent conversion to a {{LocalDateTime}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[ https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896008#comment-17896008 ] David Handermann commented on NIFI-13972: - Thanks for highlighting the parseBest() behavior [~tpalfy] and running the performance comparison [~turcsanyip]. Setting a zone on the DateTimeFormatter instance could be problematic due to the shared registry of instances. The fundamental difficulty is attempting to determine whether the specified pattern includes a timezone component, given the variety of formatting approaches. Another approach would be to perform more granular time unit queries on an unresolved TemporalAccessor. The parseBest() method is a brute-force attempt to do that, but given the performance implications, this looks like it is worth revisiting. I think it should be possible include some logic that looks for a time zone component in the unresolved TemporalAccessor, and then decide whether to build a ZonedDateTime or LocalDateTime object. I will take a look at that approach to consider the implications, and I'm open to additional options. > ObjectLocalDateTimeFieldConverter.convertField ignores time zone > > > Key: NIFI-13972 > URL: https://issues.apache.org/jira/browse/NIFI-13972 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When the incoming value is a string that contains timezone/offset information > the returned LocalDateTime simply ignores that part and returns a value as if > it was in the local timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13970) Create a DISCLAIMER file for the 1.x line indicating support posture regarding key dependencies
[ https://issues.apache.org/jira/browse/NIFI-13970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13970. - Resolution: Fixed > Create a DISCLAIMER file for the 1.x line indicating support posture > regarding key dependencies > --- > > Key: NIFI-13970 > URL: https://issues.apache.org/jira/browse/NIFI-13970 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.29.0, 1.28.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13878) Implement an option in nifi.properties that forces authentication with OpenID/SSO
[ https://issues.apache.org/jira/browse/NIFI-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895748#comment-17895748 ] David Handermann commented on NIFI-13878: - Sounds good [~deragon], I agree that leaving this open for now is worthwhile, with the potential to revisit the overall security filter structure down the road. > Implement an option in nifi.properties that forces authentication with > OpenID/SSO > - > > Key: NIFI-13878 > URL: https://issues.apache.org/jira/browse/NIFI-13878 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Affects Versions: 2.0.0-M4 > Environment: RHEL 9.4 on AWS EC2 with Okta as Identity Provider. >Reporter: Hans Deragon >Priority: Major > Labels: OpenID, certificate > > This is a feature request to implement an option in nifi.properties that > forces authentication with OpenID/SSO, or disable X.509 certificate use for > authentication. > We discovered with NiFi 2.0.0-M4 that if a personal X.509 certificate is set > in user accounts under Windows, that certificate is getting used by NiFi for > authentication instead of the normal OpenID/SSO headers. The user id in the > X.509 certificate is not the same as the one in OpenID/SSO (Okta) and thus, > the person is denied access to NiFi. > This particular certificate is not meant to be used by NiFi to authenticate > and authorize users in NiFi even though it is recognized by our Identity > Provider. We desire that NiFi only authenticate and authorize users with > OpenID/SSO (which works when I remove the personal certificate from user's > Windows workstations). > This issue has been discussed in NiFi User's mailing list but nobody seams to > come up with a solution to our problem, thus this Jira. > See: [NiFi 2.0 and possibility to ignore X.509 certificate or force > authorization with > OpenID/SSO.|https://lists.apache.org/thread/yphnmj27dpkpdm548mt7dd0jkbjn3wch] > More details: > * In > nifi-framework-bundle/.../web/security/x509/X509AuthenticationFilter.java, I > commented out code in attemptAuthentication() so it always returns null and > that works (of course). > * I attempted to play with the order of addFilterBefore() in > nifi-framework-bundle/.../web/security/configuration/WebSecurityConfiguration.java > to hope that OpenID/SSO would always be > tested first, without success. > * I had one machine (an AWS EC2) that OpenID/SSO always worked despite the > certificate. I do not understand why that happened and of course, I compared > the conf/ directory to another EC2 which uses the certificate instead to > authenticate the user and found not significant difference, nothing related > to this topic. I was using Chrome when this happened and clearing the > cookies did not change the behavior. Using Edge to access the same machine > would cause the certificate being used. Very odd behavior which I cannot > explain. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13878) Implement an option in nifi.properties that forces authentication with OpenID/SSO
[ https://issues.apache.org/jira/browse/NIFI-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13878: Affects Version/s: (was: 2.0.0-M4) > Implement an option in nifi.properties that forces authentication with > OpenID/SSO > - > > Key: NIFI-13878 > URL: https://issues.apache.org/jira/browse/NIFI-13878 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration > Environment: RHEL 9.4 on AWS EC2 with Okta as Identity Provider. >Reporter: Hans Deragon >Priority: Major > Labels: OpenID, certificate > > This is a feature request to implement an option in nifi.properties that > forces authentication with OpenID/SSO, or disable X.509 certificate use for > authentication. > We discovered with NiFi 2.0.0-M4 that if a personal X.509 certificate is set > in user accounts under Windows, that certificate is getting used by NiFi for > authentication instead of the normal OpenID/SSO headers. The user id in the > X.509 certificate is not the same as the one in OpenID/SSO (Okta) and thus, > the person is denied access to NiFi. > This particular certificate is not meant to be used by NiFi to authenticate > and authorize users in NiFi even though it is recognized by our Identity > Provider. We desire that NiFi only authenticate and authorize users with > OpenID/SSO (which works when I remove the personal certificate from user's > Windows workstations). > This issue has been discussed in NiFi User's mailing list but nobody seams to > come up with a solution to our problem, thus this Jira. > See: [NiFi 2.0 and possibility to ignore X.509 certificate or force > authorization with > OpenID/SSO.|https://lists.apache.org/thread/yphnmj27dpkpdm548mt7dd0jkbjn3wch] > More details: > * In > nifi-framework-bundle/.../web/security/x509/X509AuthenticationFilter.java, I > commented out code in attemptAuthentication() so it always returns null and > that works (of course). > * I attempted to play with the order of addFilterBefore() in > nifi-framework-bundle/.../web/security/configuration/WebSecurityConfiguration.java > to hope that OpenID/SSO would always be > tested first, without success. > * I had one machine (an AWS EC2) that OpenID/SSO always worked despite the > certificate. I do not understand why that happened and of course, I compared > the conf/ directory to another EC2 which uses the certificate instead to > authenticate the user and found not significant difference, nothing related > to this topic. I was using Chrome when this happened and clearing the > cookies did not change the behavior. Using Edge to access the same machine > would cause the certificate being used. Very odd behavior which I cannot > explain. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[
https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895747#comment-17895747
]
David Handermann commented on NIFI-13972:
-
The {{DateTuneFormatter.parseBest()}} method does try each of the Temporal
Queries supplied, but those attempts work against a parsed TemporalAccessor, so
the amount of additional work is minimized.
> ObjectLocalDateTimeFieldConverter.convertField ignores time zone
>
>
> Key: NIFI-13972
> URL: https://issues.apache.org/jira/browse/NIFI-13972
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
>
> When the incoming value is a string that contains timezone/offset information
> the returned LocalDateTime simply ignores that part and returns a value as if
> it was in the local timezone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[ https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895743#comment-17895743 ] David Handermann commented on NIFI-13972: - You're welcome, thanks for confirming! > ObjectLocalDateTimeFieldConverter.convertField ignores time zone > > > Key: NIFI-13972 > URL: https://issues.apache.org/jira/browse/NIFI-13972 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When the incoming value is a string that contains timezone/offset information > the returned LocalDateTime simply ignores that part and returns a value as if > it was in the local timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13972) ObjectLocalDateTimeFieldConverter.convertField ignores time zone
[ https://issues.apache.org/jira/browse/NIFI-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895732#comment-17895732 ] David Handermann commented on NIFI-13972: - [~tpalfy] Changes in NIFI-13579 improved this behavior, are you seeing this issue in a particular version? > ObjectLocalDateTimeFieldConverter.convertField ignores time zone > > > Key: NIFI-13972 > URL: https://issues.apache.org/jira/browse/NIFI-13972 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When the incoming value is a string that contains timezone/offset information > the returned LocalDateTime simply ignores that part and returns a value as if > it was in the local timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13971) Correct Parameter Context Logging on Flow Synchronization
[ https://issues.apache.org/jira/browse/NIFI-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13971: Fix Version/s: (was: 1.29.0) > Correct Parameter Context Logging on Flow Synchronization > - > > Key: NIFI-13971 > URL: https://issues.apache.org/jira/browse/NIFI-13971 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 1.28.1 > > Time Spent: 10m > Remaining Estimate: 0h > > The Flow Synchronization process includes several levels of logging for > informational and debugging purposes, including details related to Parameter > Context updates. Logging Parameter Context information includes unnecessary > details, which should be removed. Recent changes to Parameter Provider > integration on the main branch included these improvements, and the relevant > logging changes should be backported as part of the finalization process for > the support branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13971) Correct Parameter Context Logging on Flow Synchronization
[ https://issues.apache.org/jira/browse/NIFI-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13971: Status: Patch Available (was: Open) > Correct Parameter Context Logging on Flow Synchronization > - > > Key: NIFI-13971 > URL: https://issues.apache.org/jira/browse/NIFI-13971 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 1.29.0, 1.28.1 > > Time Spent: 10m > Remaining Estimate: 0h > > The Flow Synchronization process includes several levels of logging for > informational and debugging purposes, including details related to Parameter > Context updates. Logging Parameter Context information includes unnecessary > details, which should be removed. Recent changes to Parameter Provider > integration on the main branch included these improvements, and the relevant > logging changes should be backported as part of the finalization process for > the support branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13971) Correct Parameter Context Logging on Flow Synchronization
David Handermann created NIFI-13971: --- Summary: Correct Parameter Context Logging on Flow Synchronization Key: NIFI-13971 URL: https://issues.apache.org/jira/browse/NIFI-13971 Project: Apache NiFi Issue Type: Bug Reporter: David Handermann Assignee: David Handermann Fix For: 1.29.0, 1.28.1 The Flow Synchronization process includes several levels of logging for informational and debugging purposes, including details related to Parameter Context updates. Logging Parameter Context information includes unnecessary details, which should be removed. Recent changes to Parameter Provider integration on the main branch included these improvements, and the relevant logging changes should be backported as part of the finalization process for the support branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12836) Connection pool shut down and SocketException for many AWS processors
[ https://issues.apache.org/jira/browse/NIFI-12836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-12836: --- Assignee: David Handermann > Connection pool shut down and SocketException for many AWS processors > - > > Key: NIFI-12836 > URL: https://issues.apache.org/jira/browse/NIFI-12836 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lehel Boér >Assignee: David Handermann >Priority: Major > > Reported encountering "Connection pool shut down" errors for the > PutDynamoDBRecord, DeleteDynamoDB, and PutSQS processors while running in a > NiFi cluster on version 2.0.0-M1. > * [Stack Trace reported on the mailing list|https://codefile.io/f/ZMXYzHt89X] > https://lists.apache.org/thread/5fbtwk68yr4bcxpp2h2mtzwy0566rfqz > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-3364) NiFi start-up fails to honor N in definitions java.arg.N as made in conf/bootstrap.conf
[
https://issues.apache.org/jira/browse/NIFI-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-3364:
---
Priority: Minor (was: Critical)
> NiFi start-up fails to honor N in definitions java.arg.N as made in
> conf/bootstrap.conf
> ---
>
> Key: NIFI-3364
> URL: https://issues.apache.org/jira/browse/NIFI-3364
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 0.8.0, 0.7.1, 1.2.0, 1.1.1, 1.0.1
> Environment: Any and all platforms and environments.
>Reporter: Russell Bateman
>Assignee: Jim Steinebrey
>Priority: Minor
> Labels: easyfix, patch
> Attachments: RunNiFi.java
>
>
> In albeit rare cases, JVM argument ordering is important. The present
> implementation of the sense of java.arg.N leaves N ignored in the final list
> when presented to the JVM at launch.
> Two examples of insurmountably critical ordering are the use the Java Flight
> Recorder and the JMX remote arguments in support of using Java Mission
> Control remotely to connect to and profile (what's being done by) NiFi.
> Attached is a coded solution for replacing this block of code in RunNiFi.java:
> for (final Map.Entry entry : props.entrySet()) {
> final String key = entry.getKey();
> final String value = entry.getValue();
>
> if (key.startsWith("java.arg")) {
> javaAdditionalArgs.add(value);
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-13965) Reduce duplicate code in TestConvertRecord and make use of Java 21 features
[ https://issues.apache.org/jira/browse/NIFI-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13965. - Fix Version/s: 2.1.0 Resolution: Fixed > Reduce duplicate code in TestConvertRecord and make use of Java 21 features > --- > > Key: NIFI-13965 > URL: https://issues.apache.org/jira/browse/NIFI-13965 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Daniel Stieglitz >Assignee: Daniel Stieglitz >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Intellij is indicating the loading of the same json and schemas are loaded > multiple times. These should be done once. Also other Java 21 features > Intellij suggested should be made (e.g use of getFirst instead of get(0), > text block instead of concatenation etc). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13966) Remove unsupported security protocols from SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13966: Fix Version/s: 2.1.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Remove unsupported security protocols from SNMP processors > -- > > Key: NIFI-13966 > URL: https://issues.apache.org/jira/browse/NIFI-13966 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 20m > Remaining Estimate: 0h > > NIFI-13943 upgraded SNMP4J client library from 2.x to 3.x where MD5 and SHA > authentication protocols are not enabled by default anymore because they are > insecure (https://www.snmp4j.org/CHANGES.txt). 3DES privacy protocol is not > enabled either. > Remove these protocols from the SNMP processors' configuration options. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13915) Conduct Release for Apache NiFi 2.0.0
[ https://issues.apache.org/jira/browse/NIFI-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13915. - Resolution: Fixed > Conduct Release for Apache NiFi 2.0.0 > - > > Key: NIFI-13915 > URL: https://issues.apache.org/jira/browse/NIFI-13915 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (NIFI-13962) Overrride trusted hostname verification
[ https://issues.apache.org/jira/browse/NIFI-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann closed NIFI-13962. --- > Overrride trusted hostname verification > > > Key: NIFI-13962 > URL: https://issues.apache.org/jira/browse/NIFI-13962 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pedro Oliveira >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add a property "Trusted Hostname" to the configuration of the > StandardSSLContextService Controller Service to allow to override the trusted > hostname verification on the establishment of the HTTP connection to the HTTP > server. > This property if not empty should be set with the hostname and will be used > to validate the hostname in the certificate. > The property is need because there are situations in which the hostname in > the certificate is different from the hostname of the server. > This occurs when the server returns a certificate with an old hostname > because it is not possible to generate a new certificate with the actual > hostname. > This property existed in NiFi old versions (<1.14) in the configuration of > the processor InvokeHTTP but was removed in later versions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13962) Overrride trusted hostname verification
[ https://issues.apache.org/jira/browse/NIFI-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13962. - Resolution: Won't Fix (was: Fixed) > Overrride trusted hostname verification > > > Key: NIFI-13962 > URL: https://issues.apache.org/jira/browse/NIFI-13962 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pedro Oliveira >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add a property "Trusted Hostname" to the configuration of the > StandardSSLContextService Controller Service to allow to override the trusted > hostname verification on the establishment of the HTTP connection to the HTTP > server. > This property if not empty should be set with the hostname and will be used > to validate the hostname in the certificate. > The property is need because there are situations in which the hostname in > the certificate is different from the hostname of the server. > This occurs when the server returns a certificate with an old hostname > because it is not possible to generate a new certificate with the actual > hostname. > This property existed in NiFi old versions (<1.14) in the configuration of > the processor InvokeHTTP but was removed in later versions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (NIFI-13962) Overrride trusted hostname verification
[ https://issues.apache.org/jira/browse/NIFI-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann closed NIFI-13962. --- > Overrride trusted hostname verification > > > Key: NIFI-13962 > URL: https://issues.apache.org/jira/browse/NIFI-13962 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pedro Oliveira >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add a property "Trusted Hostname" to the configuration of the > StandardSSLContextService Controller Service to allow to override the trusted > hostname verification on the establishment of the HTTP connection to the HTTP > server. > This property if not empty should be set with the hostname and will be used > to validate the hostname in the certificate. > The property is need because there are situations in which the hostname in > the certificate is different from the hostname of the server. > This occurs when the server returns a certificate with an old hostname > because it is not possible to generate a new certificate with the actual > hostname. > This property existed in NiFi old versions (<1.14) in the configuration of > the processor InvokeHTTP but was removed in later versions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13962) Overrride trusted hostname verification
[ https://issues.apache.org/jira/browse/NIFI-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13962: Status: Reopened (was: Closed) > Overrride trusted hostname verification > > > Key: NIFI-13962 > URL: https://issues.apache.org/jira/browse/NIFI-13962 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pedro Oliveira >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add a property "Trusted Hostname" to the configuration of the > StandardSSLContextService Controller Service to allow to override the trusted > hostname verification on the establishment of the HTTP connection to the HTTP > server. > This property if not empty should be set with the hostname and will be used > to validate the hostname in the certificate. > The property is need because there are situations in which the hostname in > the certificate is different from the hostname of the server. > This occurs when the server returns a certificate with an old hostname > because it is not possible to generate a new certificate with the actual > hostname. > This property existed in NiFi old versions (<1.14) in the configuration of > the processor InvokeHTTP but was removed in later versions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13962) Overrride trusted hostname verification
[ https://issues.apache.org/jira/browse/NIFI-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13962. - Resolution: Fixed Please see the discussion on NIFI-7064 for the reasons related to removing the trusted hostname property from InvokeHTTP. At this time, this type of change will not be considered for inclusion due to the inherent security risks associated with bypass TLS certificate verification. > Overrride trusted hostname verification > > > Key: NIFI-13962 > URL: https://issues.apache.org/jira/browse/NIFI-13962 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pedro Oliveira >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add a property "Trusted Hostname" to the configuration of the > StandardSSLContextService Controller Service to allow to override the trusted > hostname verification on the establishment of the HTTP connection to the HTTP > server. > This property if not empty should be set with the hostname and will be used > to validate the hostname in the certificate. > The property is need because there are situations in which the hostname in > the certificate is different from the hostname of the server. > This occurs when the server returns a certificate with an old hostname > because it is not possible to generate a new certificate with the actual > hostname. > This property existed in NiFi old versions (<1.14) in the configuration of > the processor InvokeHTTP but was removed in later versions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13957) Upgrade Snowflake JDBC to 3.20.0 and Snowflake Ingest to 2.3.0
[ https://issues.apache.org/jira/browse/NIFI-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13957. - Fix Version/s: 2.1.0 Resolution: Fixed > Upgrade Snowflake JDBC to 3.20.0 and Snowflake Ingest to 2.3.0 > -- > > Key: NIFI-13957 > URL: https://issues.apache.org/jira/browse/NIFI-13957 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The Snowflake JDBC driver should be upgraded to > [3.20.0|https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024] > in order to mitigate CVE-2024-43382 related to client-side encryption issues > for Azure and GCP with staged files and non-default key sizes. > The Snowflake Ingest Java library indicates compatibility with JDBC 3.16.0 or > higher. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13958) Upgrade elliptic to 6.6.0 for Registry
[ https://issues.apache.org/jira/browse/NIFI-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13958. - Fix Version/s: 2.1.0 Resolution: Fixed > Upgrade elliptic to 6.6.0 for Registry > -- > > Key: NIFI-13958 > URL: https://issues.apache.org/jira/browse/NIFI-13958 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The [elliptic|https://github.com/indutny/elliptic] library is a transitive > dependency of Webpack for NiFi Registry, used during development builds, and > should be upgraded to 6.6.0 to mitigate CVE-2024-48948 related to signature > verification. As this is a development build dependency, it does not impact > any runtime behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13957) Upgrade Snowflake JDBC to 3.20.0 and Snowflake Ingest to 2.3.0
[ https://issues.apache.org/jira/browse/NIFI-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13957: Summary: Upgrade Snowflake JDBC to 3.20.0 and Snowflake Ingest to 2.3.0 (was: Upgrade Snowflake JDBC to 3.20.0) > Upgrade Snowflake JDBC to 3.20.0 and Snowflake Ingest to 2.3.0 > -- > > Key: NIFI-13957 > URL: https://issues.apache.org/jira/browse/NIFI-13957 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > > The Snowflake JDBC driver should be upgraded to > [3.20.0|https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024] > in order to mitigate CVE-2024-43382 related to client-side encryption issues > for Azure and GCP with staged files and non-default key sizes. > The Snowflake Ingest Java library indicates compatibility with JDBC 3.16.0 or > higher. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-13958) Upgrade elliptic to 6.6.0 for Registry
[ https://issues.apache.org/jira/browse/NIFI-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-13958: --- Assignee: David Handermann > Upgrade elliptic to 6.6.0 for Registry > -- > > Key: NIFI-13958 > URL: https://issues.apache.org/jira/browse/NIFI-13958 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > > The [elliptic|https://github.com/indutny/elliptic] library is a transitive > dependency of Webpack for NiFi Registry, used during development builds, and > should be upgraded to 6.6.0 to mitigate CVE-2024-48948 related to signature > verification. As this is a development build dependency, it does not impact > any runtime behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13956) Upgrade Angular to 18.2.11 or later
[ https://issues.apache.org/jira/browse/NIFI-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13956: Summary: Upgrade Angular to 18.2.11 or later (was: JS dependencies have security vulnerabilities ) > Upgrade Angular to 18.2.11 or later > --- > > Key: NIFI-13956 > URL: https://issues.apache.org/jira/browse/NIFI-13956 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 2.0.0-M4, 2.0.0 >Reporter: Dimitri John Ledkov >Priority: Major > > Can you please upgrade angularjs to latest minor point release as well as > http_proxy_middleware? Scanners are picking up that there are vulnerabilities. > > ``` > xnox@chainguard:/tmp/nifi/nifi-frontend/src/main/frontend$ npm audit > # npm audit report > > http-proxy-middleware 3.0.0 - 3.0.2 > Severity: high > Denial of service in http-proxy-middleware - > https://github.com/advisories/GHSA-c7qv-q95q-8v27 > fix available via `npm audit fix --force` > Will install @angular-devkit/build-angular@18.2.10, which is outside the > stated dependency range > node_modules/http-proxy-middleware > @angular-devkit/build-angular 18.0.0-next.0 - 18.2.9 || 19.0.0-next.0 - > 19.0.0-next.9 > Depends on vulnerable versions of http-proxy-middleware > node_modules/@angular-devkit/build-angular > > 2 high severity vulnerabilities > > To address all issues, run: > npm audit fix --force > ``` > > Note usually dependabot can help with these, and it is a good practice to run > `npm audit` prior to cutting a release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13958) Upgrade elliptic to 6.6.0 for Registry
David Handermann created NIFI-13958: --- Summary: Upgrade elliptic to 6.6.0 for Registry Key: NIFI-13958 URL: https://issues.apache.org/jira/browse/NIFI-13958 Project: Apache NiFi Issue Type: Improvement Components: NiFi Registry Reporter: David Handermann The [elliptic|https://github.com/indutny/elliptic] library is a transitive dependency of Webpack for NiFi Registry, used during development builds, and should be upgraded to 6.6.0 to mitigate CVE-2024-48948 related to signature verification. As this is a development build dependency, it does not impact any runtime behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13957) Upgrade Snowflake JDBC to 3.20.0
[ https://issues.apache.org/jira/browse/NIFI-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13957: Summary: Upgrade Snowflake JDBC to 3.20.0 (was: Upgraded Snowflake JDBC to 3.20.0) > Upgrade Snowflake JDBC to 3.20.0 > > > Key: NIFI-13957 > URL: https://issues.apache.org/jira/browse/NIFI-13957 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > > The Snowflake JDBC driver should be upgraded to > [3.20.0|https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024] > in order to mitigate CVE-2024-43382 related to client-side encryption issues > for Azure and GCP with staged files and non-default key sizes. > The Snowflake Ingest Java library indicates compatibility with JDBC 3.16.0 or > higher. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13956) Upgrade Angular to 18.2.11 or later
[ https://issues.apache.org/jira/browse/NIFI-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894854#comment-17894854 ] David Handermann commented on NIFI-13956: - For tracking, I updated the title to indicate that the Angular should be upgraded to the latest minor version, which includes the transitive dependency upgrade for http-proxy-middleware 3.0.3. As previously noted, http-proxy-middleware is a development dependency and is not included in any binary releases, so updating it can occur as part of standard dependency maintenance work. > Upgrade Angular to 18.2.11 or later > --- > > Key: NIFI-13956 > URL: https://issues.apache.org/jira/browse/NIFI-13956 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 2.0.0-M4, 2.0.0 >Reporter: Dimitri John Ledkov >Priority: Major > > Can you please upgrade angularjs to latest minor point release as well as > http_proxy_middleware? Scanners are picking up that there are vulnerabilities. > > ``` > xnox@chainguard:/tmp/nifi/nifi-frontend/src/main/frontend$ npm audit > # npm audit report > > http-proxy-middleware 3.0.0 - 3.0.2 > Severity: high > Denial of service in http-proxy-middleware - > https://github.com/advisories/GHSA-c7qv-q95q-8v27 > fix available via `npm audit fix --force` > Will install @angular-devkit/build-angular@18.2.10, which is outside the > stated dependency range > node_modules/http-proxy-middleware > @angular-devkit/build-angular 18.0.0-next.0 - 18.2.9 || 19.0.0-next.0 - > 19.0.0-next.9 > Depends on vulnerable versions of http-proxy-middleware > node_modules/@angular-devkit/build-angular > > 2 high severity vulnerabilities > > To address all issues, run: > npm audit fix --force > ``` > > Note usually dependabot can help with these, and it is a good practice to run > `npm audit` prior to cutting a release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13957) Upgraded Snowflake JDBC to 3.20.0
David Handermann created NIFI-13957: --- Summary: Upgraded Snowflake JDBC to 3.20.0 Key: NIFI-13957 URL: https://issues.apache.org/jira/browse/NIFI-13957 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: David Handermann Assignee: David Handermann The Snowflake JDBC driver should be upgraded to [3.20.0|https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024] in order to mitigate CVE-2024-43382 related to client-side encryption issues for Azure and GCP with staged files and non-default key sizes. The Snowflake Ingest Java library indicates compatibility with JDBC 3.16.0 or higher. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13956) Upgrade Angular to 18.2.11 or later
[ https://issues.apache.org/jira/browse/NIFI-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13956: Priority: Minor (was: Major) > Upgrade Angular to 18.2.11 or later > --- > > Key: NIFI-13956 > URL: https://issues.apache.org/jira/browse/NIFI-13956 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 2.0.0-M4, 2.0.0 >Reporter: Dimitri John Ledkov >Priority: Minor > > Can you please upgrade angularjs to latest minor point release as well as > http_proxy_middleware? Scanners are picking up that there are vulnerabilities. > > ``` > xnox@chainguard:/tmp/nifi/nifi-frontend/src/main/frontend$ npm audit > # npm audit report > > http-proxy-middleware 3.0.0 - 3.0.2 > Severity: high > Denial of service in http-proxy-middleware - > https://github.com/advisories/GHSA-c7qv-q95q-8v27 > fix available via `npm audit fix --force` > Will install @angular-devkit/build-angular@18.2.10, which is outside the > stated dependency range > node_modules/http-proxy-middleware > @angular-devkit/build-angular 18.0.0-next.0 - 18.2.9 || 19.0.0-next.0 - > 19.0.0-next.9 > Depends on vulnerable versions of http-proxy-middleware > node_modules/@angular-devkit/build-angular > > 2 high severity vulnerabilities > > To address all issues, run: > npm audit fix --force > ``` > > Note usually dependabot can help with these, and it is a good practice to run > `npm audit` prior to cutting a release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13956) Upgrade Angular to 18.2.11 or later
[ https://issues.apache.org/jira/browse/NIFI-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13956: Issue Type: Improvement (was: Bug) > Upgrade Angular to 18.2.11 or later > --- > > Key: NIFI-13956 > URL: https://issues.apache.org/jira/browse/NIFI-13956 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 2.0.0-M4, 2.0.0 >Reporter: Dimitri John Ledkov >Priority: Major > > Can you please upgrade angularjs to latest minor point release as well as > http_proxy_middleware? Scanners are picking up that there are vulnerabilities. > > ``` > xnox@chainguard:/tmp/nifi/nifi-frontend/src/main/frontend$ npm audit > # npm audit report > > http-proxy-middleware 3.0.0 - 3.0.2 > Severity: high > Denial of service in http-proxy-middleware - > https://github.com/advisories/GHSA-c7qv-q95q-8v27 > fix available via `npm audit fix --force` > Will install @angular-devkit/build-angular@18.2.10, which is outside the > stated dependency range > node_modules/http-proxy-middleware > @angular-devkit/build-angular 18.0.0-next.0 - 18.2.9 || 19.0.0-next.0 - > 19.0.0-next.9 > Depends on vulnerable versions of http-proxy-middleware > node_modules/@angular-devkit/build-angular > > 2 high severity vulnerabilities > > To address all issues, run: > npm audit fix --force > ``` > > Note usually dependabot can help with these, and it is a good practice to run > `npm audit` prior to cutting a release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-7421) Support SASL/OAUTHBEARER in Kafka processors
[ https://issues.apache.org/jira/browse/NIFI-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-7421: --- Description: Add support for SASL/OAUTHBEARER to the Kafka 3 processors which now have an allowable values list of SASL mechanisms. (was: Add support for SASL/OAUTHBEARER to the Kafka 2.x processors which now have an allowable values list of SASL mechanisms.) > Support SASL/OAUTHBEARER in Kafka processors > > > Key: NIFI-7421 > URL: https://issues.apache.org/jira/browse/NIFI-7421 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Priority: Major > > Add support for SASL/OAUTHBEARER to the Kafka 3 processors which now have an > allowable values list of SASL mechanisms. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13944) Upgrade Redis version to 5.2.0
[ https://issues.apache.org/jira/browse/NIFI-13944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13944: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Redis version to 5.2.0 > -- > > Key: NIFI-13944 > URL: https://issues.apache.org/jira/browse/NIFI-13944 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The current version of the Redis library is 3.10.0, which is compatible with > up to 6.2 which is EOL on Feb 2025. This should be updated to the latest > version 5.2.0 to support 6.4+. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13947) Remove unused nifi-security-utils Maven configuration
[
https://issues.apache.org/jira/browse/NIFI-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13947:
Status: Patch Available (was: Open)
> Remove unused nifi-security-utils Maven configuration
> -
>
> Key: NIFI-13947
> URL: https://issues.apache.org/jira/browse/NIFI-13947
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
>
> Following recent changes to TLS configuration, the {{nifi-security-utils}}
> module is no longer referenced, but updates did not include removing the
> Maven configuration pom.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13947) Remove unused nifi-security-utils Maven configuration
David Handermann created NIFI-13947:
---
Summary: Remove unused nifi-security-utils Maven configuration
Key: NIFI-13947
URL: https://issues.apache.org/jira/browse/NIFI-13947
Project: Apache NiFi
Issue Type: Bug
Reporter: David Handermann
Assignee: David Handermann
Fix For: 2.0.0
Following recent changes to TLS configuration, the {{nifi-security-utils}}
module is no longer referenced, but updates did not include removing the Maven
configuration pom.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13946) SNMP Integration Test Failures with SNMP4J 3
[ https://issues.apache.org/jira/browse/NIFI-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13946: Status: Patch Available (was: Open) > SNMP Integration Test Failures with SNMP4J 3 > > > Key: NIFI-13946 > URL: https://issues.apache.org/jira/browse/NIFI-13946 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > > SNMP4J version 3 introduced [several > changes|https://www.snmp4j.org/CHANGES.txt] that broke the expected behavior > of existing integration tests. These changes include defaulting to listen on > 127.0.0.1 and disabling SHA and MD5 algorithms for SNMP Version 3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13946) SNMP Integration Test Failures with SNMP4J 3
David Handermann created NIFI-13946: --- Summary: SNMP Integration Test Failures with SNMP4J 3 Key: NIFI-13946 URL: https://issues.apache.org/jira/browse/NIFI-13946 Project: Apache NiFi Issue Type: Bug Reporter: David Handermann Assignee: David Handermann Fix For: 2.0.0 SNMP4J version 3 introduced [several changes|https://www.snmp4j.org/CHANGES.txt] that broke the expected behavior of existing integration tests. These changes include defaulting to listen on 127.0.0.1 and disabling SHA and MD5 algorithms for SNMP Version 3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13843) Unknown fields not dropped by JSON Writer as expected by specified schema
[
https://issues.apache.org/jira/browse/NIFI-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13843:
Fix Version/s: 2.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Unknown fields not dropped by JSON Writer as expected by specified schema
> -
>
> Key: NIFI-13843
> URL: https://issues.apache.org/jira/browse/NIFI-13843
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.27.0, 2.0.0-M4
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Consider the following use case:
> * GFF Processor, generating a JSON with 3 fields: a, b, and c
> * ConvertRecord with JSON Reader / JSON Writer
> ** Both reader and writer are configured with a schema only specifying
> fields a and b
> The expected result is a JSON that only contains fields a and b.
> We're following the below path in the code:
> * AbstractRecordProcessor (L131)
> {code:java}
> Record firstRecord = reader.nextRecord(); {code}
> In this case, the default method for nextRecord() is defined in RecordReader
> (L50)
> {code:java}
> default Record nextRecord() throws IOException, MalformedRecordException {
> return nextRecord(true, false);
> } {code}
> where we are NOT dropping the unknown fields (Java doc needs some fixing here
> as it is saying the opposite)
> We get to
> {code:java}
> writer.write(firstRecord); {code}
> which gets us to
> * WriteJsonResult (L206)
> Here, we do a check
> {code:java}
> isUseSerializeForm(record, writeSchema) {code}
> which currently returns true when it should not. Because of this we write the
> serialised form which ignores the writer schema.
> In this method isUseSerializeForm(), we do check
> {code:java}
> record.getSchema().equals(writeSchema) {code}
> But at this point record.getSchema() returns the schema defined in the reader
> which is equal to the one defined in the writer - even though the record has
> additional fields compared to the defined schema.
> The suggested fix is check is to also add a check on
> {code:java}
> record.isDropUnknownFields() {code}
> If dropUnknownFields is false, then we do not use the serialised form.
> While this does solve the issue, I'm a bit conflicted on the current
> approach. Not only this could have a performance impact (we are likely going
> to not use the serialized form as often), but it also feels like the default
> should be to ignore the unknown fields when reading the record.
> If we consider the below scenario:
> * GFF Processor, generating a JSON with 3 fields: {{{}a{}}}, {{{}b{}}}, and
> {{c}}
> * ConvertRecord with JSON Reader / JSON Writer
> ** JSON reader with a schema only specifying fields {{a}} and {{b}}
> ** JSON writer with a schema specifying fields {{{}a{}}}, {{{}b{}}}, and
> {{c}} ({{{}c{}}} defaulting to {{{}null{}}})
> It feels like the expected result should be a JSON with the field {{c}} and a
> {{null}} value, because the reader would drop the field when reading the JSON
> and converting it into a record and pass it to the writer.
> If we agree on the above, then it may be easier to juste override
> {{nextRecord()}} in {{AbstractJsonRowRecordReader}} and default to
> {{{}nextRecord(true, true){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13945) Exclude FindBugs Annotations and Hibernate Annotations
[ https://issues.apache.org/jira/browse/NIFI-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13945: Status: Patch Available (was: Open) > Exclude FindBugs Annotations and Hibernate Annotations > -- > > Key: NIFI-13945 > URL: https://issues.apache.org/jira/browse/NIFI-13945 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Recent updates to several dependencies highlighted license compliance issues > with the inclusion of LGPL 2: > - FindBugs Annotations in nifi-opentelemetry-processors through > jackson-datatype-protobuf > - Hibernate Commons Annotations and Hibernate Core in nifi-registry-framework -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13945) Exclude FindBugs Annotations and Hibernate Annotations
David Handermann created NIFI-13945: --- Summary: Exclude FindBugs Annotations and Hibernate Annotations Key: NIFI-13945 URL: https://issues.apache.org/jira/browse/NIFI-13945 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: David Handermann Assignee: David Handermann Fix For: 2.0.0 Recent updates to several dependencies highlighted license compliance issues with the inclusion of LGPL 2: - FindBugs Annotations in nifi-opentelemetry-processors through jackson-datatype-protobuf - Hibernate Commons Annotations and Hibernate Core in nifi-registry-framework -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIP-1) Add VerifiableRegistryClient interface to verify Registry Client implementations
[ https://issues.apache.org/jira/browse/NIP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893678#comment-17893678 ] David Handermann commented on NIP-1: Thanks for summarizing the full scope of changes required [~mattyb149]. What aspects of the implementation would you plan to handle yourself as part of introducing this feature? > Add VerifiableRegistryClient interface to verify Registry Client > implementations > > > Key: NIP-1 > URL: https://issues.apache.org/jira/browse/NIP-1 > Project: NiFi Improvement Proposal > Issue Type: New Feature >Reporter: Matt Burgess >Priority: Normal > Labels: nifi-registry > > Motivation > To verify basic connectivity from a NiFiRegistryClient to a Registry instance > Scope > Add VerifiableRegistryClient interface to verify NiFiRegistryClient > implementations > > Compatibility > Existing implementations in Apache NiFi will need to implement the verify() > method, but a default no-op implementation will be offered so as not to > interrupt compatibility. The no-op implementation should log/report that no > actual verification was applied. > Description > Registry Clients didn't exist when the Verifiable (ControllerService, > Processor, ReportingTask, ParameterProvider, etc.) interfaces were added. > Verification > Implement the verify method for JerseyNiFiRegistryClient (contributed as a > separate Jira) and run with a live instance, validating that incorrect > configurations return error message(s) and a correct configuration returns > success. Fetching the available buckets should suffice for verification, > whether buckets exist or not. > Alternatives > None identified, current behavior does not allow verification of > NiFiRegistryClient instances -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13922) SplitExcel appears to drop the values of a column which are from a calculated formula
[
https://issues.apache.org/jira/browse/NIFI-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13922:
Fix Version/s: 2.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SplitExcel appears to drop the values of a column which are from a calculated
> formula
> --
>
> Key: NIFI-13922
> URL: https://issues.apache.org/jira/browse/NIFI-13922
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.27.0, 2.0.0-M4
>Reporter: Daniel Stieglitz
>Assignee: Daniel Stieglitz
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When reading the SalesData.xlsx referred to in NIFI-13418 with a flow which
> consists of
> GetFile -> SplitExcel -> ConvertRecord where the ConvertRecord is configured
> with an ExcelReader and a CsvrecordSetWriter it appears the values for column
> 'Sale_amt' which is a formula (column F * column G) is completely dropped as
> seen in the sample below with column headers and the first row
> {code:java}
> OrderDate,Region,Manager,SalesMan,Item,Units,Unit_price,Sale_amt
> 43106,East,Martha,Alexander,Television,95,1198,
> {code}
> Although when reading one of the tabs with a flow which consists of GetFile
> -> ConvertRecord the column is not dropped e.g. of the column names and the
> first row
> {code:java}
> OrderDate,Region,Manager,SalesMan,Item,Units,Unit_price,Sale_amt
> 151519680,East,Martha,Alexander,Television,95,1198,113810.0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-13939) Upgrade Logback to 1.5.12 and AWS SDK to 2.29.1 along with Spring Boot 3.3.5 and others
[ https://issues.apache.org/jira/browse/NIFI-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13939. - Resolution: Fixed > Upgrade Logback to 1.5.12 and AWS SDK to 2.29.1 along with Spring Boot 3.3.5 > and others > --- > > Key: NIFI-13939 > URL: https://issues.apache.org/jira/browse/NIFI-13939 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > ch.qos.logback1.5.11 1.5.12 > com.amazonaws 1.12.775 1.12.777 > com.github.luben zstd-jni 1.5.6-6 1.5.6-7 > com.github.wnameless.json 0.17.0 0.17.1 > com.ibm.icu icu4j 75.1 76.1 > com.nimbusds nimbus-jose-jwt 9.41.2 9.42 > io.github.rburgst 3.1.0 3.1.1 > org.testcontainers 1.20.2 1.20.3 > software.amazon.awssdk 2.28.25 2.29.1 > com.hubspot.jackson 0.9.15 0.9.16 > com.slack.api 1.44.0 1.44.1 > com.zendesk/mysql-binlog-.. 0.29.2 0.30.1 > net.datafaker 2.4.0 2.4.1 > org.flywaydb 10.20.0 10.20.1 > org.mariadb.jdbc 3.4.1 3.5.0 > org.springframework.boot 3.3.4 3.3.5 > org.springframework.security.kerberos 1.0.1.RELEASE vs 2.1.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13943) Upgrade Dropbox, Camel Salesforce, QuestDB, and SNMP4J
[ https://issues.apache.org/jira/browse/NIFI-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13943: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Dropbox, Camel Salesforce, QuestDB, and SNMP4J > -- > > Key: NIFI-13943 > URL: https://issues.apache.org/jira/browse/NIFI-13943 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The following libraries should be upgraded (the list is "package-name > currVersion latestVersionAvail"): > com.dropbox.core 5.4.6 7.0.0 > org.apache.camel/salesforce 3.14.10 3.22.2 > org.questdb 7.4.2 8.1.4 > org.snmp4j 2.8.18 3.8.2 > The Salesforce 3.x line is already EOL and has been for 2 years, but > upgrading to 4+ versions introduces an apparent conflict between > camel-salesforce JsonUtils and Jackson modules (older Jackson package names > versus Jackson Jakarta package names) causing classloader issues. This should > be investigated separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13941) Correct Maximum DNS Name Length for Generated Certificates
[
https://issues.apache.org/jira/browse/NIFI-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13941:
Priority: Minor (was: Major)
> Correct Maximum DNS Name Length for Generated Certificates
> --
>
> Key: NIFI-13941
> URL: https://issues.apache.org/jira/browse/NIFI-13941
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The {{nifi.web.proxy.host}} property provides optional DNS names for NiFi
> running behind a reverse proxy or gateway, and the automated certificate
> generation process considers property values when adding DNS Subject
> Alternative Names.
> Recent updates to remove the optional port element set a maximum length of 64
> characters for the entire DNS address, which is the limit for a label, but an
> not the entire address according to [RFC
> 1035|https://www.rfc-editor.org/rfc/rfc1035]. The length limitation should be
> changed to 255 as described in RFC 1035 Section 2.3.4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13941) Correct Maximum DNS Name Length for Generated Certificates
[
https://issues.apache.org/jira/browse/NIFI-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13941:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Correct Maximum DNS Name Length for Generated Certificates
> --
>
> Key: NIFI-13941
> URL: https://issues.apache.org/jira/browse/NIFI-13941
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The {{nifi.web.proxy.host}} property provides optional DNS names for NiFi
> running behind a reverse proxy or gateway, and the automated certificate
> generation process considers property values when adding DNS Subject
> Alternative Names.
> Recent updates to remove the optional port element set a maximum length of 64
> characters for the entire DNS address, which is the limit for a label, but an
> not the entire address according to [RFC
> 1035|https://www.rfc-editor.org/rfc/rfc1035]. The length limitation should be
> changed to 255 as described in RFC 1035 Section 2.3.4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13943) Upgrade Dropbox, Camel Salesforce, QuestDB, and SNMP4J
[ https://issues.apache.org/jira/browse/NIFI-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13943: Summary: Upgrade Dropbox, Camel Salesforce, QuestDB, and SNMP4J (was: Upgrade dependency versions for libraries becoming EOL) > Upgrade Dropbox, Camel Salesforce, QuestDB, and SNMP4J > -- > > Key: NIFI-13943 > URL: https://issues.apache.org/jira/browse/NIFI-13943 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The following libraries should be upgraded (the list is "package-name > currVersion latestVersionAvail"): > com.dropbox.core 5.4.6 7.0.0 > org.apache.camel/salesforce 3.14.10 3.22.2 > org.questdb 7.4.2 8.1.4 > org.snmp4j 2.8.18 3.8.2 > The Salesforce 3.x line is already EOL and has been for 2 years, but > upgrading to 4+ versions introduces an apparent conflict between > camel-salesforce JsonUtils and Jackson modules (older Jackson package names > versus Jackson Jakarta package names) causing classloader issues. This should > be investigated separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13868) Add pg-delete to toolkit CLI
[ https://issues.apache.org/jira/browse/NIFI-13868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13868. - Fix Version/s: 2.0.0 Resolution: Fixed > Add pg-delete to toolkit CLI > > > Key: NIFI-13868 > URL: https://issues.apache.org/jira/browse/NIFI-13868 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 1.27.0 >Reporter: Greg Solovyev >Assignee: Greg Solovyev >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We use Nifi Toolkit extensively to control a fleet of Nifi instances that are > deployed remotely. WE use it especially heavily for managing process groups. > One of the tasks that is impossible to perform via Nifi Toolkit currently is > deleting a process group. I added `delete-pg` command to our fork of Nifi, > but it would be nice to have it in the upstream, so that we can stay rebased > on upstream. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13868) Add pg-delete to toolkit CLI
[ https://issues.apache.org/jira/browse/NIFI-13868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13868: Fix Version/s: 1.29.0 > Add pg-delete to toolkit CLI > > > Key: NIFI-13868 > URL: https://issues.apache.org/jira/browse/NIFI-13868 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 1.27.0 >Reporter: Greg Solovyev >Assignee: Greg Solovyev >Priority: Minor > Fix For: 2.0.0, 1.29.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > We use Nifi Toolkit extensively to control a fleet of Nifi instances that are > deployed remotely. WE use it especially heavily for managing process groups. > One of the tasks that is impossible to perform via Nifi Toolkit currently is > deleting a process group. I added `delete-pg` command to our fork of Nifi, > but it would be nice to have it in the upstream, so that we can stay rebased > on upstream. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13930) PutAzureDataLakeStorage does not cause Azure to emit a FlushWithClose event on file write
[
https://issues.apache.org/jira/browse/NIFI-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13930:
Fix Version/s: 1.29.0
> PutAzureDataLakeStorage does not cause Azure to emit a FlushWithClose event
> on file write
> -
>
> Key: NIFI-13930
> URL: https://issues.apache.org/jira/browse/NIFI-13930
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.23.2
> Environment: amd64, Windows Server 2022, Java 21.0.3
>Reporter: Mark Ward
>Assignee: Peter Turcsanyi
>Priority: Minor
> Fix For: 2.0.0, 1.29.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Hi,
> Note: I am raising this issue on Databrick's behalf, they've requested Jira
> access and are currently awaiting approval.
> We use NiFi to write files to an Azure Storage account where our Databricks
> workspace can ingest the files using an Azure Queue and Databrick's File
> Notification feature, which then initiates a workflow/job.
> However, files are not being picked up in a timely manner by Databricks.
> We raised with Databricks and they've investigated, with the conclusion being
> that NiFi's behaviour when completing the file write, and then subsequent
> rename, does not emit and event type that would normally be expected.
> Please see Databricks's summary below for information:
> {quote} # Customer uses Apache Nifi 1.23.2, which performs the following
> operations via the Azure API
> ([src|https://github.com/apache/nifi/blob/rel/nifi-1.23.2/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java#L149-L166])
> ## Create a temp file using the [Path
> Create|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create?view=rest-storageservices-datalakestoragegen2-2019-12-12]
> API. In this case Azure emits a
> [BlobCreated|https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema#microsoftstorageblobcreated-event-data-lake-storage-gen2-1]
> event with the {{api}} field set to {{CreateFile}} (which is purposefully
> not processed by CSMS -
> [source|https://github.com/databricks/universe/blob/fc4a34b61abe58fbe363cc55cdb67edd480f985e/jobs-cloud-storage-meta/src/cloud/azure/resourcemanagement/AqsEventGridResourceManagementClient.scala#L366]).
> ##
> [Append|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L656]
> content to the file, there is no file event for this operation.
> ##
> [Flush|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L930]
> the appended content file content. Nifi’s implementation flushes the file
> without closing it
> ([source|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L935]),
> as a result Azure +doesn’t+ emit a {{FlushWithClose}} event (CSMS processes
> this event type).
> ##
> [Rename|https://learn.microsoft.com/en-us/dotnet/api/azure.storage.files.datalake.datalakefileclient.rename?view=azure-dotnet]
> the temp file to its final name, this generates a
> [BlobRenamed|https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema#microsoftstorageblobrenamed-event-data-lake-storage-gen2-1]
> that’s processed by CSMS (but ignored, see below).
> # The behaviour observed by the customer is explained by:
> ## When (1d) happens, CSMS gets the BlobRenamed event for a file for which
> it doesn’t know the metadata about the source file. The BlobRenamed event
> doesn’t contain all the information needed by CSMS (it misses the {{etag}}
> and the {{{}blob size{}}}) and therefore [CSMS ignores the
> event|https://github.com/databricks/universe/blob/master/jobs-cloud-storage-meta/src/storage/StorageHelper.scala#L600]
> and no object is created.
> ## When CSMS performs the daily full scan, it finds the renamed file and
> creates an object for it in its database. This causes the file arrival
> trigger to find the file and trigger a (delayed) run.{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13942) Improve failure message for uploaded NARs that fail install
[ https://issues.apache.org/jira/browse/NIFI-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13942: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Improve failure message for uploaded NARs that fail install > --- > > Key: NIFI-13942 > URL: https://issues.apache.org/jira/browse/NIFI-13942 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M4 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When an uploaded NAR fails to install, the message of the throwable is stored > as the failure message. For some throwabls, the message by itself is not > meaningful enough and would help to have to the class name of the throwable. > One example is a NoSuchMethodError where the message is just the name of the > method that is missing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13939) Upgrade Logback to 1.5.12 and AWS SDK to 2.29.1 along with Spring Boot 3.3.5 and others
[ https://issues.apache.org/jira/browse/NIFI-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13939: Summary: Upgrade Logback to 1.5.12 and AWS SDK to 2.29.1 along with Spring Boot 3.3.5 and others (was: Bump key dependencies such as logback/aws/spring/and more) > Upgrade Logback to 1.5.12 and AWS SDK to 2.29.1 along with Spring Boot 3.3.5 > and others > --- > > Key: NIFI-13939 > URL: https://issues.apache.org/jira/browse/NIFI-13939 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > ch.qos.logback1.5.11 1.5.12 > com.amazonaws 1.12.775 1.12.777 > com.github.luben zstd-jni 1.5.6-6 1.5.6-7 > com.github.wnameless.json 0.17.0 0.17.1 > com.ibm.icu icu4j 75.1 76.1 > com.nimbusds nimbus-jose-jwt 9.41.2 9.42 > io.github.rburgst 3.1.0 3.1.1 > org.testcontainers 1.20.2 1.20.3 > software.amazon.awssdk 2.28.25 2.29.1 > com.hubspot.jackson 0.9.15 0.9.16 > com.slack.api 1.44.0 1.44.1 > com.zendesk/mysql-binlog-.. 0.29.2 0.30.1 > net.datafaker 2.4.0 2.4.1 > org.flywaydb 10.20.0 10.20.1 > org.mariadb.jdbc 3.4.1 3.5.0 > org.springframework.boot 3.3.4 3.3.5 > org.springframework.security.kerberos 1.0.1.RELEASE vs 2.1.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13941) Correct Maximum DNS Name Length for Generated Certificates
[
https://issues.apache.org/jira/browse/NIFI-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13941:
Status: Patch Available (was: Open)
> Correct Maximum DNS Name Length for Generated Certificates
> --
>
> Key: NIFI-13941
> URL: https://issues.apache.org/jira/browse/NIFI-13941
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{nifi.web.proxy.host}} property provides optional DNS names for NiFi
> running behind a reverse proxy or gateway, and the automated certificate
> generation process considers property values when adding DNS Subject
> Alternative Names.
> Recent updates to remove the optional port element set a maximum length of 64
> characters for the entire DNS address, which is the limit for a label, but an
> not the entire address according to [RFC
> 1035|https://www.rfc-editor.org/rfc/rfc1035]. The length limitation should be
> changed to 255 as described in RFC 1035 Section 2.3.4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13941) Correct Maximum DNS Name Length for Generated Certificates
David Handermann created NIFI-13941:
---
Summary: Correct Maximum DNS Name Length for Generated Certificates
Key: NIFI-13941
URL: https://issues.apache.org/jira/browse/NIFI-13941
Project: Apache NiFi
Issue Type: Bug
Reporter: David Handermann
Assignee: David Handermann
Fix For: 2.0.0
The {{nifi.web.proxy.host}} property provides optional DNS names for NiFi
running behind a reverse proxy or gateway, and the automated certificate
generation process considers property values when adding DNS Subject
Alternative Names.
Recent updates to remove the optional port element set a maximum length of 64
characters for the entire DNS address, which is the limit for a label, but an
not the entire address according to [RFC
1035|https://www.rfc-editor.org/rfc/rfc1035]. The length limitation should be
changed to 255 as described in RFC 1035 Section 2.3.4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13937) NiFi CLI - Add command pg-empty-queues
[ https://issues.apache.org/jira/browse/NIFI-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13937: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > NiFi CLI - Add command pg-empty-queues > -- > > Key: NIFI-13937 > URL: https://issues.apache.org/jira/browse/NIFI-13937 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add a command to the NiFi CLI to recursively empty all queues in a process > group. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13934) Remove Hive 3 Catalog Support from Iceberg NAR
[ https://issues.apache.org/jira/browse/NIFI-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13934: Resolution: Abandoned Status: Resolved (was: Patch Available) > Remove Hive 3 Catalog Support from Iceberg NAR > -- > > Key: NIFI-13934 > URL: https://issues.apache.org/jira/browse/NIFI-13934 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Apache Iceberg supports a [Hive Metastore > Catalog|https://iceberg.apache.org/docs/nightly/hive/] for both Hive 3 and > Hive 4. The Hive Catalog is part of Hive 4 itself. > The Apache Hive project declared Hive 3 [end of > life|https://hive.apache.org/general/downloads/] on 8 October 2024, so that > version will not be receiving updates. As a result of these changes, Hive 3 > support should be removed for NiFi integrations for the Iceberg Hive Catalog. > Removing existing support for Hive 3 will provide a cleaner foundation for > potential future integration with Hive 4, avoiding the potential security > issues related to an unsupported version of Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13544) Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub
[ https://issues.apache.org/jira/browse/NIFI-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-13544. - Fix Version/s: 2.0.0 Assignee: Pierre Villard Resolution: Fixed > Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub > -- > > Key: NIFI-13544 > URL: https://issues.apache.org/jira/browse/NIFI-13544 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.27.0 >Reporter: Paul-Adrien Cordonnier >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The GCP java-pubsub library has a default behaviour of creating a threadpool > of 5 threads per processor on the machine. On a typical production server it > can be 200 threads per processor, mutliplied by the number of GCP publish > processor it can very well overwhelm a JVM. > [Link to the gcp library > snippet|https://github.com/googleapis/java-pubsub/blob/4e17c34880d0498d2bd891ea7caa8fe095054016/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L712-L716] > Similar to whats its done with > [putkudu|https://github.com/apache/nifi/blob/support/nifi-1.x/nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java#L148]. > I think we should be able to set the size of the threadpool as a property of > the processor. Default behaviour being the one in pubsub-library so that > there are no impact on performance after upgrading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13544) Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub
[ https://issues.apache.org/jira/browse/NIFI-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13544: Summary: Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub (was: Add property to set the size of the threadpool of pubsub publish) > Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub > -- > > Key: NIFI-13544 > URL: https://issues.apache.org/jira/browse/NIFI-13544 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.27.0 >Reporter: Paul-Adrien Cordonnier >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The GCP java-pubsub library has a default behaviour of creating a threadpool > of 5 threads per processor on the machine. On a typical production server it > can be 200 threads per processor, mutliplied by the number of GCP publish > processor it can very well overwhelm a JVM. > [Link to the gcp library > snippet|https://github.com/googleapis/java-pubsub/blob/4e17c34880d0498d2bd891ea7caa8fe095054016/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L712-L716] > Similar to whats its done with > [putkudu|https://github.com/apache/nifi/blob/support/nifi-1.x/nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java#L148]. > I think we should be able to set the size of the threadpool as a property of > the processor. Default behaviour being the one in pubsub-library so that > there are no impact on performance after upgrading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13934) Remove Hive 3 Catalog Support from Iceberg NAR
[ https://issues.apache.org/jira/browse/NIFI-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893480#comment-17893480 ] David Handermann commented on NIFI-13934: - Yes, NIFI-13938 removes all Iceberg components, including the Hive 3-based Catalog Service. > Remove Hive 3 Catalog Support from Iceberg NAR > -- > > Key: NIFI-13934 > URL: https://issues.apache.org/jira/browse/NIFI-13934 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Apache Iceberg supports a [Hive Metastore > Catalog|https://iceberg.apache.org/docs/nightly/hive/] for both Hive 3 and > Hive 4. The Hive Catalog is part of Hive 4 itself. > The Apache Hive project declared Hive 3 [end of > life|https://hive.apache.org/general/downloads/] on 8 October 2024, so that > version will not be receiving updates. As a result of these changes, Hive 3 > support should be removed for NiFi integrations for the Iceberg Hive Catalog. > Removing existing support for Hive 3 will provide a cleaner foundation for > potential future integration with Hive 4, avoiding the potential security > issues related to an unsupported version of Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13936) Remove GCP PubSub Lite components
[ https://issues.apache.org/jira/browse/NIFI-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13936: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove GCP PubSub Lite components > - > > Key: NIFI-13936 > URL: https://issues.apache.org/jira/browse/NIFI-13936 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Pub/Sub Lite is deprecated. Effective March 18, 2026, Pub/Sub Lite will be > turned down. > * Current customers: Pub/Sub Lite remains functional until March 18, 2026. > If you have not used Pub/Sub Lite within the 90-day period preceding > September 24, 2024 (June 26, 2024 - September 24, 2024), you won't be able to > access Pub/Sub Lite starting on September 24, 2024. > * New customers: Pub/Sub Lite is no longer available for new customers after > September 24, 2024. > See - > [https://cloud.google.com/pubsub/lite/docs/migrate-pubsub-lite-to-pubsub] > Given the upcoming 2.0 release, it's better to just remove the components. If > needed by someone, they can still download the NAR from 2.0-M4 to get these > specific processors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13938) Remove Iceberg Modules
David Handermann created NIFI-13938: --- Summary: Remove Iceberg Modules Key: NIFI-13938 URL: https://issues.apache.org/jira/browse/NIFI-13938 Project: Apache NiFi Issue Type: Improvement Reporter: David Handermann Assignee: David Handermann Fix For: 2.0.0 Based on issues surrounding the Hive 3 end-of-life dependencies described in NIFI-13934, current Iceberg components should be removed. The existing implementation includes tight coupling to Hive 3 and Hadoop dependencies along with support for Kerberos authentication. The class loading strategy motivated the current design, but as a result, requires tight coupling to existing dependencies. Rather than selective removal of the Hive 3 Catalog Service from existing bundles, the current Iceberg components should be removed. Previous released versions such as 2.0.0-M4 remain available in Maven Central. Removing the existing modules avoids coupling to Hive 3 and provides an opportunity for a different design. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13935) Remove HBase Processors and Services
[ https://issues.apache.org/jira/browse/NIFI-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13935: Status: Patch Available (was: Open) > Remove HBase Processors and Services > > > Key: NIFI-13935 > URL: https://issues.apache.org/jira/browse/NIFI-13935 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Apache HBase 2 has a direct dependency on Protobuf Java 2.5.0, which has > multiple known vulnerabilities related to potential resource exhaustion. > [HBase Protobuf > documentation|https://github.com/apache/hbase/blob/1e0884da4bdab6ba231333f55374df02200d1101/src/main/asciidoc/_chapters/protobuf.adoc] > describes changes to Protobuf dependencies for HBase 3, although HBase 3 is > still in beta at the time of this writing. Existing NiFi integration with > HBase 2 should be removed from the main branch to eliminate vulnerabilities > associated with Protobuf Java 2.5.0. Direct support for HBase could be > revisited with new versions of HBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13935) Remove HBase Processors and Services
David Handermann created NIFI-13935: --- Summary: Remove HBase Processors and Services Key: NIFI-13935 URL: https://issues.apache.org/jira/browse/NIFI-13935 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: David Handermann Assignee: David Handermann Apache HBase 2 has a direct dependency on Protobuf Java 2.5.0, which has multiple known vulnerabilities related to potential resource exhaustion. [HBase Protobuf documentation|https://github.com/apache/hbase/blob/1e0884da4bdab6ba231333f55374df02200d1101/src/main/asciidoc/_chapters/protobuf.adoc] describes changes to Protobuf dependencies for HBase 3, although HBase 3 is still in beta at the time of this writing. Existing NiFi integration with HBase 2 should be removed from the main branch to eliminate vulnerabilities associated with Protobuf Java 2.5.0. Direct support for HBase could be revisited with new versions of HBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13934) Remove Hive 3 Catalog Support from Iceberg NAR
[ https://issues.apache.org/jira/browse/NIFI-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893155#comment-17893155 ] David Handermann commented on NIFI-13934: - Thanks for noting the existing vulnerabilities [~joewitt], you are correct, libthrift 0.9.3 and protobuf-java 2.5.0 have several vulnerabilities, so removing Hive 3 removes these known vulnerabilities. > Remove Hive 3 Catalog Support from Iceberg NAR > -- > > Key: NIFI-13934 > URL: https://issues.apache.org/jira/browse/NIFI-13934 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Apache Iceberg supports a [Hive Metastore > Catalog|https://iceberg.apache.org/docs/nightly/hive/] for both Hive 3 and > Hive 4. The Hive Catalog is part of Hive 4 itself. > The Apache Hive project declared Hive 3 [end of > life|https://hive.apache.org/general/downloads/] on 8 October 2024, so that > version will not be receiving updates. As a result of these changes, Hive 3 > support should be removed for NiFi integrations for the Iceberg Hive Catalog. > Removing existing support for Hive 3 will provide a cleaner foundation for > potential future integration with Hive 4, avoiding the potential security > issues related to an unsupported version of Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13934) Remove Hive 3 Catalog Support from Iceberg NAR
[ https://issues.apache.org/jira/browse/NIFI-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13934: Status: Patch Available (was: Open) > Remove Hive 3 Catalog Support from Iceberg NAR > -- > > Key: NIFI-13934 > URL: https://issues.apache.org/jira/browse/NIFI-13934 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Apache Iceberg supports a [Hive Metastore > Catalog|https://iceberg.apache.org/docs/nightly/hive/] for both Hive 3 and > Hive 4. The Hive Catalog is part of Hive 4 itself. > The Apache Hive project declared Hive 3 [end of > life|https://hive.apache.org/general/downloads/] on 8 October 2024, so that > version will not be receiving updates. As a result of these changes, Hive 3 > support should be removed for NiFi integrations for the Iceberg Hive Catalog. > Removing existing support for Hive 3 will provide a cleaner foundation for > potential future integration with Hive 4, avoiding the potential security > issues related to an unsupported version of Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13934) Remove Hive 3 Catalog Support from Iceberg NAR
David Handermann created NIFI-13934: --- Summary: Remove Hive 3 Catalog Support from Iceberg NAR Key: NIFI-13934 URL: https://issues.apache.org/jira/browse/NIFI-13934 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: David Handermann Assignee: David Handermann Apache Iceberg supports a [Hive Metastore Catalog|https://iceberg.apache.org/docs/nightly/hive/] for both Hive 3 and Hive 4. The Hive Catalog is part of Hive 4 itself. The Apache Hive project declared Hive 3 [end of life|https://hive.apache.org/general/downloads/] on 8 October 2024, so that version will not be receiving updates. As a result of these changes, Hive 3 support should be removed for NiFi integrations for the Iceberg Hive Catalog. Removing existing support for Hive 3 will provide a cleaner foundation for potential future integration with Hive 4, avoiding the potential security issues related to an unsupported version of Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13933) Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings
David Handermann created NIFI-13933: --- Summary: Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings Key: NIFI-13933 URL: https://issues.apache.org/jira/browse/NIFI-13933 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: David Handermann Assignee: David Handermann Spring Security dependencies should be upgraded to 6.3.4 to address findings for CVE-2024-38821. This vulnerability applies to WebFlux libraries that NiFi does not use. In addition, recent Dependency Check Plugin reports include a number of false positives related to Azure Identity libraries. False positives should be suppressed and other impacted dependencies should be upgraded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13724) NiFi Registry Events Include Additional Metadata
[ https://issues.apache.org/jira/browse/NIFI-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13724: Summary: NiFi Registry Events Include Additional Metadata (was: Nifi Registry Events Include Additional Metadata) > NiFi Registry Events Include Additional Metadata > > > Key: NIFI-13724 > URL: https://issues.apache.org/jira/browse/NIFI-13724 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: Dye357 >Assignee: Dye357 >Priority: Minor > Fix For: 2.0.0, 1.29.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently the Nifi Registry EventFactory adds a few pieces of metadata to the > EventType - when writing a custom extension for Nifi Registry a developer is > limited in the amount of information they have about the event – this ticket > would expand the scope of metadata included in each event to include the > Name, Description and Timestamps. > As a work-around we were using the nifi-registry-client to use the provided > IDs to enrich our dataset however we've run into problems: > 1. The nifi registry client is broken when running inside an Nifi Registry > extension due to JavaX/Jakarta classpath issues. (Separate ticket) > 2. This the most important of the two points – if we have a Bucket Deleted or > Flow Deleted event only having the ID is useless because we can no longer > query the API to get the name and description for the Entity in question > which has led to some auditing/accountability concerns. > I'm putting together a PR to include the additional specified metadata in the > EventFactory. The PR should be merge-able on 1.x and 2.x -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13929) Many duplicates for Provenance Events when running Stateless flow with multiple threads
[ https://issues.apache.org/jira/browse/NIFI-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13929: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Many duplicates for Provenance Events when running Stateless flow with > multiple threads > --- > > Key: NIFI-13929 > URL: https://issues.apache.org/jira/browse/NIFI-13929 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 2.0.0 > > Attachments: events.png, flow.png > > Time Spent: 40m > Remaining Estimate: 0h > > When running a flow using the Stateless Execution Engine, we see it making a > huge number of duplicates in the provenance repository. This results in poor > performance, as well as very incorrect Provenance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13927) PublishGCPPubSub processor stop working and is stucked when using Record Oriented mode
[ https://issues.apache.org/jira/browse/NIFI-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13927: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > PublishGCPPubSub processor stop working and is stucked when using Record > Oriented mode > -- > > Key: NIFI-13927 > URL: https://issues.apache.org/jira/browse/NIFI-13927 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.26.0, 2.0.0-M4 >Reporter: Julien G. >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Attachments: EXAMPLE.json, nifi.tdump > > Time Spent: 1.5h > Remaining Estimate: 0h > > When using the PublishGCPPubSub processor with the Record Oriented mode the > processor will stay stucked. It won't processor any FlowFile. When the > processor is terminated the thread isn't released by the processor. > It FlowFile Oriented mode, it works fine. > It seems to be linked to how many records are in the FlowFile, becase when > having a single record in the FlowFile, it seems to have no issue (may be it > take more time to appear). > But if you start to have more records in the FlowFile, it won't work. To make > the processor works again you either need to restart the node or removing and > recreating the processor. But it's temporary, because it some point it will > be stucked again. When stucked the processor won't be unstucked even after > days (we had a processor stucked for 3 days straight). > You can find attached a thread dump when the processor was running and > stucked, and with ~10 threads not released . And you can also find an example > that will show the issue with a FlowFile of 500 records that instantly > stucked in the publish. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13724) Nifi Registry Events Include Additional Metadata
[ https://issues.apache.org/jira/browse/NIFI-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13724: Fix Version/s: 1.29.0 > Nifi Registry Events Include Additional Metadata > > > Key: NIFI-13724 > URL: https://issues.apache.org/jira/browse/NIFI-13724 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: Dye357 >Assignee: Dye357 >Priority: Minor > Fix For: 2.0.0, 1.29.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently the Nifi Registry EventFactory adds a few pieces of metadata to the > EventType - when writing a custom extension for Nifi Registry a developer is > limited in the amount of information they have about the event – this ticket > would expand the scope of metadata included in each event to include the > Name, Description and Timestamps. > As a work-around we were using the nifi-registry-client to use the provided > IDs to enrich our dataset however we've run into problems: > 1. The nifi registry client is broken when running inside an Nifi Registry > extension due to JavaX/Jakarta classpath issues. (Separate ticket) > 2. This the most important of the two points – if we have a Bucket Deleted or > Flow Deleted event only having the ID is useless because we can no longer > query the API to get the name and description for the Entity in question > which has led to some auditing/accountability concerns. > I'm putting together a PR to include the additional specified metadata in the > EventFactory. The PR should be merge-able on 1.x and 2.x -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13896) Improving TailFile performance
[
https://issues.apache.org/jira/browse/NIFI-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-13896:
Fix Version/s: 1.29.0
> Improving TailFile performance
> --
>
> Key: NIFI-13896
> URL: https://issues.apache.org/jira/browse/NIFI-13896
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
> Fix For: 2.0.0, 1.29.0
>
> Attachments: tailfile_test.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In case of tailing numerous files, the processor is slow because it
> repeatedly loops over a large number of tailed files and performs several
> expensive operations.
> * In the {{OnTrigger}} method, a loop (loop 1) iterates over all tailed
> files in the state object.
> * Inside this loop, for each tailed file, the {{recoverRolledFiles}} method
> is called (loop 2), which then leads to {{consumeFilesFully}} and finally
> triggers {{{}cleanup{}}}.
> * In the {{cleanup}} method, another loop (loop 3) iterates over all tailed
> files in the state again.
> * During the {{{}cleanup{}}}, {{persistState}} is invoked, which removes any
> legacy state variables from the NiFi state. These legacy state variables
> originate from NiFi 1.0, when support for "Multiple Tailed Files" was not
> available, so state keys didn’t have the "file.x." prefix. As the {{cleanup}}
> iterates over and persists each tailed file's state, the overall state size
> grows (adding six entries per tailed file). This causes the legacy cleanup
> loop to become progressively slower with each iteration as the number of
> state entries grows.
> This can lead to hours of execution time.
>
> Suggestion for improvement:
>
> * Moving out the loop that removes old state entries from cleanup. The
> cleanup of old entries should be run on the startup instead.
> {code:java}
> for(String key : oldState.toMap().keySet()) {
> // These states are stored by older version of NiFi, and won't be used
> anymore.
> // New states have 'file..' prefix.
> if (TailFileState.StateKeys.CHECKSUM.equals(key)
> || TailFileState.StateKeys.FILENAME.equals(key)
> || TailFileState.StateKeys.POSITION.equals(key)
> || TailFileState.StateKeys.TIMESTAMP.equals(key)) {
> getLogger().info("Removed state {}={} stored by older version of
> NiFi.", new Object[]{key, oldState.get(key)});
> continue;
> }
> updatedState.put(key, oldState.get(key));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13933) Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings
[ https://issues.apache.org/jira/browse/NIFI-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13933: Fix Version/s: 2.0.0 > Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings > -- > > Key: NIFI-13933 > URL: https://issues.apache.org/jira/browse/NIFI-13933 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Spring Security dependencies should be upgraded to 6.3.4 to address findings > for CVE-2024-38821. This vulnerability applies to WebFlux libraries that NiFi > does not use. > In addition, recent Dependency Check Plugin reports include a number of false > positives related to Azure Identity libraries. False positives should be > suppressed and other impacted dependencies should be upgraded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13933) Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings
[ https://issues.apache.org/jira/browse/NIFI-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-13933: Status: Patch Available (was: Open) > Upgrade Spring Security to 6.3.4 and Address Dependency Check Findings > -- > > Key: NIFI-13933 > URL: https://issues.apache.org/jira/browse/NIFI-13933 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Spring Security dependencies should be upgraded to 6.3.4 to address findings > for CVE-2024-38821. This vulnerability applies to WebFlux libraries that NiFi > does not use. > In addition, recent Dependency Check Plugin reports include a number of false > positives related to Azure Identity libraries. False positives should be > suppressed and other impacted dependencies should be upgraded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
