[jira] [Updated] (NIFI-12741) Parameters does not work with "Access Restricted Components" - "Requiring 'access keytab'"
[
https://issues.apache.org/jira/browse/NIFI-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12741:
---
Fix Version/s: 1.28.0
2.0.0-M5
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Parameters does not work with "Access Restricted Components" - "Requiring
> 'access keytab'"
> --
>
> Key: NIFI-12741
> URL: https://issues.apache.org/jira/browse/NIFI-12741
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Matthew Clarke
>Assignee: David Szabo
>Priority: Major
> Fix For: 1.28.0, 2.0.0-M5
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Parameters does not work with "Access Restricted Components" - "Requiring
> 'access keytab'".
> Reproduction steps:
> * User A has full permissions to child PG “test”
> * User A creates a parameter context that is mapped to this child PG
> * User A adds ConsumeKafka_2_6 processor
> * Admin user creates a keytab credentials service “kerb-test” within PG “test”
> * User A configures ConsumeKafKa_2_6 processor, selects “kerb-test”, and
> clicks apply. (all works as expected)
> * User A clicks on option to convert to parameter on Kerberos Credentials
> Service property in ConsumeKafla_2_6 processor and sets name to “kerb-test”.
> Property Value now reflects “#{kerb-test}. Click APPLY and encounter
> exception: “Unable to modify Components requiring additional permission:
> access keytab. Contact the system administrator. Contact the system
> administrator.”
> Verified parameter “kerb-test” was successfully added to parameter context on
> child PG “test”
> User should be able to use parameter contexts to reference keytab credentials
> service created on an authorized process PG. Policy should only block user
> from being able to create a new keytab credentials service or modify an
> existing keytab credentials service. Ability to select an already created
> keytab credentials service shoudl be controlled by authorized via "view the
> component" policy on the controller service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13605) Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE public
[ https://issues.apache.org/jira/browse/NIFI-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-13605: --- Summary: Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE public (was: Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE publicű) > Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE public > - > > Key: NIFI-13605 > URL: https://issues.apache.org/jira/browse/NIFI-13605 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Currently KERBEROS_USER_SERVICE is package private which is unnecessarily > restrictive and makes it problematic to access via inheritance or utils or > otherwise. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13605) Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE publicű
[ https://issues.apache.org/jira/browse/NIFI-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-13605: --- Summary: Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE publicű (was: Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE private) > Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE publicű > -- > > Key: NIFI-13605 > URL: https://issues.apache.org/jira/browse/NIFI-13605 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Currently KERBEROS_USER_SERVICE is package private which is unnecessarily > restrictive and makes it problematic to access via inheritance or utils or > otherwise. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-13605) Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE private
[ https://issues.apache.org/jira/browse/NIFI-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-13605: -- Assignee: Tamas Palfy > Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE private > -- > > Key: NIFI-13605 > URL: https://issues.apache.org/jira/browse/NIFI-13605 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Currently KERBEROS_USER_SERVICE is package private which is unnecessarily > restrictive and makes it problematic to access via inheritance or utils or > otherwise. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13605) Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE private
Tamas Palfy created NIFI-13605: -- Summary: Make AbstractHadoopProcessor.KERBEROS_USER_SERVICE private Key: NIFI-13605 URL: https://issues.apache.org/jira/browse/NIFI-13605 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Currently KERBEROS_USER_SERVICE is package private which is unnecessarily restrictive and makes it problematic to access via inheritance or utils or otherwise. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-13583) Drive processor Credentials issue
[
https://issues.apache.org/jira/browse/NIFI-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17868728#comment-17868728
]
Tamas Palfy edited comment on NIFI-13583 at 7/25/24 4:11 PM:
-
[~juldrixx] I tried a ListGoogleDrive and a ConsumeGCPubSub and they both work
at the same time. I did it on a support/nifif-1.x. build.
Can you please confirm this issue appears in a vanilla fresh NiFi build? If
that's the case could you please upload to this Jira a flow definition (without
your personal Google settings of course)?
was (Author: tpalfy):
[~juldrixx] I tried a ListGoogleDrive and a ConsumeGCPubSub and they both work
at the same time. I did it on a support/nifif-1.x. build.
Can you please confirm this issue appears in a vanilla fresh NiFi build? If
that's the case could you please add a flow definition (without your personal
Google settings of course)?
> Drive processor Credentials issue
> -
>
> Key: NIFI-13583
> URL: https://issues.apache.org/jira/browse/NIFI-13583
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julien G.
>Priority: Major
>
> When using a Drive processor with Default credentials, the processor fine at
> the beginning. But if we start a processor as ConsumePubSub, it won't work
> again with the following error. The only way to make it works again is by
> restarting NiFi. But if a GCP processor like ConsumePubSub is started it
> won't work again.
> {code:java}
> GET
> https://www.googleapis.com/drive/v3/files?includeItemsFromAllDrives=true='1yUM0-7Onq2U-3BcWsTAES-2WFzjehg3d'%20in%20parents%20and%20mimeType%20%3D%20'application/vnd.google-apps.folder'=true
> {
> "code": 403,
> "details": [
> {
> "@type": "type.googleapis.com/google.rpc.ErrorInfo",
> "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
> "domain": "googleapis.com",
> "metadata": {
> "method": "google.apps.drive.v3.DriveFiles.List",
> "service": "drive.googleapis.com"
> }
> }
> ],
> "errors": [
> {
> "domain": "global",
> "message": "Insufficient Permission",
> "reason": "insufficientPermissions"
> }
> ],
> "message": "Request had insufficient authentication scopes.",
> "status": "PERMISSION_DENIED"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-13583) Drive processor Credentials issue
[
https://issues.apache.org/jira/browse/NIFI-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17868728#comment-17868728
]
Tamas Palfy commented on NIFI-13583:
[~juldrixx] I tried a ListGoogleDrive and a ConsumeGCPubSub and they both work
at the same time. I did it on a support/nifif-1.x. build.
Can you please confirm this issue appears in a vanilla fresh NiFi build? If
that's the case could you please add a flow definition (without your personal
Google settings of course)?
> Drive processor Credentials issue
> -
>
> Key: NIFI-13583
> URL: https://issues.apache.org/jira/browse/NIFI-13583
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julien G.
>Priority: Major
>
> When using a Drive processor with Default credentials, the processor fine at
> the beginning. But if we start a processor as ConsumePubSub, it won't work
> again with the following error. The only way to make it works again is by
> restarting NiFi. But if a GCP processor like ConsumePubSub is started it
> won't work again.
> {code:java}
> GET
> https://www.googleapis.com/drive/v3/files?includeItemsFromAllDrives=true='1yUM0-7Onq2U-3BcWsTAES-2WFzjehg3d'%20in%20parents%20and%20mimeType%20%3D%20'application/vnd.google-apps.folder'=true
> {
> "code": 403,
> "details": [
> {
> "@type": "type.googleapis.com/google.rpc.ErrorInfo",
> "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
> "domain": "googleapis.com",
> "metadata": {
> "method": "google.apps.drive.v3.DriveFiles.List",
> "service": "drive.googleapis.com"
> }
> }
> ],
> "errors": [
> {
> "domain": "global",
> "message": "Insufficient Permission",
> "reason": "insufficientPermissions"
> }
> ],
> "message": "Request had insufficient authentication scopes.",
> "status": "PERMISSION_DENIED"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13484) HBase_2_ClientService.getResults doesn't return single row
[
https://issues.apache.org/jira/browse/NIFI-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-13484:
---
Description:
https://issues.apache.org/jira/browse/NIFI-12142 changed the deprecated
{code:java}
final Scan scan = new Scan();
scan.setStartRow(startRow);
scan.setStopRow(endRow);
{code}
to
{code:java}
Scan scan = new Scan();
scan = scan.withStartRow(startRow);
scan = scan.withStopRow(endRow);
{code}
While the two are almost identical logically, they behave differently when
_startRow_ and _endRow_ are the same. While both _setStopRow_ and _withStopRow_
are _exclusive_ by default, when _startRow_ and _endRow_ are the same, this
exclusivity doesn't prevent _startRow_ from appearing in the result for the
deprecated method variants while it does for the newer ones.
was:
https://issues.apache.org/jira/browse/NIFI-12142 changed the deprecated
{code:java}
final Scan scan = new Scan();
scan.setStartRow(startRow);
scan.setStopRow(endRow);
{code}
to
{code:java}
Scan scan = new Scan();
scan = scan.withStartRow(startRow);
scan = scan.withStopRow(endRow);
{code}
While the two are almost identical logically, they behave differently when
{{startRow}} and {{endRow}} are the same. While both {{setStopRow}} and
{{withStopRow}} are _exclusive_ by default, when {{startRow}} and {{endRow}}
are the same, this exclusivity doesn't prevent {{startRow}} from appearing in
the result for the deprecated method variants while it does for the newer ones.
> HBase_2_ClientService.getResults doesn't return single row
> --
>
> Key: NIFI-13484
> URL: https://issues.apache.org/jira/browse/NIFI-13484
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
>
> https://issues.apache.org/jira/browse/NIFI-12142 changed the deprecated
> {code:java}
> final Scan scan = new Scan();
> scan.setStartRow(startRow);
> scan.setStopRow(endRow);
> {code}
> to
> {code:java}
> Scan scan = new Scan();
> scan = scan.withStartRow(startRow);
> scan = scan.withStopRow(endRow);
> {code}
> While the two are almost identical logically, they behave differently when
> _startRow_ and _endRow_ are the same. While both _setStopRow_ and
> _withStopRow_ are _exclusive_ by default, when _startRow_ and _endRow_ are
> the same, this exclusivity doesn't prevent _startRow_ from appearing in the
> result for the deprecated method variants while it does for the newer ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13484) HBase_2_ClientService.getResults doesn't return single row
[
https://issues.apache.org/jira/browse/NIFI-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-13484:
---
Summary: HBase_2_ClientService.getResults doesn't return single row (was:
HBase_2_ClientService.scan doesn't return single row)
> HBase_2_ClientService.getResults doesn't return single row
> --
>
> Key: NIFI-13484
> URL: https://issues.apache.org/jira/browse/NIFI-13484
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
>
> https://issues.apache.org/jira/browse/NIFI-12142 changed the deprecated
> {code:java}
> final Scan scan = new Scan();
> scan.setStartRow(startRow);
> scan.setStopRow(endRow);
> {code}
> to
> {code:java}
> Scan scan = new Scan();
> scan = scan.withStartRow(startRow);
> scan = scan.withStopRow(endRow);
> {code}
> While the two are almost identical logically, they behave differently when
> {{startRow}} and {{endRow}} are the same. While both {{setStopRow}} and
> {{withStopRow}} are _exclusive_ by default, when {{startRow}} and {{endRow}}
> are the same, this exclusivity doesn't prevent {{startRow}} from appearing in
> the result for the deprecated method variants while it does for the newer
> ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13484) HBase_2_ClientService.scan doesn't return single row
Tamas Palfy created NIFI-13484:
--
Summary: HBase_2_ClientService.scan doesn't return single row
Key: NIFI-13484
URL: https://issues.apache.org/jira/browse/NIFI-13484
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Tamas Palfy
Assignee: Tamas Palfy
https://issues.apache.org/jira/browse/NIFI-12142 changed the deprecated
{code:java}
final Scan scan = new Scan();
scan.setStartRow(startRow);
scan.setStopRow(endRow);
{code}
to
{code:java}
Scan scan = new Scan();
scan = scan.withStartRow(startRow);
scan = scan.withStopRow(endRow);
{code}
While the two are almost identical logically, they behave differently when
{{startRow}} and {{endRow}} are the same. While both {{setStopRow}} and
{{withStopRow}} are _exclusive_ by default, when {{startRow}} and {{endRow}}
are the same, this exclusivity doesn't prevent {{startRow}} from appearing in
the result for the deprecated method variants while it does for the newer ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13388) Add NiFi Cli support for Flow Analysis Rules
Tamas Palfy created NIFI-13388: -- Summary: Add NiFi Cli support for Flow Analysis Rules Key: NIFI-13388 URL: https://issues.apache.org/jira/browse/NIFI-13388 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Assignee: Tamas Palfy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13139) PythonNarIT tests failing, missing StandardPythonBridge
Tamas Palfy created NIFI-13139: -- Summary: PythonNarIT tests failing, missing StandardPythonBridge Key: NIFI-13139 URL: https://issues.apache.org/jira/browse/NIFI-13139 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy The issue seems to be due to nifi-system-tests/nifi-system-test-suite/target/pythonic-instance/lib not prepared to have the nifi-py4j-nar-2.0.0-SNAPSHOT.nar. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-13084) Backport Allow Disabling Scientific Notation in JSON Writer
[ https://issues.apache.org/jira/browse/NIFI-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-13084. Fix Version/s: 1.26.0 Resolution: Fixed > Backport Allow Disabling Scientific Notation in JSON Writer > > > Key: NIFI-13084 > URL: https://issues.apache.org/jira/browse/NIFI-13084 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Zsihovszki Krisztina >Assignee: Zsihovszki Krisztina >Priority: Major > Fix For: 1.26.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The changes made for NIFI-12697 were only committed on the 2.x branch and not > backported to the support/nifi-1.x branch. The purpose of this ticket is to > backport the code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13090) Backport Improve handling of embedded JSON records
[ https://issues.apache.org/jira/browse/NIFI-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-13090: --- Fix Version/s: 1.26.0 > Backport Improve handling of embedded JSON records > -- > > Key: NIFI-13090 > URL: https://issues.apache.org/jira/browse/NIFI-13090 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Zsihovszki Krisztina >Assignee: Zsihovszki Krisztina >Priority: Major > Fix For: 1.26.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The changes made for NIFI-12480 > were only committed on the 2.x branch and not backported to the > support/nifi-1.x branch. The purpose of this ticket is to backport the code. > Another backport ticket, https://issues.apache.org/jira/browse/NIFI-13084 is > based on this change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12973) Add Process Group scope to Flow Analysis rules
[ https://issues.apache.org/jira/browse/NIFI-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12973: -- Assignee: Tamas Palfy > Add Process Group scope to Flow Analysis rules > -- > > Key: NIFI-12973 > URL: https://issues.apache.org/jira/browse/NIFI-12973 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Tamas Palfy >Priority: Major > > I think it'd be useful to have an optional property in all Flow Analysis > rules to scope a rule execution to a specific process group (and its embedded > process groups). > Most NiFi users are using NiFi in a multi tenant way and are dedicating a > process group to a given team. Different rules may apply across different > teams hence the advantage of scoping a rule to a process group (when it makes > sense for the rule). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12614) Create record reader service for Protobuf messages
[ https://issues.apache.org/jira/browse/NIFI-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-12614. Fix Version/s: 2.0.0-M3 1.26.0 Resolution: Fixed > Create record reader service for Protobuf messages > --- > > Key: NIFI-12614 > URL: https://issues.apache.org/jira/browse/NIFI-12614 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Fix For: 2.0.0-M3, 1.26.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Create a record reader implementation for Protobuf which is capable to read > encoded data and create NiFi records from it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12735) Execute flow analysis before committing flow into Registry
[
https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12735:
---
Fix Version/s: 2.0.0-M3
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Execute flow analysis before committing flow into Registry
> --
>
> Key: NIFI-12735
> URL: https://issues.apache.org/jira/browse/NIFI-12735
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 2.0.0-M3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp
> When committing a flow into a Registry, it would be beneficial to have the
> opportunity to execute the existing flow analysis rules and potentially
> interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry
> handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch:
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more
> frequent and I consider "lingering" flow definitions without snapshot
> misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version control
> AND `GenerateFlowFile` processor is added (not committed to the registry)
> WHEN the user tries to add new version of G1 to registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules
[jira] [Assigned] (NIFI-12994) Fix NPE when requesting Flow Analysis Results for a Process Group
[ https://issues.apache.org/jira/browse/NIFI-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12994: -- Assignee: Tamas Palfy > Fix NPE when requesting Flow Analysis Results for a Process Group > - > > Key: NIFI-12994 > URL: https://issues.apache.org/jira/browse/NIFI-12994 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When requesting Flow Analysis Results for a Process Group the underlying Rule > Violations are checked if registered for the same group id. That group id is > the group id of the violating component. However there can be components that > don't belong to a Process Group, in which case this group id is going to be > null. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12994) Fix NPE when requesting Flow Analysis Results for a Process Group
Tamas Palfy created NIFI-12994: -- Summary: Fix NPE when requesting Flow Analysis Results for a Process Group Key: NIFI-12994 URL: https://issues.apache.org/jira/browse/NIFI-12994 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy When requesting Flow Analysis Results for a Process Group the underlying Rule Violations are checked if registered for the same group id. That group id is the group id of the violating component. However there can be components that don't belong to a Process Group, in which case this group id is going to be null. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12924) FlowAnalysis streamline
[ https://issues.apache.org/jira/browse/NIFI-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12924: -- Assignee: Tamas Palfy > FlowAnalysis streamline > --- > > Key: NIFI-12924 > URL: https://issues.apache.org/jira/browse/NIFI-12924 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Have flow analysis occur as needed instead of as demanded or regularly. When > the flow is modified occurs or there are changes among the rules a background > thread will run a flow analysis within 5 seconds. Otherwise no analysis > occurs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12924) FlowAnalysis streamline
Tamas Palfy created NIFI-12924: -- Summary: FlowAnalysis streamline Key: NIFI-12924 URL: https://issues.apache.org/jira/browse/NIFI-12924 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Have flow analysis occur as needed instead of as demanded or regularly. When the flow is modified occurs or there are changes among the rules a background thread will run a flow analysis within 5 seconds. Otherwise no analysis occurs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12888) OAuth2 token refresh doesn't work in Email processors
Tamas Palfy created NIFI-12888: -- Summary: OAuth2 token refresh doesn't work in Email processors Key: NIFI-12888 URL: https://issues.apache.org/jira/browse/NIFI-12888 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy ConsumeIMAP and ConsumePOP3 both inherit the AbstractEmailProcessor in which the OAuth2 token refresh handling is not handled properly. The object that is receiving the emails is created and finalized only once. When the token expires, nothing makes to refresh logic to run. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12888) OAuth2 token refresh doesn't work in Email processors
[ https://issues.apache.org/jira/browse/NIFI-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12888: -- Assignee: Tamas Palfy > OAuth2 token refresh doesn't work in Email processors > - > > Key: NIFI-12888 > URL: https://issues.apache.org/jira/browse/NIFI-12888 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > ConsumeIMAP and ConsumePOP3 both inherit the AbstractEmailProcessor in which > the OAuth2 token refresh handling is not handled properly. > The object that is receiving the emails is created and finalized only once. > When the token expires, nothing makes to refresh logic to run. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12887) Additional ways for handling Strings as binary in PutDatabaseRecord
[ https://issues.apache.org/jira/browse/NIFI-12887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12887: -- Assignee: Tamas Palfy > Additional ways for handling Strings as binary in PutDatabaseRecord > --- > > Key: NIFI-12887 > URL: https://issues.apache.org/jira/browse/NIFI-12887 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > PutDatabaseRecord expects byte arrays when inserting into a binary type > column (blob, binary etc.) > Or if String is encountered it uses a UTF8 String -> byte conversion. > Using hex-string or base64-encoding to represent binary data as string is > fairly common so this Jira is about adding these capabilities to the > processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12887) Additional ways for handling Strings as binary in PutDatabaseRecord
Tamas Palfy created NIFI-12887: -- Summary: Additional ways for handling Strings as binary in PutDatabaseRecord Key: NIFI-12887 URL: https://issues.apache.org/jira/browse/NIFI-12887 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy PutDatabaseRecord expects byte arrays when inserting into a binary type column (blob, binary etc.) Or if String is encountered it uses a UTF8 String -> byte conversion. Using hex-string or base64-encoding to represent binary data as string is fairly common so this Jira is about adding these capabilities to the processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12862) FlowAnalysisResults should not leak anauthorized component details
[ https://issues.apache.org/jira/browse/NIFI-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-12862: --- Summary: FlowAnalysisResults should not leak anauthorized component details (was: FlowAnalysisResults should leak anauthorized component details) > FlowAnalysisResults should not leak anauthorized component details > -- > > Key: NIFI-12862 > URL: https://issues.apache.org/jira/browse/NIFI-12862 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > The FlowAnalysisResultEntity hold FlowAnalysisRuleViolationDTO that contain > the name of a violating component as a message describing the violation. This > usually contains details about the violating component. > A user can see these even if they don't have read permission for that > particular component. > In clustered environment the request merger filters out such violations but > in a non-clustered environment there is no such filtering phase. > The FlowAnalysisRuleViolationDTO itself should be built accordingly and leave > certain details blank when the user lacks read permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12862) FlowAnalysisResults should leak anauthorized component details
[ https://issues.apache.org/jira/browse/NIFI-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12862: -- Assignee: Tamas Palfy > FlowAnalysisResults should leak anauthorized component details > -- > > Key: NIFI-12862 > URL: https://issues.apache.org/jira/browse/NIFI-12862 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > The FlowAnalysisResultEntity hold FlowAnalysisRuleViolationDTO that contain > the name of a violating component as a message describing the violation. This > usually contains details about the violating component. > A user can see these even if they don't have read permission for that > particular component. > In clustered environment the request merger filters out such violations but > in a non-clustered environment there is no such filtering phase. > The FlowAnalysisRuleViolationDTO itself should be built accordingly and leave > certain details blank when the user lacks read permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12862) FlowAnalysisResults should leak anauthorized component details
Tamas Palfy created NIFI-12862: -- Summary: FlowAnalysisResults should leak anauthorized component details Key: NIFI-12862 URL: https://issues.apache.org/jira/browse/NIFI-12862 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy The FlowAnalysisResultEntity hold FlowAnalysisRuleViolationDTO that contain the name of a violating component as a message describing the violation. This usually contains details about the violating component. A user can see these even if they don't have read permission for that particular component. In clustered environment the request merger filters out such violations but in a non-clustered environment there is no such filtering phase. The FlowAnalysisRuleViolationDTO itself should be built accordingly and leave certain details blank when the user lacks read permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12843) If record count is set, ParquetRecordReader does not read the whole file
[ https://issues.apache.org/jira/browse/NIFI-12843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-12843. Fix Version/s: 2.0.0 1.26.0 Resolution: Fixed > If record count is set, ParquetRecordReader does not read the whole file > > > Key: NIFI-12843 > URL: https://issues.apache.org/jira/browse/NIFI-12843 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.25.0, 2.0.0-M2 >Reporter: Rajmund Takacs >Assignee: Rajmund Takacs >Priority: Major > Fix For: 2.0.0, 1.26.0 > > Attachments: parquet_reader_usecases.json > > Time Spent: 50m > Remaining Estimate: 0h > > Earlier ParquetRecordReader ignored the record.count attribue of the incoming > FlowFile. With NIFI-12241 this had been changed, and now the reader reads > only the specified number of rows from the record set. But if the Parquet > file is not produced by a record writer, then this attribute is not set > normally, and in this case the record reader reads the whole file. However, > processors producing parquet file by processing record sets, might have this > attribute set, referring to the record set the parquet file is taken from, > and not the actual content. This leads to an incorrect behavior. > For example: ConsumeKafka produces a single record FlowFile, that is a > parquet file with 1000 rows, then record.count would be set to 1, instead of > 1000, because it refers to the Kafka record set. So ParquetRecordReader now > reads only the first record of the Parquet file. > The sole reason of changing the reader to take record.count into account is > that, CalculateParquetOffsets processors generate flow files with same > content, but different offset and count attributes, representing a slice of > the original, big input. And then the parquet reader acts as if the big flow > file was only a small one, containing that slice, which makes processing more > efficient. There is no need to support files having no offset, but having a > limit (count), so changing the reader to only take record.count into account, > if offset is present too, could to be a reasonable fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12745) AvroReader silently drops record if it's malformed
[
https://issues.apache.org/jira/browse/NIFI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12745:
---
Fix Version/s: 2.0.0
1.26.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> AvroReader silently drops record if it's malformed
> --
>
> Key: NIFI-12745
> URL: https://issues.apache.org/jira/browse/NIFI-12745
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 2.0.0-M1, 1.18.0, 1.19.0, 1.20.0, 1.19.1, 1.21.0,
> 1.22.0, 1.23.0, 1.24.0, 1.23.1, 1.23.2, 1.25.0, 2.0.0-M2
>Reporter: Rajmund Takacs
>Assignee: Rajmund Takacs
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
> Attachments: ValidateRecord.json
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> See the attached example flow. It reproduces the issue very reliably.
> {{GenerateFlowFile}} is set to generate the following Json:
> {code:json}
> [{
> "field_1" : 123456789,
> "field_2" : "44",
> "field_3" : 5
> }]
> {code}
> This input is converted to Avro format, using the {{ConvertRecord}}
> processor. The 'Schema Write Strategy' of {{AvroRecordSetWriter}} is set to
> anything different than 'Embed Avro Schema'.
> Then, the resulting FF is routed to a processor that uses an {{AvroReader}}
> to work on the records. The reader is set to use a predefined, fixed schema,
> which does not match with the input avro file, contains at least an extra
> field. It does not matter if that field has a default value or not.
> {code:json}
> {
> "type":"record",
> "name":"message_name",
> "namespace":"message_namespace",
> "fields":[
> {
> "name":"field_1",
> "type":["long"]
> },
> {
> "name":"field_2",
> "type":["string"]
> },
> {
> "name":"field_3",
> "type":["int"]
> },
> {
> "name":"extra_field",
> "type":["string"],
> "default":"empty"
> }
> ]
> }
> {code}
> When this processor consumes the input, the reader silently drops the record,
> without even making an error log message. At the processor level, this is
> equivalent to having no records to process, so nothing happens. The user
> won't notice that there is a misconfiguration somewhere until they start
> noticing the missing the flow files.
> The expected behavior from the processors would be to route the malformed
> input FF to their failure relationship, and shout an error on its bulletin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12732) ListS3 should reset its tracking state after configuration change
[
https://issues.apache.org/jira/browse/NIFI-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12732:
---
Fix Version/s: 2.0.0
1.26.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ListS3 should reset its tracking state after configuration change
> -
>
> Key: NIFI-12732
> URL: https://issues.apache.org/jira/browse/NIFI-12732
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {{List}} processors can reset their listing tracking state (stored in the
> component state or in a distributed cache depending on the {{{}Listing
> Strategy{}}}) when a relevant property has been modified (e.g. the directory
> to list). The functionality was implemented in {{{}AbstractListProcessor{}}},
> the child processors only need to provide the property list when the reset
> should occur.
> {{ListS3}} is not a child of {{AbstractListProcessor}} and the listing logic
> was implemented separately. The reset function is missing in the processor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12696) Fix authorization issues when requesting FlowAnalysisResults
Tamas Palfy created NIFI-12696: -- Summary: Fix authorization issues when requesting FlowAnalysisResults Key: NIFI-12696 URL: https://issues.apache.org/jira/browse/NIFI-12696 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy When requesting FlowAnalysisResults the authorization logic performed has a couple of issues: # Doesn't handle exceptions thrown when the a component producing a result is tested to be a Port. The logic goes through possible component types and when reaches Ports it throws an exception. # As the logic goest through possible components, the mismatching ones throw ResourceNotFoundExceptions. These are captured but this is a bad practice in general. Throwing and capturing exceptions in non-exceptional cases is bad from both design and performance perspective. # The number of possible components checked is too limited. If a component is unrecognized, the corresponding violation will have a PermissionDTO attached with canRead and canWrite set to false, essentially rendering the result unavailable and thus leading to a false negative. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12696) Fix authorization issues when requesting FlowAnalysisResults
[ https://issues.apache.org/jira/browse/NIFI-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12696: -- Assignee: Tamas Palfy > Fix authorization issues when requesting FlowAnalysisResults > > > Key: NIFI-12696 > URL: https://issues.apache.org/jira/browse/NIFI-12696 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > When requesting FlowAnalysisResults the authorization logic performed has a > couple of issues: > # Doesn't handle exceptions thrown when the a component producing a result is > tested to be a Port. The logic goes through possible component types and when > reaches Ports it throws an exception. > # As the logic goest through possible components, the mismatching ones throw > ResourceNotFoundExceptions. These are captured but this is a bad practice in > general. Throwing and capturing exceptions in non-exceptional cases is bad > from both design and performance perspective. > # The number of possible components checked is too limited. If a component is > unrecognized, the corresponding violation will have a PermissionDTO attached > with canRead and canWrite set to false, essentially rendering the result > unavailable and thus leading to a false negative. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12612) Fix JASN1Reader to handle OBJECT IDENTIFIER
[ https://issues.apache.org/jira/browse/NIFI-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12612: -- Assignee: Tamas Palfy > Fix JASN1Reader to handle OBJECT IDENTIFIER > --- > > Key: NIFI-12612 > URL: https://issues.apache.org/jira/browse/NIFI-12612 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > OBJECT IDENTIFIER is a special asn1 type that is not recognized by > JASN1Reader. Instead it tries to handle it as a Record as a fallback. > The fix would be to simply treat (convert really) OBJECT IDENTIFIER types as > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12612) Fix JASN1Reader to handle OBJECT IDENTIFIER
Tamas Palfy created NIFI-12612: -- Summary: Fix JASN1Reader to handle OBJECT IDENTIFIER Key: NIFI-12612 URL: https://issues.apache.org/jira/browse/NIFI-12612 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy OBJECT IDENTIFIER is a special asn1 type that is not recognized by JASN1Reader. Instead it tries to handle it as a Record as a fallback. The fix would be to simply treat (convert really) OBJECT IDENTIFIER types as strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9464) Provenance Events files corrupted
[
https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804425#comment-17804425
]
Tamas Palfy commented on NIFI-9464:
---
Hi [~markap14]
I managed to kind of reproduce the issue by carefully debugging in a very
particular way. Simply adding a delay before renaming the .toc.tmp doesn't work
because it slows down the compress thread so much that it cannot keep up with
the timer-driven thread to cause any race condition.
I could see that the locking didn't work properly due to the different
EventManager objects.
Based on that do you think we can merge this change?
> Provenance Events files corrupted
> -
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.11.0, 1.15.0
> Environment: java 11, centos 7, nifi standalone
>Reporter: Wiktor Kubicki
>Assignee: Tamas Palfy
>Priority: Minor
> Fix For: 1.25.0, 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
> Failed to retrieve Provenance Events from repository due to: Attempted to
> skip to byte offset 9149491 for 1125432890.prov.gz but file does not have
> that many bytes (TOC
> Reader=StandardTocReader[file=//provenance_repository/toc/1125432890.toc,
> compressed=false]): java.io.EOFException: Attempted to skip to byte offset
> 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
> Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
> compressed=false])
> {code}
> It is criticaly important for me to have 100% sure of my logs. It happened
> about 100 times in last 1 year for 15 *.prov.gz files:
> {code:java}
> -rw-rw-rw-. 1 user user 1013923 Oct 17 21:17 1075441276.prov.gz
> -rw-rw-rw-. 1 user user 1345431 Oct 24 13:06 1083362251.prov.gz
> -rw-rw-rw-. 1 user user 1359282 Oct 25 13:07 1084546392.prov.gz
> -rw-rw-rw-. 1 user user 1155791 Nov 2 17:08 1094516954.prov.gz
> -rw-rw-r--. 1 user user 974136 Nov 18 22:07 1113402183.prov.gz
> -rw-rw-r--. 1 user user 1125608 Nov 28 22:00 1125097576.prov.gz
> -rw-rw-r--. 1 user user 1248319 Nov 29 04:30 1125432890.prov.gz
> -rw-rw-r--. 1 user user 832120 Feb 2 2021 661957813.prov.gz
> -rw-rw-r--. 1 user user 1110978 Mar 17 2021 734807613.prov.gz
> -rw-rw-r--. 1 user user 1506819 Apr 16 2021 786154249.prov.gz
> -rw-rw-r--. 1 user user 1763198 May 25 2021 852626782.prov.gz
> -rw-rw-r--. 1 user user 1580598 Jun 15 08:32 891934274.prov.gz
> -rw-rw-r--. 1 user user 2960296 Jun 28 17:07 917991812.prov.gz
> -rw-rw-r--. 1 user user 1808037 Jun 28 17:37 918051650.prov.gz
> -rw-rw-rw-. 1 user user 765924 Aug 14 13:09 991505484.prov.gz
> {code}
> BTW it's interesting why thera ere different chmods
> My config for provenance (BTW if you see posibbility for tune it, please tell
> me):
> {code:java}
> nifi.provenance.repository.directory.default=/../provenance_repository
> nifi.provenance.repository.max.storage.time=730 days
> nifi.provenance.repository.max.storage.size=512 GB
> nifi.provenance.repository.rollover.time=10 mins
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID
> nifi.provenance.repository.indexed.attributes=
> nifi.provenance.repository.index.shard.size=1 GB
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=1
> nifi.provenance.repository.buffer.size=10
> {code}
> Now my provenance repo has 140GB of data.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-9464) Provenance Events files corrupted
[
https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796442#comment-17796442
]
Tamas Palfy commented on NIFI-9464:
---
[~markap14] I think I found the root cause of this issue but couldn't confirm
it 100% due to the complexity of the provenance repository bundle and because
it's very hard to reproduce the issue. Maybe you can help with it and review my
pull request?
The provenance journal files are regularly compressed on dedicated threads.
When it is done the content of the uncompressed .prov file is compressed and
written into a .prov.gz file, after which the .prov file gets deleted. Also,
the corresponding .toc file is updated but in a different way. The new .toc
file is named as .toc.tmp and after the .prov file is deleted the .toc.tmp is
renamed to .toc.
The SiteToSiteProvenanceReportingTask of course runs on a Timer-Driven thread.
It is important that when it references a compressed .prov.gz it also properly
references the new .toc file.
However a race condition can occur and it's possible for the Time-Driven thread
to run when the .prov file is deleted but the .toc.tmp file is not yet renamed
to .toc. The original .toc file still exists though so it is paired with the
.prov.gz. That can lead to the reported error.
This wasn't always the case. Both places use an EventFileManager to acquire
locks properly.
However [PR|https://github.com/apache/nifi/pull/1686] for NIFI-3594 introduced
a change in WriteAheadProvenanceRepository after which the 2 places started
using a different EventFileManager object.
This is the important part:
{code:java}
@Override
public synchronized void initialize(final EventReporter eventReporter,
final Authorizer authorizer, final ProvenanceAuthorizableFactory
resourceFactory, final IdentifierLookup idLookup) throws IOException {
final EventFileManager fileManager = new EventFileManager();
final RecordReaderFactory recordReaderFactory = (file, logs, maxChars)
-> {
fileManager.obtainReadLock(file);
try {
return RecordReaders.newRecordReader(file, logs, maxChars);
} finally {
fileManager.releaseReadLock(file);
}
};
init(recordWriterFactory, recordReaderFactory, eventReporter,
authorizer, resourceFactory);
}
synchronized void init(RecordWriterFactory recordWriterFactory,
RecordReaderFactory recordReaderFactory,
final EventReporter eventReporter, final Authorizer
authorizer,
final ProvenanceAuthorizableFactory resourceFactory)
throws IOException {
final EventFileManager fileManager = new EventFileManager();
...
}
{code}
My assessment is that this change (that the EventFileManager instances should
be different) is unintentional but the aforementioned change is too complex for
me to be able to tell 100%.
I'm going to open a pull request according to my assessment.
> Provenance Events files corrupted
> -
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.11.0, 1.15.0
> Environment: java 11, centos 7, nifi standalone
>Reporter: Wiktor Kubicki
>Assignee: Tamas Palfy
>Priority: Minor
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
> Failed to retrieve Provenance Events from repository due to: Attempted to
> skip to byte offset 9149491 for 1125432890.prov.gz but file does not have
> that many bytes (TOC
> Reader=StandardTocReader[file=//provenance_repository/toc/1125432890.toc,
> compressed=false]): java.io.EOFException: Attempted to skip to byte offset
> 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
> Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
> compressed=false])
> {code}
> It is criticaly important for me to have 100% sure of my logs. It happened
> about 100 times in last 1 year for 15 *.prov.gz files:
> {code:java}
> -rw-rw-rw-. 1 user user 1013923 Oct 17 21:17 1075441276.prov.gz
> -rw-rw-rw-. 1 user user 1345431 Oct 24 13:06 1083362251.prov.gz
> -rw-rw-rw-. 1 user user 1359282 Oct 25 13:07 1084546392.prov.gz
> -rw-rw-rw-. 1 user user 1155791 Nov 2 17:08 1094516954.prov.gz
> -rw-rw-r--. 1 user user 974136 Nov 18 22:07 1113402183.prov.gz
> -rw-rw-r--. 1 user user 1125608 Nov 28 22:00 1125097576.prov.gz
> -rw-rw-r--. 1 user user 1248319 Nov 29 04:30 1125432890.prov.gz
> -rw-rw-r--. 1 user user 832120 Feb 2 2021 661957813.prov.gz
> -rw-rw-r--. 1 user user 1110978 Mar 17 2021 734807613.prov.gz
> -rw-rw-r--. 1 user user 1506819 Apr 16
[jira] [Assigned] (NIFI-9464) Provenance Events files corrupted
[
https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy reassigned NIFI-9464:
-
Assignee: Tamas Palfy
> Provenance Events files corrupted
> -
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.11.0, 1.15.0
> Environment: java 11, centos 7, nifi standalone
>Reporter: Wiktor Kubicki
>Assignee: Tamas Palfy
>Priority: Minor
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
> Failed to retrieve Provenance Events from repository due to: Attempted to
> skip to byte offset 9149491 for 1125432890.prov.gz but file does not have
> that many bytes (TOC
> Reader=StandardTocReader[file=//provenance_repository/toc/1125432890.toc,
> compressed=false]): java.io.EOFException: Attempted to skip to byte offset
> 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
> Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
> compressed=false])
> {code}
> It is criticaly important for me to have 100% sure of my logs. It happened
> about 100 times in last 1 year for 15 *.prov.gz files:
> {code:java}
> -rw-rw-rw-. 1 user user 1013923 Oct 17 21:17 1075441276.prov.gz
> -rw-rw-rw-. 1 user user 1345431 Oct 24 13:06 1083362251.prov.gz
> -rw-rw-rw-. 1 user user 1359282 Oct 25 13:07 1084546392.prov.gz
> -rw-rw-rw-. 1 user user 1155791 Nov 2 17:08 1094516954.prov.gz
> -rw-rw-r--. 1 user user 974136 Nov 18 22:07 1113402183.prov.gz
> -rw-rw-r--. 1 user user 1125608 Nov 28 22:00 1125097576.prov.gz
> -rw-rw-r--. 1 user user 1248319 Nov 29 04:30 1125432890.prov.gz
> -rw-rw-r--. 1 user user 832120 Feb 2 2021 661957813.prov.gz
> -rw-rw-r--. 1 user user 1110978 Mar 17 2021 734807613.prov.gz
> -rw-rw-r--. 1 user user 1506819 Apr 16 2021 786154249.prov.gz
> -rw-rw-r--. 1 user user 1763198 May 25 2021 852626782.prov.gz
> -rw-rw-r--. 1 user user 1580598 Jun 15 08:32 891934274.prov.gz
> -rw-rw-r--. 1 user user 2960296 Jun 28 17:07 917991812.prov.gz
> -rw-rw-r--. 1 user user 1808037 Jun 28 17:37 918051650.prov.gz
> -rw-rw-rw-. 1 user user 765924 Aug 14 13:09 991505484.prov.gz
> {code}
> BTW it's interesting why thera ere different chmods
> My config for provenance (BTW if you see posibbility for tune it, please tell
> me):
> {code:java}
> nifi.provenance.repository.directory.default=/../provenance_repository
> nifi.provenance.repository.max.storage.time=730 days
> nifi.provenance.repository.max.storage.size=512 GB
> nifi.provenance.repository.rollover.time=10 mins
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID
> nifi.provenance.repository.indexed.attributes=
> nifi.provenance.repository.index.shard.size=1 GB
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=1
> nifi.provenance.repository.buffer.size=10
> {code}
> Now my provenance repo has 140GB of data.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12241) Efficient Parquet Splitting
[
https://issues.apache.org/jira/browse/NIFI-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12241:
---
Fix Version/s: 1.25.0
2.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Efficient Parquet Splitting
> ---
>
> Key: NIFI-12241
> URL: https://issues.apache.org/jira/browse/NIFI-12241
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Rajmund Takacs
>Assignee: Rajmund Takacs
>Priority: Major
> Labels: feature, performance, pull-request-available
> Fix For: 1.25.0, 2.0.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> SplitParquet processor that expects as input a FlowFile with Parquet content
> and would take as parameter a number of records as the split configuration.
> The processor would generate X flow files with unmodified content and would
> add attributes with the offsets required to read the group of rows in the
> flowfile's content.
> Then the Parquet Reader would be improved to accept optional flow file
> attributes containing the information so that the reader can only read the
> required part of the data.
> Instead of having something like
> {noformat}
> X -> SplitRecord (Parquet / JSON) -> ...{noformat}
> It'd be something like
> {noformat}
> X -> SplitParquet -> ConvertRecord (Parquet / JSON) -> ...{noformat}
> The goal here is to increase the overall efficiency of this operation for
> extremely large Parquet files (hundreds of GBs). With the second approach, it
> could leverage multi-threading for processing a single file.
> SplitParquet processor should also have a property (true/false) to write
> zero-content flow files. The existing FetchParquet processor should be
> enhanced to accept the flow file attributes for giving offsets. It'd give
> something like
> {noformat}
> X -> SplitParquet -> FetchParquet (JSON Writer) -> ...{noformat}
> This way, a load balanced connection could be used between SplitParquet and
> FetchParquet in order to distribute the work across the nodes (without
> transferring a lot of data across the nodes of the cluster).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12092) Add exponential backoff to JettyWebsocketClient reconnect
[ https://issues.apache.org/jira/browse/NIFI-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-12092: --- Fix Version/s: 2.0.0 1.24.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add exponential backoff to JettyWebsocketClient reconnect > - > > Key: NIFI-12092 > URL: https://issues.apache.org/jira/browse/NIFI-12092 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.24.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The reconnection attempts were happening too quickly in JettyWebsocketClient. > A customizable exponential backoff with 20% jitter is added to prevent > overloading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12159) Flow Analysis results should provide name for Funnels
[ https://issues.apache.org/jira/browse/NIFI-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-12159: --- Description: Flow Analysis results show a displayed name for the violating component. This defaults to the name of component set by the user. Funnels have no name so the displayed name becomes empty. In this case the result object should have the constant "Funnel" as displayed name. was: Flow Analysis results report the name of the violating component. Funnels have no name so it becomes empty. In this case the result object should have the constant "Funnel" as name. > Flow Analysis results should provide name for Funnels > - > > Key: NIFI-12159 > URL: https://issues.apache.org/jira/browse/NIFI-12159 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Flow Analysis results show a displayed name for the violating component. This > defaults to the name of component set by the user. Funnels have no name so > the displayed name becomes empty. > In this case the result object should have the constant "Funnel" as displayed > name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12159) Flow Analysis results should provide name for Funnels
Tamas Palfy created NIFI-12159: -- Summary: Flow Analysis results should provide name for Funnels Key: NIFI-12159 URL: https://issues.apache.org/jira/browse/NIFI-12159 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Flow Analysis results report the name of the violating component. Funnels have no name so it becomes empty. In this case the result object should have the constant "Funnel" as name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12159) Flow Analysis results should provide name for Funnels
[ https://issues.apache.org/jira/browse/NIFI-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12159: -- Assignee: Tamas Palfy > Flow Analysis results should provide name for Funnels > - > > Key: NIFI-12159 > URL: https://issues.apache.org/jira/browse/NIFI-12159 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Flow Analysis results report the name of the violating component. Funnels > have no name so it becomes empty. > In this case the result object should have the constant "Funnel" as name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12131) Fix potential NPE when RuleViolationManager is null in AbstractFlowManager.
[ https://issues.apache.org/jira/browse/NIFI-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-12131: -- Assignee: Tamas Palfy > Fix potential NPE when RuleViolationManager is null in AbstractFlowManager. > --- > > Key: NIFI-12131 > URL: https://issues.apache.org/jira/browse/NIFI-12131 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12131) Fix potential NPE when RuleViolationManager is null in AbstractFlowManager.
Tamas Palfy created NIFI-12131: -- Summary: Fix potential NPE when RuleViolationManager is null in AbstractFlowManager. Key: NIFI-12131 URL: https://issues.apache.org/jira/browse/NIFI-12131 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11978) NPE when trying to disable Management Controller Service
[
https://issues.apache.org/jira/browse/NIFI-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-11978:
---
Description:
-With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.-
With https://issues.apache.org/jira/browse/NIFI-11556 when verifying if we can
update controller service referencing components we check if the Process Group
of the Controller Service allows scheduling of components individually. However
this part of the change didn't take into account that Management Controller
Services don't belong to any Process Group and a NullPointerException is thrown
when we try to disable one:
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
NIFI-11556
was:
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
NIFI-11556
> NPE when trying to disable Management Controller Service
>
>
> Key: NIFI-11978
> URL: https://issues.apache.org/jira/browse/NIFI-11978
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> -With https://issues.apache.org/jira/browse/NIFI-11556 controller services
> implement the GroupedComponent interface. However Management Controller
> Services (the ones available and managed from the Controller Settings) don’t
> have groups.-
> With https://issues.apache.org/jira/browse/NIFI-11556 when verifying if we
> can update controller service referencing components we check if the Process
> Group of the Controller Service allows scheduling of components individually.
> However this part of the change didn't take into account that Management
> Controller Services don't belong to any Process Group and a
> NullPointerException is thrown when we try to disable one:
> {code:java}
> public class StandardControllerServiceDAO extends ComponentDAO implements
> ControllerServiceDAO {
> ...
> @Override
> public void verifyUpdateReferencingComponents(final String
> controllerServiceId, final ScheduledState scheduledState, final
> ControllerServiceState controllerServiceState) {
> final ControllerServiceNode controllerService =
> locateControllerService(controllerServiceId);
>
> controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
> {code}
> which throws NullPointerException (as there is no process group).
> Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
> NIFI-11556
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11978) NPE when trying to disable Management Controller Service
[
https://issues.apache.org/jira/browse/NIFI-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy reassigned NIFI-11978:
--
Assignee: Tamas Palfy
> NPE when trying to disable Management Controller Service
>
>
> Key: NIFI-11978
> URL: https://issues.apache.org/jira/browse/NIFI-11978
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
>
> With https://issues.apache.org/jira/browse/NIFI-11556 controller services
> implement the GroupedComponent interface. However Management Controller
> Services (the ones available and managed from the Controller Settings) don’t
> have groups.
> Not sure about the extent of the issues but I can confirm a concrete one: the
> management controller services can’t be disabled because that requires a
> check of the referencing components and the logic includes
> {code:java}
> public class StandardControllerServiceDAO extends ComponentDAO implements
> ControllerServiceDAO {
> ...
> @Override
> public void verifyUpdateReferencingComponents(final String
> controllerServiceId, final ScheduledState scheduledState, final
> ControllerServiceState controllerServiceState) {
> final ControllerServiceNode controllerService =
> locateControllerService(controllerServiceId);
>
> controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
> {code}
> which throws NullPointerException (as there is no process group).
> Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
> NIFI-11556
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11978) NPE when trying to disable Management Controller Service
[
https://issues.apache.org/jira/browse/NIFI-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-11978:
---
Description:
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
NIFI-11556
was:
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
> NPE when trying to disable Management Controller Service
>
>
> Key: NIFI-11978
> URL: https://issues.apache.org/jira/browse/NIFI-11978
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Tamas Palfy
>Priority: Major
>
> With https://issues.apache.org/jira/browse/NIFI-11556 controller services
> implement the GroupedComponent interface. However Management Controller
> Services (the ones available and managed from the Controller Settings) don’t
> have groups.
> Not sure about the extent of the issues but I can confirm a concrete one: the
> management controller services can’t be disabled because that requires a
> check of the referencing components and the logic includes
> {code:java}
> public class StandardControllerServiceDAO extends ComponentDAO implements
> ControllerServiceDAO {
> ...
> @Override
> public void verifyUpdateReferencingComponents(final String
> controllerServiceId, final ScheduledState scheduledState, final
> ControllerServiceState controllerServiceState) {
> final ControllerServiceNode controllerService =
> locateControllerService(controllerServiceId);
>
> controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
> {code}
> which throws NullPointerException (as there is no process group).
> Tagging [~markap14] and [~exceptionfactory] as author and reviewer of
> NIFI-11556
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11978) NPE when trying to disable Management Controller Service
[
https://issues.apache.org/jira/browse/NIFI-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-11978:
---
Description:
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
was:
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
> NPE when trying to disable Management Controller Service
>
>
> Key: NIFI-11978
> URL: https://issues.apache.org/jira/browse/NIFI-11978
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Tamas Palfy
>Priority: Major
>
> With https://issues.apache.org/jira/browse/NIFI-11556 controller services
> implement the GroupedComponent interface. However Management Controller
> Services (the ones available and managed from the Controller Settings) don’t
> have groups.
> Not sure about the extent of the issues but I can confirm a concrete one: the
> management controller services can’t be disabled because that requires a
> check of the referencing components and the logic includes
> {code:java}
> public class StandardControllerServiceDAO extends ComponentDAO implements
> ControllerServiceDAO {
> ...
> @Override
> public void verifyUpdateReferencingComponents(final String
> controllerServiceId, final ScheduledState scheduledState, final
> ControllerServiceState controllerServiceState) {
> final ControllerServiceNode controllerService =
> locateControllerService(controllerServiceId);
>
> controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
> {code}
> which throws NullPointerException (as there is no process group).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11978) NPE when trying to disable Management Controller Service
Tamas Palfy created NIFI-11978:
--
Summary: NPE when trying to disable Management Controller Service
Key: NIFI-11978
URL: https://issues.apache.org/jira/browse/NIFI-11978
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Tamas Palfy
With https://issues.apache.org/jira/browse/NIFI-11556 controller services
implement the GroupedComponent interface. However Management Controller
Services (the ones available and managed from the Controller Settings) don’t
have groups.
Not sure about the extent of the issues but I can confirm a concrete one: the
management controller services can’t be disabled because that requires a check
of the referencing components and the logic includes
{code:java}
public class StandardControllerServiceDAO extends ComponentDAO implements
ControllerServiceDAO {
...
@Override
public void verifyUpdateReferencingComponents(final String
controllerServiceId, final ScheduledState scheduledState, final
ControllerServiceState controllerServiceState) {
final ControllerServiceNode controllerService =
locateControllerService(controllerServiceId);
controllerService.getProcessGroup().verifyCanScheduleComponentsIndividually();
{code}
which throws NullPointerException (as there is no process group).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11907) Backport from NIFI-11241 moving classes from nifi-record-serialization-services to nifi-json-record-utils
[ https://issues.apache.org/jira/browse/NIFI-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11907: --- Description: NIFI-11241 Moved JsonRecordSource and JsonSchemaInference classes from the nifi-record-serialization-services to the nifi-json-record-utils modul. This change should be backported to the version 1 support branch so that other modules can use these classes (without depending on nifi-record-serialization-services that exposes components - which would lead to double exposure). was:NIFI-11241 Moved JsonRecordSource and JsonSchemaInference classes from > Backport from NIFI-11241 moving classes from > nifi-record-serialization-services to nifi-json-record-utils > - > > Key: NIFI-11907 > URL: https://issues.apache.org/jira/browse/NIFI-11907 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Priority: Major > > NIFI-11241 Moved JsonRecordSource and JsonSchemaInference classes from the > nifi-record-serialization-services to the nifi-json-record-utils modul. > This change should be backported to the version 1 support branch so that > other modules can use these classes (without depending on > nifi-record-serialization-services that exposes components - which would lead > to double exposure). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11907) Backport from NIFI-11241 moving classes from nifi-record-serialization-services to nifi-json-record-utils
[ https://issues.apache.org/jira/browse/NIFI-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11907: --- Description: NIFI-11241 Moved JsonRecordSource and JsonSchemaInference classes from > Backport from NIFI-11241 moving classes from > nifi-record-serialization-services to nifi-json-record-utils > - > > Key: NIFI-11907 > URL: https://issues.apache.org/jira/browse/NIFI-11907 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Priority: Major > > NIFI-11241 Moved JsonRecordSource and JsonSchemaInference classes from -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11907) Backport from NIFI-11241 moving classes from nifi-record-serialization-services to nifi-json-record-utils
Tamas Palfy created NIFI-11907: -- Summary: Backport from NIFI-11241 moving classes from nifi-record-serialization-services to nifi-json-record-utils Key: NIFI-11907 URL: https://issues.apache.org/jira/browse/NIFI-11907 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-11178:
---
Fix Version/s: 2.0.0
1.23.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Fix For: 2.0.0, 1.23.0
>
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png, image-2023-06-28-20-53-21-210.png,
> image-2023-06-28-20-54-00-887.png, image-2023-06-28-20-54-32-188.png
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can
[jira] [Updated] (NIFI-11369) MergeContent unexpected behaviour
[ https://issues.apache.org/jira/browse/NIFI-11369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11369: --- Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > MergeContent unexpected behaviour > - > > Key: NIFI-11369 > URL: https://issues.apache.org/jira/browse/NIFI-11369 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.20.0 >Reporter: Tamas Neumer >Assignee: Peter Turcsanyi >Priority: Minor > Fix For: 2.0.0, 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Hi, > I would like to merge FlowFiles and came across the following documentation: > [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.MergeContent/additionalDetails.html|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/[%E2%80%A6].nifi.processors.standard.MergeContent/additionalDetails.html] > Quoting from it: _"It is not necessary that all FlowFiles have this value, > but at least one FlowFile in the bin must have this value or the bin will > never be complete..."_ > I have built a flow that assigns the following attributes to the FlowFiles: > * fragment.identifier > * fragment.index > * segment.original.filename > As stated in the documentation mentioned above, the attribute fragment.count > is just assigned to the last FlowFiles with the correct number. > The error I experience is that all the flow files without fragment.count > provoke an error stating: > .. Cannot Defragment ...because it does not have an integer value for the > fragment.count attribute; routing to failure > > Could you help us out please? > Thank you! > Kind regards, > Florian and Tamas -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11224) FF attribute support in Where condition of QuerySalesforceObject
[ https://issues.apache.org/jira/browse/NIFI-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11224: --- Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > FF attribute support in Where condition of QuerySalesforceObject > > > Key: NIFI-11224 > URL: https://issues.apache.org/jira/browse/NIFI-11224 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Pierre Villard >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > We say that we support FF attributes for the WHERE condition of the > QuerySalesforceObject processor > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/QuerySalesforceObject.java#L222] > However, we do not evaluate expression language: > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/QuerySalesforceObject.java#L346] > This should be fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11474) Add missing endpoint mergers.
Tamas Palfy created NIFI-11474: -- Summary: Add missing endpoint mergers. Key: NIFI-11474 URL: https://issues.apache.org/jira/browse/NIFI-11474 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy The following endpoints and entities don't have corresponding endpoint mergers: * ConfigurationAnalysisEntity * DELETE component endpoints (e.g. in ReportingTaskResource) * GET PropertyDescriptorEntity endpoints (e.g. in ReportingTaskResource) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11386) Add "resource" and "audience" support for StandardOauth2AccessTokenProvider
Tamas Palfy created NIFI-11386: -- Summary: Add "resource" and "audience" support for StandardOauth2AccessTokenProvider Key: NIFI-11386 URL: https://issues.apache.org/jira/browse/NIFI-11386 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Assignee: Tamas Palfy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11234) NPE in QuerySalesforceObject
[ https://issues.apache.org/jira/browse/NIFI-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11234: --- Fix Version/s: 1.21.0 > NPE in QuerySalesforceObject > > > Key: NIFI-11234 > URL: https://issues.apache.org/jira/browse/NIFI-11234 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In case of Property Based queries if the RecordWriter service is not set an > NPE is thrown. The RecordWriter should be made mandatory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11147) Allow QuerySalesforceObject to query all existing fields
[ https://issues.apache.org/jira/browse/NIFI-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11147: --- Fix Version/s: 1.21.0 > Allow QuerySalesforceObject to query all existing fields > > > Key: NIFI-11147 > URL: https://issues.apache.org/jira/browse/NIFI-11147 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently the Field Names property of QuerySalesforceObject is required and > must contain the names of the fields the user wants to return. However in a > schema drift use case, the user may want to add a field to a Salesforce > object and have the NiFi flow continue without needing alteration. > This Jira is to make it possible for QuerySalesforceObject to return all > fields from an object. A suggestion is to make Field Names optional and if it > is not set, all fields are queried. The documentation should be updated to > match the behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10966) Add the feature to QuerySalesforceObject to accept custom queries
[ https://issues.apache.org/jira/browse/NIFI-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10966: --- Fix Version/s: 1.21.0 > Add the feature to QuerySalesforceObject to accept custom queries > - > > Key: NIFI-10966 > URL: https://issues.apache.org/jira/browse/NIFI-10966 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Extend the QuerySalesforceObject processor with a new property that accepts > custom SOQL queries. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11094) Allow CaptureChangeMySQL to send multiple events per FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11094: --- Fix Version/s: 1.21.0 > Allow CaptureChangeMySQL to send multiple events per FlowFile > - > > Key: NIFI-11094 > URL: https://issues.apache.org/jira/browse/NIFI-11094 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > It would be nice if there could be a Events Per FlowFile Strategy property > for CaptureChangeMySQL that could allow for things like N events per FlowFile > or one full transaction per FlowFile. It can help lower overhead downstream > and increase the overall performance of the flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11234) NPE in QuerySalesforceObject
[ https://issues.apache.org/jira/browse/NIFI-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11234: --- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > NPE in QuerySalesforceObject > > > Key: NIFI-11234 > URL: https://issues.apache.org/jira/browse/NIFI-11234 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In case of Property Based queries if the RecordWriter service is not set an > NPE is thrown. The RecordWriter should be made mandatory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10966) Add the feature to QuerySalesforceObject to accept custom queries
[ https://issues.apache.org/jira/browse/NIFI-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10966: --- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add the feature to QuerySalesforceObject to accept custom queries > - > > Key: NIFI-10966 > URL: https://issues.apache.org/jira/browse/NIFI-10966 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Extend the QuerySalesforceObject processor with a new property that accepts > custom SOQL queries. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11144) Fix failing tests for ConsumeJMS/PublishJMS
[ https://issues.apache.org/jira/browse/NIFI-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-11144. Fix Version/s: 2.0.0 Resolution: Fixed > Fix failing tests for ConsumeJMS/PublishJMS > --- > > Key: NIFI-11144 > URL: https://issues.apache.org/jira/browse/NIFI-11144 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Nandor Soma Abonyi >Assignee: Nandor Soma Abonyi >Priority: Major > Labels: JMS > Fix For: 2.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11094) Allow CaptureChangeMySQL to send multiple events per FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-11094: --- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Allow CaptureChangeMySQL to send multiple events per FlowFile > - > > Key: NIFI-11094 > URL: https://issues.apache.org/jira/browse/NIFI-11094 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > It would be nice if there could be a Events Per FlowFile Strategy property > for CaptureChangeMySQL that could allow for things like N events per FlowFile > or one full transaction per FlowFile. It can help lower overhead downstream > and increase the overall performance of the flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10846) GetSmbFile issue after upgrading to Nifi 1.18.0
[
https://issues.apache.org/jira/browse/NIFI-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-10846:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> GetSmbFile issue after upgrading to Nifi 1.18.0
> ---
>
> Key: NIFI-10846
> URL: https://issues.apache.org/jira/browse/NIFI-10846
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.18.0
> Environment: Nifi Docker on RHEL7.9
> Samba Server : SVM on a metrocluster Netapp, model AFFA400, Ontap version 9.6
>Reporter: Florent
>Assignee: Peter Turcsanyi
>Priority: Blocker
> Fix For: 1.20.0
>
> Attachments: getsmb_error.png, nifi-app-debug.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After upgrading Nifi from 1.17.0 to 1.18.0, we saw some issue regarding
> Processors o.apache.nifi.processors.smb.*
>
> a Simple GetSmbFile works perfectly in 1.17.0 and after upgrading to 1.18.0
> we saw this error
> {code:java}
> 022-11-21 10:16:29,272 ERROR [Timer-Driven Process Thread-5]
> o.apache.nifi.processors.smb.GetSmbFile
> GetSmbFile[id=8b56acf9-0184-1000-ac23-874fb1140496] Could not establish smb
> connection because of error com.hierynomus.mssmb2.SMBApiException:
> STATUS_ACCESS_DENIED (0xc022): Create failed for
> \\MYSERVER\MySHARE\Directory
> at com.hierynomus.smbj.share.Share.receive(Share.java:380)
> at com.hierynomus.smbj.share.Share.sendReceive(Share.java:359)
> at com.hierynomus.smbj.share.Share.createFile(Share.java:156)
> at
> com.hierynomus.smbj.share.DiskShare.createFileAndResolve(DiskShare.java:75)
> at com.hierynomus.smbj.share.DiskShare.access$100(DiskShare.java:55)
> at com.hierynomus.smbj.share.DiskShare$2.apply(DiskShare.java:109)
> at com.hierynomus.smbj.share.DiskShare$2.apply(DiskShare.java:105)
> at
> com.hierynomus.smbj.paths.PathResolver$1.resolve(PathResolver.java:32)
> at
> com.hierynomus.smbj.paths.SymlinkPathResolver.resolve(SymlinkPathResolver.java:62)
> at
> com.hierynomus.smbj.share.DiskShare.resolveAndCreateFile(DiskShare.java:105)
> at com.hierynomus.smbj.share.DiskShare.open(DiskShare.java:65)
> at com.hierynomus.smbj.share.DiskShare.exists(DiskShare.java:214)
> at
> com.hierynomus.smbj.share.DiskShare.folderExists(DiskShare.java:210)
> at
> org.apache.nifi.processors.smb.GetSmbFile.performListing(GetSmbFile.java:334)
> at
> org.apache.nifi.processors.smb.GetSmbFile.onTrigger(GetSmbFile.java:404)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Suppressed: com.hierynomus.mssmb2.SMBApiException:
> STATUS_ACCESS_DENIED (0xc022): Error closing connection to
> \\MYSERVER\MySHARE
> at
> com.hierynomus.smbj.share.TreeConnect.close(TreeConnect.java:72)
> at com.hierynomus.smbj.share.Share.close(Share.java:116)
> at org.apache.nifi.processors.s {code}
> {{{}We have the same error "{}}}{{{}STATUS_ACCESS_DENIED
> (0xc022){}}}{{{}" with other processor like ListSmb{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-11123) Fine tune GCP Vision processor's JSON Payload default value and docs
[ https://issues.apache.org/jira/browse/NIFI-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-11123. Resolution: Fixed > Fine tune GCP Vision processor's JSON Payload default value and docs > > > Key: NIFI-11123 > URL: https://issues.apache.org/jira/browse/NIFI-11123 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Kalman Jantner >Assignee: Kalman Jantner >Priority: Major > Fix For: 1.20.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > GCP Vision processor has such default value for JSON Payload property that > could cause side effect as the output could trigger the input ListGCSObject > processor again and would trigger a loop. > > The goal is to find a way where the example flow remain simple but has no > side effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11107) ConsumeIMAP and ConsumePOP3 with OAUTH
Tamas Palfy created NIFI-11107: -- Summary: ConsumeIMAP and ConsumePOP3 with OAUTH Key: NIFI-11107 URL: https://issues.apache.org/jira/browse/NIFI-11107 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy In ConsumeIMAP and ConsumePOP3 add support for OAUTH based authorization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11107) ConsumeIMAP and ConsumePOP3 with OAUTH
[ https://issues.apache.org/jira/browse/NIFI-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-11107: -- Assignee: Tamas Palfy > ConsumeIMAP and ConsumePOP3 with OAUTH > -- > > Key: NIFI-11107 > URL: https://issues.apache.org/jira/browse/NIFI-11107 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > In ConsumeIMAP and ConsumePOP3 add support for OAUTH based authorization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10974) Incorrect warning message on memory usage from MonitorMemory
[
https://issues.apache.org/jira/browse/NIFI-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-10974:
---
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Incorrect warning message on memory usage from MonitorMemory
>
>
> Key: NIFI-10974
> URL: https://issues.apache.org/jira/browse/NIFI-10974
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
> Fix For: 1.20.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> There are certain cases when the MonitorMemory reporting task is sending
> incorrect warning messages on memory usage crossing the threshold.
> {+}Repro steps{+}:
> * In the MonitorMemory reporting task, set the threshold value to low enough
> that the current memory usage is crossing that value.
> * Restart NiFi while the monitoring task is still running.
> * After the restart set the threshold to higher value above the actual
> memory usage.
> * The warning messages will be still send despite the threshold is above the
> actual memory usage.
> The main problem is that the *isCollectionUsageThresholdExceeded()* method
> call on *MemoryPoolMXBean* have a *gcSensor* registered at the VM that's 'on'
> attribute will get stuck on 'true' value. This will cause the method to
> always return with exceeded. The sensor will get reseted on the next garbage
> collection but until that it will send false messages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-10953) GCP Vision AI processors
[ https://issues.apache.org/jira/browse/NIFI-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-10953. Fix Version/s: 1.20.0 Resolution: Fixed > GCP Vision AI processors > > > Key: NIFI-10953 > URL: https://issues.apache.org/jira/browse/NIFI-10953 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Kalman Jantner >Priority: Major > Fix For: 1.20.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Create processors to interact with GCP Vision AI managed service to: > * detect text in images > * detect handwriting in images > * detect text in files > * detect faces > * detect image properties > * detect labels > * etc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11050) PutEmail with OAuth
[ https://issues.apache.org/jira/browse/NIFI-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-11050: -- Assignee: Tamas Palfy > PutEmail with OAuth > --- > > Key: NIFI-11050 > URL: https://issues.apache.org/jira/browse/NIFI-11050 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Improve the PutEmail processor so that it's possible to send emails with > OAuth authentication. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11050) PutEmail with OAuth
Tamas Palfy created NIFI-11050: -- Summary: PutEmail with OAuth Key: NIFI-11050 URL: https://issues.apache.org/jira/browse/NIFI-11050 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy Improve the PutEmail processor so that it's possible to send emails with OAuth authentication. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10969) Create extension point for Signer Override in AWS S3 processors
[
https://issues.apache.org/jira/browse/NIFI-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-10969:
---
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Create extension point for Signer Override in AWS S3 processors
> ---
>
> Key: NIFI-10969
> URL: https://issues.apache.org/jira/browse/NIFI-10969
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Labels: aws
> Fix For: 1.20.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> AWS S3 processors have the {{Signer Override}} property where V4 or V2
> signers can be configured. These signers are part of the AWS SDK for Java
> library.
> However, some special use cases require to use a user defined signer (e.g
> when a gateway is located between NiFi and the AWS Service and it changes the
> request).
> To support these scenarios, add an extension point in the processors for
> configuring user provided custom signers.
> The S3 processors may be configured to use Assume Role credentials which
> accesses the STS service in the backround. For this reason, the custom signer
> extension point is needed for that service too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10832) Create PutSalesforceObject processor
[ https://issues.apache.org/jira/browse/NIFI-10832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10832: --- Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Create PutSalesforceObject processor > > > Key: NIFI-10832 > URL: https://issues.apache.org/jira/browse/NIFI-10832 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 1.20.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Processor able to create new entries in Salesforce -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10955) In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
Tamas Palfy created NIFI-10955:
--
Summary: In JASN1Reader allow preprocess of ASN files to reconcile
unsupported features
Key: NIFI-10955
URL: https://issues.apache.org/jira/browse/NIFI-10955
Project: Apache NiFi
Issue Type: Improvement
Reporter: Tamas Palfy
The ASN specification allows the creation of valid ASN files that has features
unrecognized by the asn1bean library we are using in JASN1Reader.
We can add a preprocessing step that creates modified versions of the provided
ASN files (leaving the originals intact) that removes unsupported features in a
way that makes them less strict but otherwise should still be compatible with
incoming data.
Identified unsupported features:
* Certain constraint types
** E.g.
{code:java}
SomeType ::= INTEGER (ALL EXCEPT (0..15))
{code}
* Extension marker (a.k.a "ellipsis" or
{code:java}
...
{code}
)
** Can occur in constraints (e.g.
{code:java}
SomeType ::= INTEGER (0..8,...,100..200)
{code}
although
{code:java}
SomeType ::= INTEGER (0..8,...)
{code}
works)
** or in type definitions (e.g.
{code:java}
RootType::= SEQUENCE {
field1 INTEGER,
field2 INTEGER,
...,
field3 INTEGER
}
{code}
but this seems to work as well).
* Version brackets e.g.
{code:java}
SomeType ::= SEQUENCE {
integerField1 INTEGER,
integerField2 INTEGER,
...,
[[ -- from version 2
integerField3 INTEGER,
integerField4 INTEGER ]]
}
{code}
Seems to require extension marker as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10871) Intermittent CSRF HTTP 403 in Clustered Deployments
[
https://issues.apache.org/jira/browse/NIFI-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-10871:
---
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Intermittent CSRF HTTP 403 in Clustered Deployments
>
>
> Key: NIFI-10871
> URL: https://issues.apache.org/jira/browse/NIFI-10871
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Security
>Affects Versions: 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.20.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> NiFi 1.14.0 introduced Cross-Site Request Forgery mitigation as part of
> updates to support JSON Web Token resolution using HttpOnly Session cookies.
> The standard Spring Security
> [CsrfFilter|https://github.com/spring-projects/spring-security/blob/5.7.5/web/src/main/java/org/springframework/security/web/csrf/CsrfFilter.java]
> includes a Request Matcher property to control whether filtering operations
> should be applied, but the CsrfFilter checks the Request Matcher after
> generating and saving a new token.
> Standalone deployments of NiFi can reuse the CSRF Request Token when the HTTP
> request includes the value in a {{Cookie}} header, but the NiFi HTTP Request
> Replicator removes the CSRF Request Token cookie before sending the request
> to other cluster nodes.
> As a result of these implementation details, NiFi cluster nodes receiving
> replicated HTTP requests generate and return a new CSRF Request Token. The
> NiFi user interface receives the new CSRF Request Token and uses it to set
> the custom {{Request-Token}} HTTP Header on subsequent requests. This is not
> an issue for HTTP GET requests, but requests using methods such as POST, PUT,
> or DELETE can return an HTTP 403 Forbidden response from the Spring Security
> CsrfFilter due to receiving mismatched {{__Secure-Request-Token}} Cookie and
> {{Request-Token}} Header values.
> This issue is intermittent because it depends on the web browser
> simultaneously receiving an HTTP response with a new Secure-Request-Token
> Cookie while preparing to send a new HTTP request with a {{Request-Token}}
> Header that contains the value from the previously received cookie.
> Resolving the problem should include adjusting the behavior of the CsrfFilter
> to avoid setting a new cookie on requests that do not require filtering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-10793) Comment is not populated when process group is created via API
[
https://issues.apache.org/jira/browse/NIFI-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy resolved NIFI-10793.
Fix Version/s: 1.19.0
Resolution: Fixed
> Comment is not populated when process group is created via API
> --
>
> Key: NIFI-10793
> URL: https://issues.apache.org/jira/browse/NIFI-10793
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0
>Reporter: Zsihovszki Krisztina
>Assignee: Zsihovszki Krisztina
>Priority: Major
> Fix For: 1.19.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The comment field is not set for the newly created process group.
>
> {code:java}
> curl -H "Content-Type: application/json" -X POST -H "Authorization: Bearer
> " -k -d '{"revision": {"version": 0}, "position": {"x": 0.0, "y":
> 0.0}, "component": {"name": "Curl Test Group", "comments": "This is a
> comment"}}'
> https://:8443/nifi-api/process-groups//process-groups
> {code}
>
> The comment ("This is a comment") is not shown on the created process group.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10764) Fix handling of BIT STRING in nifi-asn1-service
[ https://issues.apache.org/jira/browse/NIFI-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10764: --- Status: Patch Available (was: In Progress) > Fix handling of BIT STRING in nifi-asn1-service > --- > > Key: NIFI-10764 > URL: https://issues.apache.org/jira/browse/NIFI-10764 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently when trying to convert a BIT STRING from an asn data file it > results in an exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10765) Add better error reporting for JASN1Reader
[ https://issues.apache.org/jira/browse/NIFI-10765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-10765: -- Assignee: Tamas Palfy > Add better error reporting for JASN1Reader > -- > > Key: NIFI-10765 > URL: https://issues.apache.org/jira/browse/NIFI-10765 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > The JASN1Reader relies on https://github.com/beanit/asn1bean to compile ASN > schema files and to deserialize ASN data files. > The schema compilation step can fail for many different reasons and the error > is not reported to the user properly (or in any way really) so understanding > and fixing errors in the schema file is very hard currently. > We should report meaningful error messages to the bulletin and/or the > controller service validation error indicator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10765) Add better error reporting for JASN1Reader
Tamas Palfy created NIFI-10765: -- Summary: Add better error reporting for JASN1Reader Key: NIFI-10765 URL: https://issues.apache.org/jira/browse/NIFI-10765 Project: Apache NiFi Issue Type: Improvement Reporter: Tamas Palfy The JASN1Reader relies on https://github.com/beanit/asn1bean to compile ASN schema files and to deserialize ASN data files. The schema compilation step can fail for many different reasons and the error is not reported to the user properly (or in any way really) so understanding and fixing errors in the schema file is very hard currently. We should report meaningful error messages to the bulletin and/or the controller service validation error indicator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10722) Add handling of TBCD-STRING in nifi-asn1-services
[ https://issues.apache.org/jira/browse/NIFI-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10722: --- Status: Patch Available (was: In Progress) > Add handling of TBCD-STRING in nifi-asn1-services > - > > Key: NIFI-10722 > URL: https://issues.apache.org/jira/browse/NIFI-10722 > Project: Apache NiFi > Issue Type: Task >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > While TBCD-STRING as a datatype is not part of the ASN specification, it is > widely used among telco companies who are the most prominent users of this > technology. > NiFi treats TBCD-STRING no different than the underlying ASN type (which is > OCTET STRING) but in telco environment it requires a specific handling. > We could extend our current implementation so that when TBCD-STRING is > encountered it handles it according to the telco expectations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10764) Fix handling of BIT STRING in nifi-asn1-service
[ https://issues.apache.org/jira/browse/NIFI-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-10764: -- Assignee: Tamas Palfy > Fix handling of BIT STRING in nifi-asn1-service > --- > > Key: NIFI-10764 > URL: https://issues.apache.org/jira/browse/NIFI-10764 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Currently when trying to convert a BIT STRING from an asn data file it > results in an exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10764) Fix handling of BIT STRING in nifi-asn1-service
Tamas Palfy created NIFI-10764: -- Summary: Fix handling of BIT STRING in nifi-asn1-service Key: NIFI-10764 URL: https://issues.apache.org/jira/browse/NIFI-10764 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy Currently when trying to convert a BIT STRING from an asn data file it results in an exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10722) Add handling of TBCD-STRING in nifi-asn1-services
Tamas Palfy created NIFI-10722: -- Summary: Add handling of TBCD-STRING in nifi-asn1-services Key: NIFI-10722 URL: https://issues.apache.org/jira/browse/NIFI-10722 Project: Apache NiFi Issue Type: Task Reporter: Tamas Palfy While TBCD-STRING as a datatype is not part of the ASN specification, it is widely used among telco companies who are the most prominent users of this technology. NiFi treats TBCD-STRING no different than the underlying ASN type (which is OCTET STRING) but in telco environment it requires a specific handling. We could extend our current implementation so that when TBCD-STRING is encountered it handles it according to the telco expectations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10722) Add handling of TBCD-STRING in nifi-asn1-services
[ https://issues.apache.org/jira/browse/NIFI-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-10722: -- Assignee: Tamas Palfy > Add handling of TBCD-STRING in nifi-asn1-services > - > > Key: NIFI-10722 > URL: https://issues.apache.org/jira/browse/NIFI-10722 > Project: Apache NiFi > Issue Type: Task >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > While TBCD-STRING as a datatype is not part of the ASN specification, it is > widely used among telco companies who are the most prominent users of this > technology. > NiFi treats TBCD-STRING no different than the underlying ASN type (which is > OCTET STRING) but in telco environment it requires a specific handling. > We could extend our current implementation so that when TBCD-STRING is > encountered it handles it according to the telco expectations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10597) Change 'App Config File' property to external resource in Box processors
[
https://issues.apache.org/jira/browse/NIFI-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy resolved NIFI-10597.
Resolution: Fixed
> Change 'App Config File' property to external resource in Box processors
>
>
> Key: NIFI-10597
> URL: https://issues.apache.org/jira/browse/NIFI-10597
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Zsihovszki Krisztina
>Assignee: Zsihovszki Krisztina
>Priority: Minor
> Fix For: 1.19.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The Box app configuration can be specified as JSON file for Box processors,
> see
> "app-config-file" in JsonConfigBasedBoxClientService.
> Google processors use GCPCredentialsControllerService which includes a
> similar property for the configuration file in JSON format:
> "service-account-json-file".
> The "service-account-json-file" property uses
> {code:java}
> .identifiesExternalResource(ResourceCardinality.SINGLE,
> ResourceType.FILE){code}
> in its PropertyDescriptor,
> while "app-config-file" in Box case uses
> {code:java}
> .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR){code}
> It'd be useful to replace the StandardValidators.FILE_EXISTS_VALIDATOR usage
> with
> identifiesExternalResource in the Box case. It contains the same check for
> file existence and also checks file accessbility, does not allow to specify
> directory instead of file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10528) Extract JSON readers to util module for use in Salesforce NAR
[ https://issues.apache.org/jira/browse/NIFI-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10528: --- Fix Version/s: 1.18.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Extract JSON readers to util module for use in Salesforce NAR > - > > Key: NIFI-10528 > URL: https://issues.apache.org/jira/browse/NIFI-10528 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 1.18.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The Salesforce NAR currently depends on the record serialization services NAR > in order to use the JsonTreeRecordReader. We should extract these readers to > a util module so that they can be reused and the Salesforce NAR can depend on > standard services API NAR. > Also the OAuth2 API jar is being included when it should be provided from > standard services API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10513) QuerySalesforceObject can only list the first 2000 records
[ https://issues.apache.org/jira/browse/NIFI-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10513: --- Fix Version/s: 1.18.0 Resolution: Fixed Status: Resolved (was: Patch Available) > QuerySalesforceObject can only list the first 2000 records > -- > > Key: NIFI-10513 > URL: https://issues.apache.org/jira/browse/NIFI-10513 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 1.18.0 > > Time Spent: 50m > Remaining Estimate: 0h > > QuerySalesforceObject does not use pagination to list object with more than > 2000 records. In case of more than 2000 records, a page cursor is returned in > the response body. In order to capture cursor fields in the future we should > extends JsonTreeRowRecordReader with the capability of capturing non-record > fields based on a predicate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10523) Additional documentation for List/FetchGoogleDive
Tamas Palfy created NIFI-10523: -- Summary: Additional documentation for List/FetchGoogleDive Key: NIFI-10523 URL: https://issues.apache.org/jira/browse/NIFI-10523 Project: Apache NiFi Issue Type: Task Reporter: Tamas Palfy Update used documentation with the following info: * How to get the Folder ID * How to enable Google Drive API in GCP * How to grant access to a Google Drive folder for a Service Account -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10364) Simplify connection/session handling in SmbjClientService
[
https://issues.apache.org/jira/browse/NIFI-10364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-10364:
---
Fix Version/s: 1.18.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Simplify connection/session handling in SmbjClientService
> -
>
> Key: NIFI-10364
> URL: https://issues.apache.org/jira/browse/NIFI-10364
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 1.18.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {{SmbjClientServiceProvider}} and {{SmbjClientService}} share a couple of
> data fields and also the connection and session handling logic is overlapping
> the two classes. The connection/session (re)creation logic could be moved to
> {{SmbjClientServiceProvider}} and {{SmbjClientService}} would only contain
> and handle the live session (if creation was successful).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10427) Add List-/Fetch Box processors
[ https://issues.apache.org/jira/browse/NIFI-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10427: --- Status: Patch Available (was: In Progress) > Add List-/Fetch Box processors > -- > > Key: NIFI-10427 > URL: https://issues.apache.org/jira/browse/NIFI-10427 > Project: Apache NiFi > Issue Type: Task >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Add processor that are able list and fetch files from a given Box folder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10230) FetchSmb processor
[ https://issues.apache.org/jira/browse/NIFI-10230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-10230. Fix Version/s: 1.18.0 Resolution: Fixed > FetchSmb processor > -- > > Key: NIFI-10230 > URL: https://issues.apache.org/jira/browse/NIFI-10230 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Kulik Gábor >Assignee: Kulik Gábor >Priority: Major > Fix For: 1.18.0 > > Time Spent: 4h > Remaining Estimate: 0h > > As a user I'd like to be able to fetch remote files shared via SMB protocol > using a NiFi processor. > - create FetchSmb processor > - use SmbConnectionPoolService added in NIFI-10212 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10404) TailFile processor's persistent state not cleaned up
[ https://issues.apache.org/jira/browse/NIFI-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-10404. Fix Version/s: 1.18.0 Resolution: Fixed > TailFile processor's persistent state not cleaned up > > > Key: NIFI-10404 > URL: https://issues.apache.org/jira/browse/NIFI-10404 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zsihovszki Krisztina >Assignee: Zsihovszki Krisztina >Priority: Major > Fix For: 1.18.0 > > Time Spent: 50m > Remaining Estimate: 0h > > TailFile processor's persistent state map is not maintained properly, entries > are added to this map but the references to non existent files are not moved > from the map. > At NiFi restart the states map (storing the TailFileObjects) is empty and it > is > [filled|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L360] > with entries based on the persistent state map. > Each TailFileObject contains a byte buffer with the size of 65536 bytes. > Since the persistent state map is not cleaned up and it may contain thousands > of file references, a lot of TailFileObject may be created at startup, each > consuming 65536 bytes which can end up in out of memory error. > The persistent state map needs to be actualized based on the actual files as > it is done for the [states > map|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L374]. > The allocated byte buffer size (65536 bytes) should be configurable to be > able to use smaller buffer size (default value would be 65536 bytes for > backward compatibility). > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10427) Add List-/Fetch Box processors
Tamas Palfy created NIFI-10427: -- Summary: Add List-/Fetch Box processors Key: NIFI-10427 URL: https://issues.apache.org/jira/browse/NIFI-10427 Project: Apache NiFi Issue Type: Task Reporter: Tamas Palfy Add processor that are able list and fetch files from a given Box folder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10427) Add List-/Fetch Box processors
[ https://issues.apache.org/jira/browse/NIFI-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-10427: -- Assignee: Tamas Palfy > Add List-/Fetch Box processors > -- > > Key: NIFI-10427 > URL: https://issues.apache.org/jira/browse/NIFI-10427 > Project: Apache NiFi > Issue Type: Task >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Add processor that are able list and fetch files from a given Box folder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10379) FechGoogleDrive record-based input handling revoke
[ https://issues.apache.org/jira/browse/NIFI-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10379: --- Description: *FechGoogleDrive* can read input from flowfile contents as records. After an extensive review several technical and conceptual issues have emerged from this functionality. Also the same result can be achieved by adding other processors between the List and the Fetch processor. Overall this functionality is best to be removed. was: *FechGoogleDrive* can read input from flowfile contents as records. However the following cases are not handled correctly: # There is no property on the processor to configure what field in the records are to be used as file identifiers (based on which fetch is executed). Instead it is a hardcoded string value. *Edit:* Upon further consideration the record-based approach poses technical challenges if we wanted to be properly customizable. Instead we will consider this to mainly support the output of the ListGoogleDrive processor. Still can be used from differenc sources but the expected schema is pre-defined. # It is possible for the processor to fetch successfully some files but later experience a general error which sends the incoming flowfile to an error relationship. This is a mixed result (some records succeed but then _all records_ fail) and should not happen. Either all records should be processed (resulting one flowfile for each sent to a success or an error relationship) or none of them (resulting only the incoming flowfile being sent to another error relationship). # _session.xxx(flowFile)_ calls are used as if those were _viod_ (and operated on the incoming flowFile). Should use the returned flowfile instead. > FechGoogleDrive record-based input handling revoke > -- > > Key: NIFI-10379 > URL: https://issues.apache.org/jira/browse/NIFI-10379 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *FechGoogleDrive* can read input from flowfile contents as records. > After an extensive review several technical and conceptual issues have > emerged from this functionality. Also the same result can be achieved by > adding other processors between the List and the Fetch processor. > Overall this functionality is best to be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10379) FechGoogleDrive record-based input handling revoke
[ https://issues.apache.org/jira/browse/NIFI-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10379: --- Summary: FechGoogleDrive record-based input handling revoke (was: FechGoogleDrive record-based input handling improvement) > FechGoogleDrive record-based input handling revoke > -- > > Key: NIFI-10379 > URL: https://issues.apache.org/jira/browse/NIFI-10379 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *FechGoogleDrive* can read input from flowfile contents as records. > However the following cases are not handled correctly: > # There is no property on the processor to configure what field in the > records are to be used as file identifiers (based on which fetch is > executed). Instead it is a hardcoded string value. > *Edit:* Upon further consideration the record-based approach poses technical > challenges if we wanted to be properly customizable. > Instead we will consider this to mainly support the output of the > ListGoogleDrive processor. Still can be used from differenc sources but the > expected schema is pre-defined. > # It is possible for the processor to fetch successfully some files but > later experience a general error which sends the incoming flowfile to an > error relationship. This is a mixed result (some records succeed but then > _all records_ fail) and should not happen. Either all records should be > processed (resulting one flowfile for each sent to a success or an error > relationship) or none of them (resulting only the incoming flowfile being > sent to another error relationship). > # _session.xxx(flowFile)_ calls are used as if those were _viod_ (and > operated on the incoming flowFile). Should use the returned flowfile instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10379) FechGoogleDrive record-based input handling improvement
[ https://issues.apache.org/jira/browse/NIFI-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-10379: --- Description: *FechGoogleDrive* can read input from flowfile contents as records. However the following cases are not handled correctly: # There is no property on the processor to configure what field in the records are to be used as file identifiers (based on which fetch is executed). Instead it is a hardcoded string value. *Edit:* Upon further consideration the record-based approach poses technical challenges if we wanted to be properly customizable. Instead we will consider this to mainly support the output of the ListGoogleDrive processor. Still can be used from differenc sources but the expected schema is pre-defined. # It is possible for the processor to fetch successfully some files but later experience a general error which sends the incoming flowfile to an error relationship. This is a mixed result (some records succeed but then _all records_ fail) and should not happen. Either all records should be processed (resulting one flowfile for each sent to a success or an error relationship) or none of them (resulting only the incoming flowfile being sent to another error relationship). # _session.xxx(flowFile)_ calls are used as if those were _viod_ (and operated on the incoming flowFile). Should use the returned flowfile instead. was: *FechGoogleDrive* can read input from flowfile contents as records. However the following cases are not handled correctly: # There is no property on the processor to configure what field in the records are to be used as file identifiers (based on which fetch is executed). Instead it is a hardcoded string value. # It is possible for the processor to fetch successfully some files but later experience a general error which sends the incoming flowfile to an error relationship. This is a mixed result (some records succeed but then _all records_ fail) and should not happen. Either all records should be processed (resulting one flowfile for each sent to a success or an error relationship) or none of them (resulting only the incoming flowfile being sent to another error relationship). # _session.xxx(flowFile)_ calls are used as if those were _viod_ (and operated on the incoming flowFile). Should use the returned flowfile instead. > FechGoogleDrive record-based input handling improvement > --- > > Key: NIFI-10379 > URL: https://issues.apache.org/jira/browse/NIFI-10379 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > *FechGoogleDrive* can read input from flowfile contents as records. > However the following cases are not handled correctly: > # There is no property on the processor to configure what field in the > records are to be used as file identifiers (based on which fetch is > executed). Instead it is a hardcoded string value. > *Edit:* Upon further consideration the record-based approach poses technical > challenges if we wanted to be properly customizable. > Instead we will consider this to mainly support the output of the > ListGoogleDrive processor. Still can be used from differenc sources but the > expected schema is pre-defined. > # It is possible for the processor to fetch successfully some files but > later experience a general error which sends the incoming flowfile to an > error relationship. This is a mixed result (some records succeed but then > _all records_ fail) and should not happen. Either all records should be > processed (resulting one flowfile for each sent to a success or an error > relationship) or none of them (resulting only the incoming flowfile being > sent to another error relationship). > # _session.xxx(flowFile)_ calls are used as if those were _viod_ (and > operated on the incoming flowFile). Should use the returned flowfile instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
