[jira] [Created] (NIFI-13417) Using Zookeeper as distributed map cache service
Malthe Borch created NIFI-13417: --- Summary: Using Zookeeper as distributed map cache service Key: NIFI-13417 URL: https://issues.apache.org/jira/browse/NIFI-13417 Project: Apache NiFi Issue Type: New Feature Reporter: Malthe Borch The [AtomicDistributedMapCacheClient|https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-standard-services/nifi-distributed-cache-client-service-api/src/main/java/org/apache/nifi/distributed/cache/client/AtomicDistributedMapCacheClient.java] interface is trivially supported by Zookeeper (which is already used for cluster coordination, an easy job in many deployments because there's not much to coordinate when the number of nodes is low). Instead of having to deploy a cache such as Redis (or Redis Sentinel) separately, we should make it possible for Zookeeper to pull double duty here – Zookeeper can support "multi-tenancy" using the "chroot" client configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17855212#comment-17855212
]
Malthe Borch commented on NIFI-5443:
[~pepov] if the flow.xml.gz file is pulled either from the current primary node
or from a backup (in case we're alone), there shouldn't be the problem you
mention – "stop a node for a while".
There's a risk that changes to the flow configuration happen exactly during our
joining the cluster, but that's in frequent and we can just crash letting the
pod restart, hopefully having more luck next time.
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Minor
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11294) ConsumeAzureEventHub should default to processor state for checkpointing
[ https://issues.apache.org/jira/browse/NIFI-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701186#comment-17701186 ] Malthe Borch commented on NIFI-11294: - In that case, Azure Blob Storage would have to be the default strategy which is an acceptable compromise I suppose – it would be more ideal to have Component State as the default because it works without further configuration, but it's the price of compatibility I suppose. > ConsumeAzureEventHub should default to processor state for checkpointing > > > Key: NIFI-11294 > URL: https://issues.apache.org/jira/browse/NIFI-11294 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Major > > The current implementation requires an Azure Blob Storage account to persist > offsets (checkpointing) between restarts. > In NiFi, we have a readily available state store which is the processor state. > We should default to using this storage such that "Storage Account Name" > becomes a non-required field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11294) ConsumeAzureEventHub should default to processor state for checkpointing
Malthe Borch created NIFI-11294: --- Summary: ConsumeAzureEventHub should default to processor state for checkpointing Key: NIFI-11294 URL: https://issues.apache.org/jira/browse/NIFI-11294 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch The current implementation requires an Azure Blob Storage account to persist offsets (checkpointing) between restarts. In NiFi, we have a readily available state store which is the processor state. We should default to using this storage such that "Storage Account Name" becomes a non-required field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10721) PutAzureBlobStorage_v12 should not query for properties after success
Malthe Borch created NIFI-10721: --- Summary: PutAzureBlobStorage_v12 should not query for properties after success Key: NIFI-10721 URL: https://issues.apache.org/jira/browse/NIFI-10721 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch Since we're in control of all the properties of the blob, there is no reason to query for the properties after success. Using the information we already have will obviate the need for this additional roundtrip. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10632) AWS S3 CopyObject
Malthe Borch created NIFI-10632:
---
Summary: AWS S3 CopyObject
Key: NIFI-10632
URL: https://issues.apache.org/jira/browse/NIFI-10632
Project: Apache NiFi
Issue Type: Improvement
Reporter: Malthe Borch
AWS S3 provides a
[CopyObject|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html]
API call which:
{quote}Creates a copy of an object that is already stored in Amazon S3.{quote}
In NiFi, the PutS3Object processor could be extended to support this API such
that one can specify a source for the copy operation, either a different AWS
Credentials Provider Service or simply a different bucket and/or object name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-9972) Azure Blob Storage "Blob from URL"
[ https://issues.apache.org/jira/browse/NIFI-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch reassigned NIFI-9972: -- Assignee: Malthe Borch > Azure Blob Storage "Blob from URL" > -- > > Key: NIFI-9972 > URL: https://issues.apache.org/jira/browse/NIFI-9972 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Assignee: Malthe Borch >Priority: Major > > The Azure SDK for Azure Blob Storage supports the "Put Blob from URL" and > "Copy from URL" operations which allow server-to-server copying of blobs from > one storage account to another. > This is perhaps antithetical to data processing in NiFi (since the > implication is that the client does not actually see any of the data being > transferred), but it is a very relevant data operation. > Today, to accomplish the same from NiFi you would need to call out to for > example > [azcopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy] > which means you lose out on credentials management etc. > Practically speaking, you would provide two credentials, one for the source > and another for the destination and the would-be processor would then engage > in the server-to-server copying operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10491) Azure Blob Storage should have conflict resolution (overwrite mode)
[ https://issues.apache.org/jira/browse/NIFI-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch updated NIFI-10491: Description: PutAzureBlobStorage_v12 currently does not have configurable conflict resolution. If a blob already exists, the flow file is lead to the failure relationship. Instead, there should be three options for conflict behavior: # Fail # Replace # Ignore This is similar to PutDataLakeStorage. was: PutAzureBlobStorage_v12 currently does not have configurable conflict resolution. If a blob already exists, the flow file is lead to the failure relationship. Instead, there should be three options for conflict behavior: # Overwrite # Skip # Skip if source is newer This matches the capability of [azcopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy]. > Azure Blob Storage should have conflict resolution (overwrite mode) > --- > > Key: NIFI-10491 > URL: https://issues.apache.org/jira/browse/NIFI-10491 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Malthe Borch >Assignee: Malthe Borch >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > PutAzureBlobStorage_v12 currently does not have configurable conflict > resolution. > If a blob already exists, the flow file is lead to the failure relationship. > Instead, there should be three options for conflict behavior: > # Fail > # Replace > # Ignore > This is similar to PutDataLakeStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10491) Azure Blob Storage should have conflict resolution (overwrite mode)
[ https://issues.apache.org/jira/browse/NIFI-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch updated NIFI-10491: Description: PutAzureBlobStorage_v12 currently does not have configurable conflict resolution. If a blob already exists, the flow file is lead to the failure relationship. Instead, there should be three options for conflict behavior: # Fail # Replace # Ignore This is similar to PutAzureDataLakeStorage. was: PutAzureBlobStorage_v12 currently does not have configurable conflict resolution. If a blob already exists, the flow file is lead to the failure relationship. Instead, there should be three options for conflict behavior: # Fail # Replace # Ignore This is similar to PutDataLakeStorage. > Azure Blob Storage should have conflict resolution (overwrite mode) > --- > > Key: NIFI-10491 > URL: https://issues.apache.org/jira/browse/NIFI-10491 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Malthe Borch >Assignee: Malthe Borch >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > PutAzureBlobStorage_v12 currently does not have configurable conflict > resolution. > If a blob already exists, the flow file is lead to the failure relationship. > Instead, there should be three options for conflict behavior: > # Fail > # Replace > # Ignore > This is similar to PutAzureDataLakeStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10571) Retrieve groups from Azure AD app registration
[ https://issues.apache.org/jira/browse/NIFI-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch reassigned NIFI-10571: --- Assignee: Malthe Borch > Retrieve groups from Azure AD app registration > -- > > Key: NIFI-10571 > URL: https://issues.apache.org/jira/browse/NIFI-10571 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Malthe Borch >Assignee: Malthe Borch >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > When using OIDC authentication with Azure AD app registration, it's > theoretically possible to retrieve _app roles_ along with the identity token > payload. > These app roles can be transformed into groups. > However, for the NiFi authorization system to work, it should be possible to > list the app roles such that policies can be configured based on them. That > is, a user group provider could enumerate the app roles every so often and > make them available for policy assignment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-7823) Retrieve groups from OIDC token using KeyClock
[ https://issues.apache.org/jira/browse/NIFI-7823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch reassigned NIFI-7823: -- Assignee: Malthe Borch > Retrieve groups from OIDC token using KeyClock > -- > > Key: NIFI-7823 > URL: https://issues.apache.org/jira/browse/NIFI-7823 > Project: Apache NiFi > Issue Type: Wish > Components: Security >Reporter: Raz Dobkies >Assignee: Malthe Borch >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Nifi will can retrueve groups from a token if we will develop > KeyCloakUserGroupProvider. > Keycloak does implement a lot of related functionality, including groups, > users, and such. It would be great to have a first-class integration > available. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10571) Retrieve groups from Azure AD app registration
Malthe Borch created NIFI-10571: --- Summary: Retrieve groups from Azure AD app registration Key: NIFI-10571 URL: https://issues.apache.org/jira/browse/NIFI-10571 Project: Apache NiFi Issue Type: New Feature Reporter: Malthe Borch When using OIDC authentication with Azure AD app registration, it's theoretically possible to retrieve _app roles_ along with the identity token payload. These app roles can be transformed into groups. However, for the NiFi authorization system to work, it should be possible to list the app roles such that policies can be configured based on them. That is, a user group provider could enumerate the app roles every so often and make them available for policy assignment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10491) Azure Blob Storage should have conflict resolution (overwrite mode)
[ https://issues.apache.org/jira/browse/NIFI-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch reassigned NIFI-10491: --- Assignee: Malthe Borch > Azure Blob Storage should have conflict resolution (overwrite mode) > --- > > Key: NIFI-10491 > URL: https://issues.apache.org/jira/browse/NIFI-10491 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Malthe Borch >Assignee: Malthe Borch >Priority: Major > > PutAzureBlobStorage_v12 currently does not have configurable conflict > resolution. > If a blob already exists, the flow file is lead to the failure relationship. > Instead, there should be three options for conflict behavior: > # Overwrite > # Skip > # Skip if source is newer > This matches the capability of > [azcopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10491) Azure Blob Storage should have conflict resolution (overwrite mode)
Malthe Borch created NIFI-10491: --- Summary: Azure Blob Storage should have conflict resolution (overwrite mode) Key: NIFI-10491 URL: https://issues.apache.org/jira/browse/NIFI-10491 Project: Apache NiFi Issue Type: New Feature Reporter: Malthe Borch PutAzureBlobStorage_v12 currently does not have configurable conflict resolution. If a blob already exists, the flow file is lead to the failure relationship. Instead, there should be three options for conflict behavior: # Overwrite # Skip # Skip if source is newer This matches the capability of [azcopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-7628) Parameter context in multi tenant environment
[ https://issues.apache.org/jira/browse/NIFI-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598958#comment-17598958 ] Malthe Borch commented on NIFI-7628: [~a630963] is that not because permission to read parameter context has been given at the top-level process group? Instead, I suppose that permission should be given on the individual process groups. > Parameter context in multi tenant environment > - > > Key: NIFI-7628 > URL: https://issues.apache.org/jira/browse/NIFI-7628 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC >Reporter: naveen kumar saharan >Priority: Major > > In multi-tenant environment, its not possible to restrict the parameters for > different BU/vertical/tenant. > Currently either you can give access(view or modify) to a PG or not. > > But in multi-tenant environment we want the BU/vertical tenant specific > access on Param context so that people don't disturb each others work and its > important wrt to security and compliance also -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10423) Virtual workspaces
Malthe Borch created NIFI-10423:
---
Summary: Virtual workspaces
Key: NIFI-10423
URL: https://issues.apache.org/jira/browse/NIFI-10423
Project: Apache NiFi
Issue Type: New Feature
Reporter: Malthe Borch
In NiFi, there is just a single main canvas.
It would be useful in many cases to be able to create a set of isolated "main
canvases" (each of which being a process group).
The main benefit is that users can navigate to a specific canvas (or {_}virtual
workspace{_}) without having to locate a process group in 2D space.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-10400) Groups and/or roles via KnoxSSO
Malthe Borch created NIFI-10400:
---
Summary: Groups and/or roles via KnoxSSO
Key: NIFI-10400
URL: https://issues.apache.org/jira/browse/NIFI-10400
Project: Apache NiFi
Issue Type: New Feature
Reporter: Malthe Borch
When using KnoxSSO with OIDC we're able to get both _roles_ and _groups_ as
part of user authentication.
{code:java}
2022-08-27 10:27:39,608 DEBUG filter.Pac4jIdentityAdapter
(Pac4jIdentityAdapter.java:doFilter(92)) - User authenticated as: #OidcProfile#
| id: [REDACTED] | attributes: {sub=[REDACTED], amr=["pwd"], roles=["Reader"],
iss=https://sts.windows.net/[REDACTED]/, oid=[REDACTED],
preferred_username=mbo...@gmail.com, tid=[REDACTED], ipaddr=[REDACTED], exp=Sat
Aug 27 11:27:38 CEST 2022, iat=Sat Aug 27 10:22:38 CEST 2022, email=[REDACTED],
ver=1.0, groups=["[\"[REDACTED]\",\"[REDACTED]\"]"], uti=[REDACTED],
given_name=[REDACTED], token_expiration_advance=-1, aud=[[REDACTED]],
unique_name=[REDACTED], nbf=Sat Aug 27 10:22:38 CEST 2022, idp=live.com,
rh=[REDACTED], name=[REDACTED], expiration=Sat Aug 27 11:27:38 CEST 2022,
family_name=[REDACTED]} | roles: [] | permissions: [] | isRemembered: false |
clientName: OidcClient | linkedId: null |
{code}
The roles are more immediately useful because they can be plain text rather
than opaque ids, for example I have assigned the role "Reader".
Note that this is using Azure AD where roles are assigned using {_}app roles{_}.
It would be very useful if any roles and/or groups were available as groups
when authorizing the identity in NiFi:
{code:java}
2022-08-27 10:27:40,056 INFO [NiFi Web Server-147]
o.a.n.w.a.c.AccessDeniedExceptionMapper identity[[REDACTED]], groups[] does not
have permission to access the requested resource. Unknown user with identity
'[REDACTED]'. Returning Forbidden response.
{code}
As shown above, the groups here are an empty array.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585380#comment-17585380 ] Malthe Borch commented on NIFI-10313: - I can confirm that this patch works for us. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Fix For: 1.18.0 > > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > Time Spent: 40m > Remaining Estimate: 0h > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9993) Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archi
[
https://issues.apache.org/jira/browse/NIFI-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582774#comment-17582774
]
Malthe Borch commented on NIFI-9993:
The fix was merged into the 1.16 support branch on May 12, so it would seem
that the fix doesn't entirely fix the issue.
> Processors cannot write to Content Repo: Unable to write flowfile content to
> content repository container default due to archive file size constraints;
> waiting for archive cleanup
> ---
>
> Key: NIFI-9993
> URL: https://issues.apache.org/jira/browse/NIFI-9993
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Sometimes, we can see NiFi log the following log message over and over:
> {code:java}
> 2022-05-05 10:47:03,267 WARN [Timer-Driven Process Thread-142]
> o.a.n.c.repository.FileSystemRepository Unable to write to container default
> due to archive file size constraints; waiting for archive cleanup {code}
> However, we can also see that there's nothing in the archive to cleanup,
> based on the fact that we also continually see this log message:
> {code:java}
> 2022-05-05 10:01:02,515 INFO [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> bytes) from archive {code}
> If there are any files in the archive directory, we should see them getting
> cleaned up. The fact that we constantly see "Successfully deleted 0 files (0
> bytes) from archive" implies there is nothing in the archive.
> But if there is nothing in the archive, then we should not be seeing any
> notices about "Unable to write to container default due to archive file size
> constraints."
> This results in processors getting blocked for indefinite periods of time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-9993) Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archi
[
https://issues.apache.org/jira/browse/NIFI-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582755#comment-17582755
]
Malthe Borch commented on NIFI-9993:
[~mayki] according to the issue metadata, there's a fix out for 1.16.2 which
should imply that it's also in 1.16.3.
> Processors cannot write to Content Repo: Unable to write flowfile content to
> content repository container default due to archive file size constraints;
> waiting for archive cleanup
> ---
>
> Key: NIFI-9993
> URL: https://issues.apache.org/jira/browse/NIFI-9993
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Sometimes, we can see NiFi log the following log message over and over:
> {code:java}
> 2022-05-05 10:47:03,267 WARN [Timer-Driven Process Thread-142]
> o.a.n.c.repository.FileSystemRepository Unable to write to container default
> due to archive file size constraints; waiting for archive cleanup {code}
> However, we can also see that there's nothing in the archive to cleanup,
> based on the fact that we also continually see this log message:
> {code:java}
> 2022-05-05 10:01:02,515 INFO [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> bytes) from archive {code}
> If there are any files in the archive directory, we should see them getting
> cleaned up. The fact that we constantly see "Successfully deleted 0 files (0
> bytes) from archive" implies there is nothing in the archive.
> But if there is nothing in the archive, then we should not be seeing any
> notices about "Unable to write to container default due to archive file size
> constraints."
> This results in processors getting blocked for indefinite periods of time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580672#comment-17580672 ] Malthe Borch edited comment on NIFI-10313 at 8/17/22 8:44 AM: -- I suppose with SPNEGO authentication at least, the NiFi UI should simply not call this expiration endpoint – but perhaps it's unable to make that decision. was (Author: malthe): I suppose with Kerberos authentication at least, the NiFi UI should simply not call this expiration endpoint – but perhaps it's unable to make that decision. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580672#comment-17580672 ] Malthe Borch commented on NIFI-10313: - I suppose with Kerberos authentication at least, the NiFi UI should simply not call this expiration endpoint – but perhaps it's unable to make that decision. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-4890) OIDC Token Refresh is not done correctly
[ https://issues.apache.org/jira/browse/NIFI-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579646#comment-17579646 ] Malthe Borch commented on NIFI-4890: [~ecerulm], I think this is better handled in the NiFi server code. If a JWT token is expired (this is checked by the server code), then today, the response becomes a 401 Unauthorized. Instead, what should happen is that NiFi should: # Check that the JWT wasn't revoked # Check if we have a stored refresh token # Call the refresh url to get an updated access token and group memberships and such # Issue a new JWT This would then be completely transparent to the user. > OIDC Token Refresh is not done correctly > > > Key: NIFI-4890 > URL: https://issues.apache.org/jira/browse/NIFI-4890 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.5.0 > Environment: Environment: > Browser: Chrome / Firefox > Configuration of NiFi: > - SSL certificate for the server (no client auth) > - OIDC configuration including end_session_endpoint (see the link > https://auth.s.orchestracities.com/auth/realms/default/.well-known/openid-configuration) > >Reporter: Federico Michele Facca >Assignee: Raz Dobkies >Priority: Major > > It looks like the NIFI UI is not refreshing the OIDC token in background, and > because of that, when the token expires, tells you that your session is > expired. and you need to refresh the page, to get a new token. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10343) OIDC identity provider may provide "null" as error message
Malthe Borch created NIFI-10343: --- Summary: OIDC identity provider may provide "null" as error message Key: NIFI-10343 URL: https://issues.apache.org/jira/browse/NIFI-10343 Project: Apache NiFi Issue Type: Bug Reporter: Malthe Borch We've observed an error message "An error occurred while invoking the Token endpoint: null" which seems to stem from the code pulling the error description which may be null. Instead, it should provide both the description and the code – and possibly the HTTP status code as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577456#comment-17577456 ] Malthe Borch commented on NIFI-10313: - I wouldn't say it's resolved because this is still a regression since 1.16 in that we were previously fine running this setup on Chrome (which supports SPNEGO) and now the missing cookie seems to cause problems. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577437#comment-17577437
]
Malthe Borch commented on NIFI-10313:
-
We've figured out why the cookie is missing – it seems that for a browser that
is able to use SPNEGO authentication (such as Chrome in our systems), no
{{__Secure-Authorization-Bearer}} cookie is sent.
Meanwhile, in Firefox (which does not support SPNEGO in our system), we do see
the cookie.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575998#comment-17575998
]
Malthe Borch commented on NIFI-10313:
-
That seems reasonable.
However, given that we're talking hours here and that it's been working up
until 1.17.0, it would seem likely that the main issue here is not about the
expiration of these values.
You mention a cookie named "{{{}__Secure-Authorization-Bearer". I see
"__Secure-Request-Token" instead (with a UUID value assigned).{}}}
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575972#comment-17575972 ] Malthe Borch commented on NIFI-10313: - The "Authentication Expiration" of my `ldap-provider` is 12 hours – and I suppose that as long as it's longer than the JSON Web Token expiration then it should be valid. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575929#comment-17575929
]
Malthe Borch commented on NIFI-10313:
-
Actually, the service object is configured with ClientIP-based session
affinity, relevant snippet:
{code:java}
apiVersion: v1
kind: Service
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
type: LoadBalancer{code}
That is, it should already have a 3-hour affinity to a single pod in the
statefulset.
I checked just now again with 1.16.3 and here there is no problem with our
configuration.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575893#comment-17575893
]
Malthe Borch commented on NIFI-10313:
-
We are not able to use cookie-based (session) affinity in the current setup
since we've enabled SSL passthrough:
{{nginx.ingress.kubernetes.io/ssl-passthrough: true}}
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575898#comment-17575898 ] Malthe Borch commented on NIFI-10313: - [~exceptionfactory] I'm trying to find a reference on why sticky sessions would be required, reading for example [https://exceptionfactory.com/posts/2021/10/23/improving-jwt-authentication-in-apache-nifi/.|https://exceptionfactory.com/posts/2021/10/23/improving-jwt-authentication-in-apache-nifi/] It does mention that the private key which signs the tokens is kept in memory and perhaps specific to each node but it doesn't mention sticky/affinity directly. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575876#comment-17575876 ] Malthe Borch commented on NIFI-10313: - Immediately I load the NiFi UI I see the following resources failing to load (although this does not seem to impact on anything): !NiFi-Errors.PNG! > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch updated NIFI-10313: Attachment: NiFi-Errors.PNG > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313 ] Malthe Borch deleted comment on NIFI-10313: - was (Author: malthe): Immediately I load the NiFi UI I see the following resources failing to load (although this does not seem to impact on anything): !NiFi-Errors.PNG! > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575875#comment-17575875 ] Malthe Borch commented on NIFI-10313: - Immediately I load the NiFi UI I see the following resources failing to load (although this does not seem to impact on anything): !NiFi-Errors.PNG! > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575075#comment-17575075
]
Malthe Borch commented on NIFI-10313:
-
nifi.properties we have:
{code:java}
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=https://login.microsoftonline.com//.well-known/openid-configuration
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=upn{code}
{code:java}
# User login configuration
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider= {code}
Please also see attached [^authorizers.xml].
We are not using sticky sessions. Our ingress controller annotations:
{code:java}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: true
nginx.ingress.kubernetes.io/ssl-redirect: true {code}
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575075#comment-17575075
]
Malthe Borch edited comment on NIFI-10313 at 8/4/22 7:18 AM:
-
In {{nifi.properties}} we have:
{code:java}
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=https://login.microsoftonline.com//.well-known/openid-configuration
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=upn{code}
{code:java}
# User login configuration
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider= {code}
Please also see attached [^authorizers.xml].
We are not using sticky sessions. Our ingress controller annotations:
{code:java}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: true
nginx.ingress.kubernetes.io/ssl-redirect: true {code}
was (Author: malthe):
nifi.properties we have:
{code:java}
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=https://login.microsoftonline.com//.well-known/openid-configuration
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=upn{code}
{code:java}
# User login configuration
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider= {code}
Please also see attached [^authorizers.xml].
We are not using sticky sessions. Our ingress controller annotations:
{code:java}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: true
nginx.ingress.kubernetes.io/ssl-redirect: true {code}
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch updated NIFI-10313: Attachment: authorizers.xml > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574942#comment-17574942
]
Malthe Borch edited comment on NIFI-10313 at 8/3/22 10:06 PM:
--
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{login-identity-providers.xml}} and
OpenId as configured via {{{}nifi.security.user.oidc.*{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
In our setup, NiFi is running via Kubernetes and it's been working fine up
until 1.17.0. We have direct communication between nodes and HTTPS enabled
between them (via keystore and truststore).
was (Author: malthe):
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{login-identity-providers.xml}} and
OpenId as configured via {{{}nifi.security.user.oidc.*{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
In our setup, NiFi is running via Kubernetes and it's been working fine up
until 1.17.0. We have direct communication between nodes and HTTPS enabled
between them.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574942#comment-17574942
]
Malthe Borch edited comment on NIFI-10313 at 8/3/22 10:01 PM:
--
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{login-identity-providers.xml}} and
OpenId as configured via {{{}nifi.security.user.oidc.*{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
In our setup, NiFi is running via Kubernetes and it's been working fine up
until 1.17.0. We have direct communication between nodes and HTTPS enabled
between them.
was (Author: malthe):
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
In our setup, NiFi is running via Kubernetes and it's been working fine up
until 1.17.0. We have direct communication between nodes and HTTPS enabled
between them.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574942#comment-17574942
]
Malthe Borch edited comment on NIFI-10313 at 8/3/22 10:00 PM:
--
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
In our setup, NiFi is running via Kubernetes and it's been working fine up
until 1.17.0. We have direct communication between nodes and HTTPS enabled
between them.
was (Author: malthe):
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574942#comment-17574942
]
Malthe Borch edited comment on NIFI-10313 at 8/3/22 9:58 PM:
-
We're actually not using SAML at all and do not have a setting for
{{{}nifi.security.user.saml.authentication.expiration{}}}.
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
was (Author: malthe):
We're actually not using SAML at all and do not have a setting for
{}nifi.security.user.saml.authentication.expiration{}}}.}}
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574942#comment-17574942
]
Malthe Borch commented on NIFI-10313:
-
We're actually not using SAML at all and do not have a setting for
{}nifi.security.user.saml.authentication.expiration{}}}.}}
We use LDAP and Kerberos, configured in {{{}login-identity-providers.xml{}}}.
Looking through {{nifi-request.log}} it doesn't seem to have any useful
information on this.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574925#comment-17574925 ] Malthe Borch commented on NIFI-10313: - It would seem that NIFI-9849 is a likely culprit of this issue since the traceback mentions code that was introduced here. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Priority: Major > Attachments: nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10313) Unexpected "Access Token not found"
Malthe Borch created NIFI-10313: --- Summary: Unexpected "Access Token not found" Key: NIFI-10313 URL: https://issues.apache.org/jira/browse/NIFI-10313 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.17.0 Reporter: Malthe Borch Attachments: nifi.log I'm experiencing some unexpected "Access Token not found" errors after upgrading to 1.17.0. See attached traceback. What happens is that the NiFi UI seems to work but after a short while the view is redirected to a conflict page (Unable to communicate with NiFi). There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-9849) Refactor SAML 2 Support using Spring Security 5
[
https://issues.apache.org/jira/browse/NIFI-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574922#comment-17574922
]
Malthe Borch edited comment on NIFI-9849 at 8/3/22 8:59 PM:
I'm experiencing some unexpected "Access Token not found" errors that seem
related to this change.
What happens is that the NiFi UI seems to work but after a short while the view
is redirected to a conflict page (Unable to communicate with NiFi).
was (Author: malthe):
I'm experiencing some unexpected "Access Token not found" errors that seem
related to this change (traceback below).
What happens is that the NiFi UI seems to work but after a short while the view
is redirected to a conflict page (Unable to communicate with NiFi). There are
no other problems or evidence of the issue to be found in the logs.
{code:java}
2022-08-02 12:38:53,940 WARN [NiFi Web Server-146]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Access
Token not found. Returning Conflict response.
java.lang.IllegalStateException: Access Token not found
at
org.apache.nifi.web.api.AccessResource.getAccessTokenExpiration(AccessResource.java:459)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
{code}
> Refactor SAML 2 Support using Spring Security 5
> ---
>
> Key: NIFI-9849
> URL: https://issues.apache.org/jira/browse/NIFI-9849
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.17.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Spring Security SAML|https://spring.io/projects/spring-security-saml]
> project, which provides the implementation for SAML integration in NiFi
> reached end of life in October 2021. Spring Security 5.5 and 5.6 include a
> new implementation of support for SAML integration.
> NiFi integration with SAML should be refactored to remove the dependency on
> the unmaintained Spring Security SAML library. The new implementation should
> be built on Spring Security 5
[jira] [Comment Edited] (NIFI-9849) Refactor SAML 2 Support using Spring Security 5
[
https://issues.apache.org/jira/browse/NIFI-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574922#comment-17574922
]
Malthe Borch edited comment on NIFI-9849 at 8/3/22 8:56 PM:
I'm experiencing some unexpected "Access Token not found" errors that seem
related to this change (traceback below).
What happens is that the NiFi UI seems to work but after a short while the view
is redirected to a conflict page (Unable to communicate with NiFi). There are
no other problems or evidence of the issue to be found in the logs.
{code:java}
2022-08-02 12:38:53,940 WARN [NiFi Web Server-146]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Access
Token not found. Returning Conflict response.
java.lang.IllegalStateException: Access Token not found
at
org.apache.nifi.web.api.AccessResource.getAccessTokenExpiration(AccessResource.java:459)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
{code}
was (Author: malthe):
I'm experiencing some unexpected "Access Token not found" errors that seem
related to this change (traceback below).
What happens is that the NiFi UI seems to work but after a short while the view
is redirected to a conflict page. There are no other problems or evidence of
the issue to be found in the logs.
{code:java}
2022-08-02 12:38:53,940 WARN [NiFi Web Server-146]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Access
Token not found. Returning Conflict response.
java.lang.IllegalStateException: Access Token not found
at
org.apache.nifi.web.api.AccessResource.getAccessTokenExpiration(AccessResource.java:459)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
[jira] [Commented] (NIFI-9849) Refactor SAML 2 Support using Spring Security 5
[
https://issues.apache.org/jira/browse/NIFI-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574922#comment-17574922
]
Malthe Borch commented on NIFI-9849:
I'm experiencing some unexpected "Access Token not found" errors that seem
related to this change (traceback below).
What happens is that the NiFi UI seems to work but after a short while the view
is redirected to a conflict page. There are no other problems or evidence of
the issue to be found in the logs.
{code:java}
2022-08-02 12:38:53,940 WARN [NiFi Web Server-146]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Access
Token not found. Returning Conflict response.
java.lang.IllegalStateException: Access Token not found
at
org.apache.nifi.web.api.AccessResource.getAccessTokenExpiration(AccessResource.java:459)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
{code}
> Refactor SAML 2 Support using Spring Security 5
> ---
>
> Key: NIFI-9849
> URL: https://issues.apache.org/jira/browse/NIFI-9849
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.17.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Spring Security SAML|https://spring.io/projects/spring-security-saml]
> project, which provides the implementation for SAML integration in NiFi
> reached end of life in October 2021. Spring Security 5.5 and 5.6 include a
> new implementation of support for SAML integration.
> NiFi integration with SAML should be refactored to remove the dependency on
> the unmaintained Spring Security SAML library. The new implementation should
> be built on Spring Security 5 libraries, which also includes a number of
> transitive dependency upgrades.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-10134) Azure Log Analytics reporting task needs timeout configuration
Malthe Borch created NIFI-10134: --- Summary: Azure Log Analytics reporting task needs timeout configuration Key: NIFI-10134 URL: https://issues.apache.org/jira/browse/NIFI-10134 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch The [AzureLogAnalyticsReportingTask|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.16.2/org.apache.nifi.reporting.azure.loganalytics.AzureLogAnalyticsReportingTask/] does not currently have a timeout, defaulting to a system setting (apparently) which seems to be unset (i.e. infinite wait). We should expose these options to the user and perhaps provide more sane defaults. It would also make sense to consider reusing the client. See [https://stackoverflow.com/a/24935907/647151] for reference on setting timeouts. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-2835) GetAzureEventHub processor should leverage partition offset to better handle restarts
[ https://issues.apache.org/jira/browse/NIFI-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544975#comment-17544975 ] Malthe Borch commented on NIFI-2835: Why isn't GetAzureEventHub checkpointing using processor state? I understand that ConsumeAzureEventHub can use blob storage to do the same but this seems unnecessary when you've got processor state available. > GetAzureEventHub processor should leverage partition offset to better handle > restarts > - > > Key: NIFI-2835 > URL: https://issues.apache.org/jira/browse/NIFI-2835 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Percivall >Assignee: Eric Ulicny >Priority: Major > > The GetAzureEventHub processor utilizes the Azure client that consists of > receivers for each partition. The processor stores them in a map[1] that gets > cleared every time the processor is stopped[2]. These receivers have > partition offsets which keep track of which message it's currently on and > which it should receive next. So currently, when the processor is > stopped/restarted, any tracking of which message is next to be received is > lost. > If instead of clearing the map each time, we hold onto the receivers, or kept > track of the partitionId/Offsets when stopping, (barring any relevant > configuration changes) the processor would restart exactly where it left off > with no loss of data. > This would work very well with NIFI-2826. > [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122 > [2] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-9993) Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archi
[
https://issues.apache.org/jira/browse/NIFI-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538166#comment-17538166
]
Malthe Borch commented on NIFI-9993:
Do we know when this issue was introduced or re-introduced?
> Processors cannot write to Content Repo: Unable to write flowfile content to
> content repository container default due to archive file size constraints;
> waiting for archive cleanup
> ---
>
> Key: NIFI-9993
> URL: https://issues.apache.org/jira/browse/NIFI-9993
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Sometimes, we can see NiFi log the following log message over and over:
> {code:java}
> 2022-05-05 10:47:03,267 WARN [Timer-Driven Process Thread-142]
> o.a.n.c.repository.FileSystemRepository Unable to write to container default
> due to archive file size constraints; waiting for archive cleanup {code}
> However, we can also see that there's nothing in the archive to cleanup,
> based on the fact that we also continually see this log message:
> {code:java}
> 2022-05-05 10:01:02,515 INFO [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> bytes) from archive {code}
> If there are any files in the archive directory, we should see them getting
> cleaned up. The fact that we constantly see "Successfully deleted 0 files (0
> bytes) from archive" implies there is nothing in the archive.
> But if there is nothing in the archive, then we should not be seeing any
> notices about "Unable to write to container default due to archive file size
> constraints."
> This results in processors getting blocked for indefinite periods of time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (NIFI-9972) Azure Blob Storage "Blob from URL"
Malthe Borch created NIFI-9972: -- Summary: Azure Blob Storage "Blob from URL" Key: NIFI-9972 URL: https://issues.apache.org/jira/browse/NIFI-9972 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch The Azure SDK for Azure Blob Storage supports the "Put Blob from URL" and "Copy from URL" operations which allow server-to-server copying of blobs from one storage account to another. This is perhaps antithetical to data processing in NiFi (since the implication is that the client does not actually see any of the data being transferred), but it is a very relevant data operation. Today, to accomplish the same from NiFi you would need to call out to for example [azcopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy] which means you lose out on credentials management etc. Practically speaking, you would provide two credentials, one for the source and another for the destination and the would-be processor would then engage in the server-to-server copying operation. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (NIFI-9888) Publishing metrics to Log Analytics ocassionally fails
Malthe Borch created NIFI-9888: -- Summary: Publishing metrics to Log Analytics ocassionally fails Key: NIFI-9888 URL: https://issues.apache.org/jira/browse/NIFI-9888 Project: Apache NiFi Issue Type: Bug Reporter: Malthe Borch We're experiencing that occasionally, using the Log Analytics reporting task to publish metrics fails with a "403 Forbidden". -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9866) Missing error detail when failing to publish metrics to Log Analytics
Malthe Borch created NIFI-9866: -- Summary: Missing error detail when failing to publish metrics to Log Analytics Key: NIFI-9866 URL: https://issues.apache.org/jira/browse/NIFI-9866 Project: Apache NiFi Issue Type: Bug Reporter: Malthe Borch When pushing metrics to Log Analytics fails (e.g. 403 Forbidden or other non-200 status), only the status code and reason string is reported. The response body should contain a more detailed error however, useful for troubleshooting. We should read this and log the contents. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7593) Azure Blob Storage uses sensitive property for account name
[ https://issues.apache.org/jira/browse/NIFI-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335024#comment-17335024 ] Malthe Borch commented on NIFI-7593: I don't think that's a valid argument since the storage account name will typically be subject to logging as well. What's sensitive in this context is the authentication. > Azure Blob Storage uses sensitive property for account name > --- > > Key: NIFI-7593 > URL: https://issues.apache.org/jira/browse/NIFI-7593 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Minor > Labels: easyfix > > Processors such as > [AzureListBlobStorage|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/] > has account name as a sensitive property. > This seems wrong and also causes some usability issues with parameter context > parameters (which are either sensitive or non-sensitive). > Note that while the fix for this is trivial, it could have real implications > for existing flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7837) Add MDC-mode to LogAttribute
[ https://issues.apache.org/jira/browse/NIFI-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malthe Borch updated NIFI-7837: --- Description: The LogAttribute processor creates a custom-formatted text message out of the attribute and optionally content payload. An optional JSON mode would go some way in making this more accessible to a consumer, however some logging sinks such as [Azure Application Insights|https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-trace-logs] support properties (or _custom dimensions_) that are both more ergonomic and performant. SL4J has support [Mapped Diagnostics Context|http://www.slf4j.org/manual.html#mdc] (MDC) and this is [implemented in some backends|http://logback.qos.ch/manual/mdc.html] such as Logback. Finally, some Logback sinks (e.g. the _appender_ for Microsoft Application Insights) turn that context (a list of key/value entries) into properties. Supporting MDC as a logging mode in addition to the current custom-formatted message and possibly a JSON mode as well would go a long way in enabling better logging from NiFi flows. was: The LogAttribute processor creates a custom-formatted text message out of the attribute and optionally content payload. An optional JSON mode would go some way in making this more accessible to a consumer, however some logging sinks such as [Azure Application Insights|https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-trace-logs] support properties (or _custom dimensions_) that are both more ergonomic and performant. Logback has support for [Mapped Diagnostics Context|http://logback.qos.ch/manual/mdc.html] (MDC) and again, some logging sinks turn that context (a list of key/value entries) into properties. Supporting MDC as a logging mode in addition to the current custom-formatted message and possibly a JSON mode as well would go a long way in enabling better logging from NiFi flows. > Add MDC-mode to LogAttribute > > > Key: NIFI-7837 > URL: https://issues.apache.org/jira/browse/NIFI-7837 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Minor > > The LogAttribute processor creates a custom-formatted text message out of the > attribute and optionally content payload. > An optional JSON mode would go some way in making this more accessible to a > consumer, however some logging sinks such as [Azure Application > Insights|https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-trace-logs] > support properties (or _custom dimensions_) that are both more ergonomic and > performant. > SL4J has support [Mapped Diagnostics > Context|http://www.slf4j.org/manual.html#mdc] (MDC) and this is [implemented > in some backends|http://logback.qos.ch/manual/mdc.html] such as Logback. > Finally, some Logback sinks (e.g. the _appender_ for Microsoft Application > Insights) turn that context (a list of key/value entries) into properties. > Supporting MDC as a logging mode in addition to the current custom-formatted > message and possibly a JSON mode as well would go a long way in enabling > better logging from NiFi flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7837) Add MDC-mode to LogAttribute
Malthe Borch created NIFI-7837: -- Summary: Add MDC-mode to LogAttribute Key: NIFI-7837 URL: https://issues.apache.org/jira/browse/NIFI-7837 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch The LogAttribute processor creates a custom-formatted text message out of the attribute and optionally content payload. An optional JSON mode would go some way in making this more accessible to a consumer, however some logging sinks such as [Azure Application Insights|https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-trace-logs] support properties (or _custom dimensions_) that are both more ergonomic and performant. Logback has support for [Mapped Diagnostics Context|http://logback.qos.ch/manual/mdc.html] (MDC) and again, some logging sinks turn that context (a list of key/value entries) into properties. Supporting MDC as a logging mode in addition to the current custom-formatted message and possibly a JSON mode as well would go a long way in enabling better logging from NiFi flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7677) Upgrade Azure Storage API to latest (v12) to support extra SPP
[ https://issues.apache.org/jira/browse/NIFI-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200593#comment-17200593 ] Malthe Borch commented on NIFI-7677: Why not 1.5 series – it does not break anything from the user perspective. Meanwhile, Microsoft has done this 2.x to 12.x upgrade across the board on all platforms – I think it would be good for NiFi to make the transition sooner rather than later unless there are very compelling arguments for holding back. > Upgrade Azure Storage API to latest (v12) to support extra SPP > -- > > Key: NIFI-7677 > URL: https://issues.apache.org/jira/browse/NIFI-7677 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.11.4 >Reporter: Stijn IJzermans >Priority: Minor > Labels: azure, cloud, encryption, sensitive > Time Spent: 3h 50m > Remaining Estimate: 0h > > To enable support for extra types of SPP (service principal providers) when > connecting to storage, we need to upgrade the Azure storage dependencies. > This is required for the improvement referred in NIFI-6615 and related to > epic NIFI-5481. > From version 12 onwards, the Azure storage SDK's support the identity > objects, and hence we open up support for new types of authentication such as: > * ManagedIdentityCredential > * EnvironmentCredential > * ClientSecretCredential > * ClientCertificateCredential > (Referred here: > [https://docs.microsoft.com/en-us/java/api/overview/azure/identity-readme?view=azure-java-stable)] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7769) Wait/Notify processors should default to keeping state in ZK
Malthe Borch created NIFI-7769: -- Summary: Wait/Notify processors should default to keeping state in ZK Key: NIFI-7769 URL: https://issues.apache.org/jira/browse/NIFI-7769 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch Most processors that keep state can now use ZooKeeper as a default in addition to a distributed cache service. This should also apply to Wait/Notify. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7763) Split and map functions in NiFi Expression Language
Malthe Borch created NIFI-7763:
--
Summary: Split and map functions in NiFi Expression Language
Key: NIFI-7763
URL: https://issues.apache.org/jira/browse/NIFI-7763
Project: Apache NiFi
Issue Type: Improvement
Reporter: Malthe Borch
This is a proposal to extend the NiFi Expression Language with functions
{{split}} and {{map}} that combine to allow expression such as:
{{${keys:split(","):map(key => "t1.${key} = t2.${key}"):join(" and ")}}}
That is, split the {{keys}} attribute and dynamically create a join condition
based on a set of primary keys.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-3785) Allow user to move Controller Service
[ https://issues.apache.org/jira/browse/NIFI-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183934#comment-17183934 ] Malthe Borch commented on NIFI-3785: [~pvillard] are you imaging a cut/copy and paste process or move to parent group? > Allow user to move Controller Service > - > > Key: NIFI-3785 > URL: https://issues.apache.org/jira/browse/NIFI-3785 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Mark Payne >Priority: Major > > Currently, there is no way to move a Controller Service from one Process > Group to another. I would like to be able to move a Controller Service from > its Process Group to the "parent" Process Group in order to broaden its scope. > The ability to move the Controller Service to a child group in order to > narrow its scope would be convenient as well but may be difficult to provide > an intuitive user experience around. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-3785) Allow user to move Controller Service
[ https://issues.apache.org/jira/browse/NIFI-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183934#comment-17183934 ] Malthe Borch edited comment on NIFI-3785 at 8/25/20, 11:06 AM: --- [~pvillard] are you imagining a cut/copy and paste process or move to parent group? was (Author: malthe): [~pvillard] are you imaging a cut/copy and paste process or move to parent group? > Allow user to move Controller Service > - > > Key: NIFI-3785 > URL: https://issues.apache.org/jira/browse/NIFI-3785 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Mark Payne >Priority: Major > > Currently, there is no way to move a Controller Service from one Process > Group to another. I would like to be able to move a Controller Service from > its Process Group to the "parent" Process Group in order to broaden its scope. > The ability to move the Controller Service to a child group in order to > narrow its scope would be convenient as well but may be difficult to provide > an intuitive user experience around. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7168) List/Fetch SFTP parameter object not a ECParameterSpec
[
https://issues.apache.org/jira/browse/NIFI-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178902#comment-17178902
]
Malthe Borch commented on NIFI-7168:
[~alopresto] we have been running into this snag as well – it seems like you've
found a solution but is this something that could be patched into the NiFi
distribution?
> List/Fetch SFTP parameter object not a ECParameterSpec
> --
>
> Key: NIFI-7168
> URL: https://issues.apache.org/jira/browse/NIFI-7168
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.10.0, 1.11.1
>Reporter: Raymond van den Bedum
>Priority: Minor
> Labels: fetchsftp, listsftp
> Attachments: Screenshot 2020-02-19 at 09.53.06.png
>
>
> While fetching data from a sftp site an error occurs.
>
> ListSFTP[id=0159101f-586d-11a7-03ab-a8b0637edcc7] Failed to perform listing
> on remote host due to parameter object not a ECParameterSpec:
> net.schmizz.sshj.transport.TransportException: [KEY_EXCHANGE_FAILED]
> parameter object not a ECParameterSpec
>
>
> Auto-refresh
> [KEY_EXCHANGE_FAILED], msg=[parameter object not a ECParameterSpec]
> debug output:
> {code:java}
> 2020-02-19 09:40:23,963 INFO [Timer-Driven Process Thread-4]
> n.s.s.t.random.BouncyCastleRandom Generating random seed from SecureRandom.
> 2020-02-19 09:40:23,963 DEBUG [Timer-Driven Process Thread-4]
> n.s.s.t.random.BouncyCastleRandom Creating random seed took 0 ms
> 2020-02-19 09:40:24,291 INFO [Timer-Driven Process Thread-4]
> net.schmizz.sshj.transport.TransportImpl Client identity string:
> SSH-2.0-SSHJ_0.27.0
> 2020-02-19 09:40:24,557 INFO [Timer-Driven Process Thread-4]
> net.schmizz.sshj.transport.TransportImpl Server identity string:
> SSH-2.0-Serv-U_15.1.7.238
> 2020-02-19 09:40:24,557 DEBUG [Timer-Driven Process Thread-4]
> net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_KEXINIT
> 2020-02-19 09:40:24,635 DEBUG [reader]
> net.schmizz.sshj.transport.KeyExchanger Received SSH_MSG_KEXINIT
> 2020-02-19 09:40:24,635 DEBUG [reader]
> net.schmizz.sshj.transport.KeyExchanger Negotiated algorithms: [
> kex=ecdh-sha2-nistp521; sig=ssh-dss; c2sCipher=aes256-cbc;
> s2cCipher=aes256-cbc; c2sMAC=hmac-sha2-256; s2cMAC=hmac-sha2-256;
> c2sComp=none; s2cComp=none ]
> 2020-02-19 09:40:24,666 ERROR [reader]
> net.schmizz.sshj.transport.TransportImpl Dying because - parameter object not
> a ECParameterSpec
> net.schmizz.sshj.transport.TransportException: parameter object not a
> ECParameterSpec
> at
> net.schmizz.sshj.transport.KeyExchanger.gotKexInit(KeyExchanger.java:240)
> at net.schmizz.sshj.transport.KeyExchanger.handle(KeyExchanger.java:356)
> at
> net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:503)
> at net.schmizz.sshj.transport.Decoder.decodeMte(Decoder.java:159)
> at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:79)
> at net.schmizz.sshj.transport.Decoder.received(Decoder.java:231)
> at net.schmizz.sshj.transport.Reader.run(Reader.java:59)
> Caused by: java.security.InvalidAlgorithmParameterException: parameter object
> not a ECParameterSpec
> at
> org.bouncycastle.jcajce.provider.asymmetric.ec.KeyPairGeneratorSpi$EC.initialize(Unknown
> Source)
> at java.security.KeyPairGenerator.initialize(KeyPairGenerator.java:411)
> at net.schmizz.sshj.transport.kex.ECDH.init(ECDH.java:45)
> at net.schmizz.sshj.transport.kex.ECDHNistP.initDH(ECDHNistP.java:82)
> at net.schmizz.sshj.transport.kex.AbstractDHG.init(AbstractDHG.java:46)
> at
> net.schmizz.sshj.transport.KeyExchanger.gotKexInit(KeyExchanger.java:236)
> ... 6 common frames omitted
> 2020-02-19 09:40:24,666 INFO [reader]
> net.schmizz.sshj.transport.TransportImpl Disconnected - KEY_EXCHANGE_FAILED
> 2020-02-19 09:40:24,666 DEBUG [reader]
> net.schmizz.sshj.transport.KeyExchanger Got notified of
> net.schmizz.sshj.transport.TransportException: [KEY_EXCHANGE_FAILED]
> parameter object not a ECParameterSpec
> 2020-02-19 09:40:24,666 DEBUG [reader] n.s.s.t.TransportImpl$NullService
> Notified of net.schmizz.sshj.transport.TransportException:
> [KEY_EXCHANGE_FAILED] parameter object not a ECParameterSpec
> 2020-02-19 09:40:24,666 DEBUG [reader]
> net.schmizz.sshj.transport.TransportImpl Setting active service to
> null-service
> 2020-02-19 09:40:24,666 DEBUG [reader]
> net.schmizz.sshj.transport.TransportImpl Sending SSH_MSG_DISCONNECT:
> reason=[KEY_EXCHANGE_FAILED], msg=[parameter object not a ECParameterSpec]
> 2020-02-19 09:40:24,666 DEBUG [reader] net.schmizz.sshj.transport.Reader
> Stopping
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-7746) ExecuteStreamCommand should support expression language in dynamic properties
Malthe Borch created NIFI-7746: -- Summary: ExecuteStreamCommand should support expression language in dynamic properties Key: NIFI-7746 URL: https://issues.apache.org/jira/browse/NIFI-7746 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch Dynamic properties are mapped to environment variables. There should be no negative impact in enabling expression language for them (it is exceedingly unlikely that a static value would clash with expression language syntax). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163913#comment-17163913 ] Malthe Borch commented on NIFI-2072: [~otto] Nice work. I think this is ready for the next step (not sure who/how that works). > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7666) Allow parametrizing SQL statements based on custom processor fields
Malthe Borch created NIFI-7666:
--
Summary: Allow parametrizing SQL statements based on custom
processor fields
Key: NIFI-7666
URL: https://issues.apache.org/jira/browse/NIFI-7666
Project: Apache NiFi
Issue Type: New Feature
Reporter: Malthe Borch
Today, query processors such as ExecuteSQL allow parametrized SQL statements
based on incoming flow file attributes on the form
"sql.args.N.{type;value;format}".
This is a proposal to add a secondary mode where these parameters are drawn
instead from custom properties.
When the secondary mode (i.e. "Use custom properties to set parameters") is
enabled, custom properties are used instead of flow file attributes.
* The NiFi Expression Language can be used in these properties (will be
evaluated using flow file attributes and variable registry).
* In addition to "sql.args.N.value" there will be a sensitive version
"sql.args.N.sensitive". It's an error to provide both.
That is, in addition to providing improved ergonomics (allowing the processor
to control the parameters, possibly drawing from more human-friendly attribute
names), this also enables the use of sensitive properties. As a specific
example, a database can have a "COPY INTO" statement which supports the
ingestion of data from blob storage requiring an authentication token.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152904#comment-17152904 ] Malthe Borch edited comment on NIFI-2072 at 7/7/20, 5:14 PM: - I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. I often enough see named captures mixed with unnamed ones, simply because the author has not bothered to use a non-capturing group. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). It would then be an error to use a regex that has unnamed capture groups. was (Author: malthe): I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152904#comment-17152904 ] Malthe Borch commented on NIFI-2072: I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151827#comment-17151827 ] Malthe Borch edited comment on NIFI-2072 at 7/6/20, 7:20 AM: - Is it really necessary to _enable_ named capture group rather than just use them? If I don't want a named capture group, I suppose I am just not going to name them, opting instead for enumerated ones. was (Author: malthe): Is it really necessary to _enable _named capture group rather than just use them? If I don't want a named capture group, I suppose I am just not going to name them, opting instead for enumerated ones. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151827#comment-17151827 ] Malthe Borch edited comment on NIFI-2072 at 7/6/20, 7:20 AM: - Is it really necessary to _enable _named capture group rather than just use them? If I don't want a named capture group, I suppose I am just not going to name them, opting instead for enumerated ones. was (Author: malthe): Is it really necessary to enable named capture group rather than just use them? If I don't want a named capture group, I suppose I am just not going to name them, opting instead for enumerated ones. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151827#comment-17151827 ] Malthe Borch commented on NIFI-2072: Is it really necessary to enable named capture group rather than just use them? If I don't want a named capture group, I suppose I am just not going to name them, opting instead for enumerated ones. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7597) Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile as file
Malthe Borch created NIFI-7597: -- Summary: Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile as file Key: NIFI-7597 URL: https://issues.apache.org/jira/browse/NIFI-7597 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch ExecuteStreamCommand uses a [BufferedOutputStream|https://docs.oracle.com/javase/7/docs/api/java/io/BufferedOutputStream.html] as the stream which is attached as STDIN to the spawned child process. However, some programs depend on STDIN being a file to enable certain modes of operation. For example, Microsoft Azure's [AzCopy|https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10] utility. This is an improvement proposal to add a toggle that controls whether to use a buffered output stream or a file-based output stream. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7593) Azure Blob Storage uses sensitive property for account name
Malthe Borch created NIFI-7593: -- Summary: Azure Blob Storage uses sensitive property for account name Key: NIFI-7593 URL: https://issues.apache.org/jira/browse/NIFI-7593 Project: Apache NiFi Issue Type: Improvement Reporter: Malthe Borch Processors such as [AzureListBlobStorage|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/] has account name as a sensitive property. This seems wrong and also causes some usability issues with parameter context parameters (which are either sensitive or non-sensitive). Note that while the fix for this is trivial, it could have real implications for existing flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[
https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146104#comment-17146104
]
Malthe Borch edited comment on NIFI-2072 at 6/26/20, 7:53 AM:
--
[~pvillard] did you ever make any headway with this or is it open for work,
assuming that you are still happy with the suggested behavior?
I did not know about the {{ExtractGrok}} processor before I saw this issue. In
terms of usability, I think it still does make sense to improve {{ExtractText}}
to support named capturing groups.
I bet most users will not be familiar with Grok and not immediately understand
that it might be useful to them.
was (Author: malthe):
[~pvillard] did you ever make any headway with this or is it open for work,
assuming that you are still happy with the suggested behavior?
> Support named captures in ExtractText
> -
>
> Key: NIFI-2072
> URL: https://issues.apache.org/jira/browse/NIFI-2072
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Priority: Major
>
> ExtractText currently captures and creates attributes using numeric indices
> (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture
> groups are named, i.e., patterns like (?\w+).
> In addition to being more faithful to the provided regexes, named captures
> could help simplify data flows because you wouldn't have to add superfluous
> UpdateAttribute steps which are just renaming the indexed captures to more
> interpretable names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146104#comment-17146104 ] Malthe Borch commented on NIFI-2072: [~pvillard] did you ever make any headway with this or is it open for work, assuming that you are still happy with the suggested behavior? > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Priority: Major > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6508) Test failures caused by timezone differences
[
https://issues.apache.org/jira/browse/NIFI-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Malthe Borch updated NIFI-6508:
---
Attachment: 0001-NIFI-6508-Fix-test-failure-caused-by-timezone-and-or.patch
> Test failures caused by timezone differences
>
>
> Key: NIFI-6508
> URL: https://issues.apache.org/jira/browse/NIFI-6508
> Project: Apache NiFi
> Issue Type: Test
> Components: Core Framework
>Reporter: Malthe Borch
>Priority: Minor
> Attachments:
> 0001-NIFI-6508-Fix-test-failure-caused-by-timezone-and-or.patch
>
>
> Some tests in {{VolatileComponentStatusRepositoryTest}} fail due to incorrect
> timezone conversion, for example {{testFilterDatesUsingStartFilter}}:
> {{java.lang.AssertionError: expected: but
> was: at
> org.apache.nifi.controller.status.history.VolatileComponentStatusRepositoryTest.testFilterDatesUsingStartFilter(VolatileComponentStatusRepositoryTest.java:132)}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (NIFI-6508) Test failures caused by timezone differences
Malthe Borch created NIFI-6508:
--
Summary: Test failures caused by timezone differences
Key: NIFI-6508
URL: https://issues.apache.org/jira/browse/NIFI-6508
Project: Apache NiFi
Issue Type: Test
Components: Core Framework
Reporter: Malthe Borch
Some tests in {{VolatileComponentStatusRepositoryTest}} fail due to incorrect
timezone conversion, for example {{testFilterDatesUsingStartFilter}}:
{{java.lang.AssertionError: expected: but
was: at
org.apache.nifi.controller.status.history.VolatileComponentStatusRepositoryTest.testFilterDatesUsingStartFilter(VolatileComponentStatusRepositoryTest.java:132)}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (NIFI-6496) Add compression support to record reader processor
[
https://issues.apache.org/jira/browse/NIFI-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896179#comment-16896179
]
Malthe Borch edited comment on NIFI-6496 at 7/30/19 2:25 PM:
-
Adding compression-support to {{FlowFile}} itself seems ideal, but ideally
exposed only through interfaces such as {{CompressContent}}.
In this model, {{CompressContent}} would have an optional streaming (or "lazy")
mode such that the unpacked file contents would not have to be written to disk.
The effect of running the processor would effectively be to set an internal
flag that enables transparent decompression in a subsequent step. The
{{fileSize}} should not need to be updated because effectively, the size has
not changed (this should be mostly of interest in the context of provenance).
If in some cases content is not streamed (but loaded entirely into memory) then
I would think that it an issue that can fixed separately?
was (Author: malthe):
Adding compression-support to {{FlowFile}} itself seems ideal, but ideally
exposed only through interfaces such as `CompressContent`.
In this model, {{CompressContent}} would have an optional streaming (or "lazy")
mode such that the unpacked file contents would not have to be written to disk.
The effect of running the processor would effectively be to set an internal
flag that enables transparent decompression in a subsequent step. The
{{fileSize}} should not need to be updated because effectively, the size has
not changed (this should be mostly of interest in the context of provenance).
If in some cases content is not streamed (but loaded entirely into memory) then
I would think that it an issue that can fixed separately?
> Add compression support to record reader processor
> --
>
> Key: NIFI-6496
> URL: https://issues.apache.org/jira/browse/NIFI-6496
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Malthe Borch
>Priority: Minor
> Labels: easyfix, usability
>
> Text-based record formats such as CSV, JSON and XML compress well and will
> often be transmitted in a compressed format. If compression support is added
> to the relevant processors, users will not need to explicitly unpack files
> before processing (which may not be feasible or practical due to space
> requirements).
> There are at least two ways of implementing this, using either a generic
> approach where a {{CompressedRecordReaderFactory}} is the basis for a new
> controller service that wraps the underlying record reader controller service
> (e.g. {{CSVReader}}); or adding the functionality at the relevant record
> reader implementations.
> The latter option may provide a better UX because no additional
> {{ControllerService}} has to be configured.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (NIFI-6496) Add compression support to record reader processor
[
https://issues.apache.org/jira/browse/NIFI-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896179#comment-16896179
]
Malthe Borch commented on NIFI-6496:
Adding compression-support to {{FlowFile}} itself seems ideal, but ideally
exposed only through interfaces such as `CompressContent`.
In this model, {{CompressContent}} would have an optional streaming (or "lazy")
mode such that the unpacked file contents would not have to be written to disk.
The effect of running the processor would effectively be to set an internal
flag that enables transparent decompression in a subsequent step. The
{{fileSize}} should not need to be updated because effectively, the size has
not changed (this should be mostly of interest in the context of provenance).
If in some cases content is not streamed (but loaded entirely into memory) then
I would think that it an issue that can fixed separately?
> Add compression support to record reader processor
> --
>
> Key: NIFI-6496
> URL: https://issues.apache.org/jira/browse/NIFI-6496
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Malthe Borch
>Priority: Minor
> Labels: easyfix, usability
>
> Text-based record formats such as CSV, JSON and XML compress well and will
> often be transmitted in a compressed format. If compression support is added
> to the relevant processors, users will not need to explicitly unpack files
> before processing (which may not be feasible or practical due to space
> requirements).
> There are at least two ways of implementing this, using either a generic
> approach where a {{CompressedRecordReaderFactory}} is the basis for a new
> controller service that wraps the underlying record reader controller service
> (e.g. {{CSVReader}}); or adding the functionality at the relevant record
> reader implementations.
> The latter option may provide a better UX because no additional
> {{ControllerService}} has to be configured.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (NIFI-6496) Add compression support to record reader processor
Malthe Borch created NIFI-6496:
--
Summary: Add compression support to record reader processor
Key: NIFI-6496
URL: https://issues.apache.org/jira/browse/NIFI-6496
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Malthe Borch
Text-based record formats such as CSV, JSON and XML compress well and will
often be transmitted in a compressed format. If compression support is added to
the relevant processors, users will not need to explicitly unpack files before
processing (which may not be feasible or practical due to space requirements).
There are at least two ways of implementing this, using either a generic
approach where a {{CompressedRecordReaderFactory}} is the basis for a new
controller service that wraps the underlying record reader controller service
(e.g. {{CSVReader}}); or adding the functionality at the relevant record reader
implementations.
The latter option may provide a better UX because no additional
{{ControllerService}} has to be configured.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
