Re: alternate web root directory?
Cool, thanks. I wasn't aware of that project. I'll check it out. On Fri, Mar 20, 2020 at 1:21 PM Chris Sampson wrote: > Keycloak Gatekeeper - https://github.com/keycloak/keycloak-gatekeeper > > Then nginx-proxy to set the required X-ProxiedEntity header. > > > *Chris Sampson* > IT Consultant > *Tel:* 07867 843 675 > chris.samp...@naimuri.com > > > > On Fri, 20 Mar 2020 at 17:15, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> >> How are you doing OpenID authentication to nifi-registry? We are using >> Keycloak OpenID auth to nifi itself, but couldn't configure it for >> nifi-registry. Everything I've read says its not currently possible. >> >> >> https://community.cloudera.com/t5/Support-Questions/Nifi-Registry-openid/m-p/290935 >> https://issues.apache.org/jira/browse/NIFIREG-313 >> >> >> >> On Fri, Mar 20, 2020 at 12:09 PM Chris Sampson >> wrote: >> >>> We've got a separate NiFi and NiFi Registry ingresses and OIDC for both >>> (different secure cookies used to differentiate the logins in the user's >>> browser). Users are able to login to both within the same browser session >>> without issue. >>> >>> OIDC AuthN in Registry is achieved using keycloak-gatekeeper in front of >>> the Registry instance with an nginx-proxy in between the two to add the >>> required X-ProxiedEntity header for Registry to be able to identify the >>> user and then apply AuthZ. >>> >>> Connection from NiFi to NiFi Registry doesn't go via the ingress, rather >>> they're direct between "internal" services within the k8s namespace. >>> AuthN/Z for the NiFi instances to Registry is via PKI certificates issued >>> (currently) by NiFi Toolkit (may look to use k8s cert-manager in future >>> instead). >>> >>> You're right that it's frustrating the two products don't use the same >>> AuthN/Z connectors, maybe there's a ticket on the Registry backlog for such >>> an update. The NiFi OIDC connector also doesn't use Refresh Token, which is >>> unfortunate as it means the lifetime of the ID/Account Token has to be >>> extended to prevent the user continually being logged out of the UI (which >>> is less secure than having regular refreshes) - think I've seen something >>> on the NiFi backlog to address that, but no work on it last time I checked. >>> >>> >>> *Chris Sampson* >>> IT Consultant >>> *Tel:* 07867 843 675 >>> chris.samp...@naimuri.com >>> >>> >>> >>> On Fri, 20 Mar 2020 at 15:33, Wyllys Ingersoll < >>> wyllys.ingers...@keepertech.com> wrote: >>> Yup, I came up with a similar configuration for my NiFi cluster - fully secured using OIDC directly from NIFI. The issue I had with the nifi-registry was that it had to use different authentication (Kerberos vs OIDC) (aside: why aren't the 2 packages sharing the same authentication modules and configuration models??) but because they were both behind the same k8s ingress hostname, whenever one would authenticate (nifi vs nifi-registry) the browser would overwrite the authorization header so we could only be logged into one at a time. The solution was to add a DNS alias so we could still share the same k8s ingress but using different hostnames so that the browser would keep the authorization headers separate and we could stay authenticated to both within the same browser. Wyllys Ingersoll On Fri, Mar 20, 2020 at 10:15 AM Chris Sampson < chris.samp...@naimuri.com> wrote: > We've found a clustered k8s NiFi with OIDC auth works OK once: > >- Session Affinity annotations added to the nginx Ingress (change >cookie name appropriately): > > ingress.kubernetes.io/affinity: cookie >> ingress.kubernetes.io/session-cookie-name: "my-nifi" >> > >- Ingress Service (used by the Ingress to talk to NiFi) is >configured with: > > sessionAffinity: ClientIP >> > > We originally had a separate nginx reverse proxy (configured to the > NiFi instance root "/"), but removed that once we fully configured > security > in the NiFi cluster instances directly and end-to-end through our > ingress/service chain (i.e. all https, OIDC done directly by NiFi). > > We do still have a reverse proxy in front of our NiFi Registry > instance though so it can do OIDC auth for us (as Registry doesn't have > that capability baked in) - this, again, is configured to go directly to > the Registry webroot "/". Bit different for Registry of course as it's not > as complex as NiFi and isn't clustered. > > > *Chris Sampson* > IT Consultant > *Tel:* 07867 843 675 > chris.samp...@naimuri.com > > > > On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> >> Yes, it would be easier to configure behind a single root context >> path, but since nifi uses hardcoded root paths in the responses, it makes >>
Re: alternate web root directory?
Keycloak Gatekeeper - https://github.com/keycloak/keycloak-gatekeeper Then nginx-proxy to set the required X-ProxiedEntity header. *Chris Sampson* IT Consultant *Tel:* 07867 843 675 chris.samp...@naimuri.com On Fri, 20 Mar 2020 at 17:15, Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > How are you doing OpenID authentication to nifi-registry? We are using > Keycloak OpenID auth to nifi itself, but couldn't configure it for > nifi-registry. Everything I've read says its not currently possible. > > > https://community.cloudera.com/t5/Support-Questions/Nifi-Registry-openid/m-p/290935 > https://issues.apache.org/jira/browse/NIFIREG-313 > > > > On Fri, Mar 20, 2020 at 12:09 PM Chris Sampson > wrote: > >> We've got a separate NiFi and NiFi Registry ingresses and OIDC for both >> (different secure cookies used to differentiate the logins in the user's >> browser). Users are able to login to both within the same browser session >> without issue. >> >> OIDC AuthN in Registry is achieved using keycloak-gatekeeper in front of >> the Registry instance with an nginx-proxy in between the two to add the >> required X-ProxiedEntity header for Registry to be able to identify the >> user and then apply AuthZ. >> >> Connection from NiFi to NiFi Registry doesn't go via the ingress, rather >> they're direct between "internal" services within the k8s namespace. >> AuthN/Z for the NiFi instances to Registry is via PKI certificates issued >> (currently) by NiFi Toolkit (may look to use k8s cert-manager in future >> instead). >> >> You're right that it's frustrating the two products don't use the same >> AuthN/Z connectors, maybe there's a ticket on the Registry backlog for such >> an update. The NiFi OIDC connector also doesn't use Refresh Token, which is >> unfortunate as it means the lifetime of the ID/Account Token has to be >> extended to prevent the user continually being logged out of the UI (which >> is less secure than having regular refreshes) - think I've seen something >> on the NiFi backlog to address that, but no work on it last time I checked. >> >> >> *Chris Sampson* >> IT Consultant >> *Tel:* 07867 843 675 >> chris.samp...@naimuri.com >> >> >> >> On Fri, 20 Mar 2020 at 15:33, Wyllys Ingersoll < >> wyllys.ingers...@keepertech.com> wrote: >> >>> >>> Yup, I came up with a similar configuration for my NiFi cluster - fully >>> secured using OIDC directly from NIFI. The issue I had with the >>> nifi-registry was that it had to use different authentication (Kerberos vs >>> OIDC) (aside: why aren't the 2 packages sharing the same authentication >>> modules and configuration models??) but because they were both behind the >>> same k8s ingress hostname, whenever one would authenticate (nifi vs >>> nifi-registry) the browser would overwrite the authorization header so we >>> could only be logged into one at a time. The solution was to add a DNS >>> alias so we could still share the same k8s ingress but using different >>> hostnames so that the browser would keep the authorization headers separate >>> and we could stay authenticated to both within the same browser. >>> >>> Wyllys Ingersoll >>> >>> On Fri, Mar 20, 2020 at 10:15 AM Chris Sampson < >>> chris.samp...@naimuri.com> wrote: >>> We've found a clustered k8s NiFi with OIDC auth works OK once: - Session Affinity annotations added to the nginx Ingress (change cookie name appropriately): ingress.kubernetes.io/affinity: cookie > ingress.kubernetes.io/session-cookie-name: "my-nifi" > - Ingress Service (used by the Ingress to talk to NiFi) is configured with: sessionAffinity: ClientIP > We originally had a separate nginx reverse proxy (configured to the NiFi instance root "/"), but removed that once we fully configured security in the NiFi cluster instances directly and end-to-end through our ingress/service chain (i.e. all https, OIDC done directly by NiFi). We do still have a reverse proxy in front of our NiFi Registry instance though so it can do OIDC auth for us (as Registry doesn't have that capability baked in) - this, again, is configured to go directly to the Registry webroot "/". Bit different for Registry of course as it's not as complex as NiFi and isn't clustered. *Chris Sampson* IT Consultant *Tel:* 07867 843 675 chris.samp...@naimuri.com On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > Yes, it would be easier to configure behind a single root context > path, but since nifi uses hardcoded root paths in the responses, it makes > things very challenging. I managed to get things working *mostly* with > the > existing paths and some regular expression based path mapping in the k8s > ingress (i.e. nginx proxy), but I maintain that things would be MUCH > easier
Re: alternate web root directory?
How are you doing OpenID authentication to nifi-registry? We are using Keycloak OpenID auth to nifi itself, but couldn't configure it for nifi-registry. Everything I've read says its not currently possible. https://community.cloudera.com/t5/Support-Questions/Nifi-Registry-openid/m-p/290935 https://issues.apache.org/jira/browse/NIFIREG-313 On Fri, Mar 20, 2020 at 12:09 PM Chris Sampson wrote: > We've got a separate NiFi and NiFi Registry ingresses and OIDC for both > (different secure cookies used to differentiate the logins in the user's > browser). Users are able to login to both within the same browser session > without issue. > > OIDC AuthN in Registry is achieved using keycloak-gatekeeper in front of > the Registry instance with an nginx-proxy in between the two to add the > required X-ProxiedEntity header for Registry to be able to identify the > user and then apply AuthZ. > > Connection from NiFi to NiFi Registry doesn't go via the ingress, rather > they're direct between "internal" services within the k8s namespace. > AuthN/Z for the NiFi instances to Registry is via PKI certificates issued > (currently) by NiFi Toolkit (may look to use k8s cert-manager in future > instead). > > You're right that it's frustrating the two products don't use the same > AuthN/Z connectors, maybe there's a ticket on the Registry backlog for such > an update. The NiFi OIDC connector also doesn't use Refresh Token, which is > unfortunate as it means the lifetime of the ID/Account Token has to be > extended to prevent the user continually being logged out of the UI (which > is less secure than having regular refreshes) - think I've seen something > on the NiFi backlog to address that, but no work on it last time I checked. > > > *Chris Sampson* > IT Consultant > *Tel:* 07867 843 675 > chris.samp...@naimuri.com > > > > On Fri, 20 Mar 2020 at 15:33, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> >> Yup, I came up with a similar configuration for my NiFi cluster - fully >> secured using OIDC directly from NIFI. The issue I had with the >> nifi-registry was that it had to use different authentication (Kerberos vs >> OIDC) (aside: why aren't the 2 packages sharing the same authentication >> modules and configuration models??) but because they were both behind the >> same k8s ingress hostname, whenever one would authenticate (nifi vs >> nifi-registry) the browser would overwrite the authorization header so we >> could only be logged into one at a time. The solution was to add a DNS >> alias so we could still share the same k8s ingress but using different >> hostnames so that the browser would keep the authorization headers separate >> and we could stay authenticated to both within the same browser. >> >> Wyllys Ingersoll >> >> On Fri, Mar 20, 2020 at 10:15 AM Chris Sampson >> wrote: >> >>> We've found a clustered k8s NiFi with OIDC auth works OK once: >>> >>>- Session Affinity annotations added to the nginx Ingress (change >>>cookie name appropriately): >>> >>> ingress.kubernetes.io/affinity: cookie ingress.kubernetes.io/session-cookie-name: "my-nifi" >>> >>>- Ingress Service (used by the Ingress to talk to NiFi) is >>>configured with: >>> >>> sessionAffinity: ClientIP >>> >>> We originally had a separate nginx reverse proxy (configured to the NiFi >>> instance root "/"), but removed that once we fully configured security in >>> the NiFi cluster instances directly and end-to-end through our >>> ingress/service chain (i.e. all https, OIDC done directly by NiFi). >>> >>> We do still have a reverse proxy in front of our NiFi Registry instance >>> though so it can do OIDC auth for us (as Registry doesn't have that >>> capability baked in) - this, again, is configured to go directly to the >>> Registry webroot "/". Bit different for Registry of course as it's not as >>> complex as NiFi and isn't clustered. >>> >>> >>> *Chris Sampson* >>> IT Consultant >>> *Tel:* 07867 843 675 >>> chris.samp...@naimuri.com >>> >>> >>> >>> On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < >>> wyllys.ingers...@keepertech.com> wrote: >>> Yes, it would be easier to configure behind a single root context path, but since nifi uses hardcoded root paths in the responses, it makes things very challenging. I managed to get things working *mostly* with the existing paths and some regular expression based path mapping in the k8s ingress (i.e. nginx proxy), but I maintain that things would be MUCH easier if NiFi wasn't so dependent on hardcoded path names and could unify things behind a single root path. As I said before, if authentication is not involved, it's not a problem, but once authentication and tokens are involved, it becomes more challenging. Cookie and session affinity settings in the proxy are helpful but in the most common k8s cluster configuration, the cluster of NiFi nodes (i.e. stateful set of pods) is behind
Re: alternate web root directory?
We've got a separate NiFi and NiFi Registry ingresses and OIDC for both (different secure cookies used to differentiate the logins in the user's browser). Users are able to login to both within the same browser session without issue. OIDC AuthN in Registry is achieved using keycloak-gatekeeper in front of the Registry instance with an nginx-proxy in between the two to add the required X-ProxiedEntity header for Registry to be able to identify the user and then apply AuthZ. Connection from NiFi to NiFi Registry doesn't go via the ingress, rather they're direct between "internal" services within the k8s namespace. AuthN/Z for the NiFi instances to Registry is via PKI certificates issued (currently) by NiFi Toolkit (may look to use k8s cert-manager in future instead). You're right that it's frustrating the two products don't use the same AuthN/Z connectors, maybe there's a ticket on the Registry backlog for such an update. The NiFi OIDC connector also doesn't use Refresh Token, which is unfortunate as it means the lifetime of the ID/Account Token has to be extended to prevent the user continually being logged out of the UI (which is less secure than having regular refreshes) - think I've seen something on the NiFi backlog to address that, but no work on it last time I checked. *Chris Sampson* IT Consultant *Tel:* 07867 843 675 chris.samp...@naimuri.com On Fri, 20 Mar 2020 at 15:33, Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > Yup, I came up with a similar configuration for my NiFi cluster - fully > secured using OIDC directly from NIFI. The issue I had with the > nifi-registry was that it had to use different authentication (Kerberos vs > OIDC) (aside: why aren't the 2 packages sharing the same authentication > modules and configuration models??) but because they were both behind the > same k8s ingress hostname, whenever one would authenticate (nifi vs > nifi-registry) the browser would overwrite the authorization header so we > could only be logged into one at a time. The solution was to add a DNS > alias so we could still share the same k8s ingress but using different > hostnames so that the browser would keep the authorization headers separate > and we could stay authenticated to both within the same browser. > > Wyllys Ingersoll > > On Fri, Mar 20, 2020 at 10:15 AM Chris Sampson > wrote: > >> We've found a clustered k8s NiFi with OIDC auth works OK once: >> >>- Session Affinity annotations added to the nginx Ingress (change >>cookie name appropriately): >> >> ingress.kubernetes.io/affinity: cookie >>> ingress.kubernetes.io/session-cookie-name: "my-nifi" >>> >> >>- Ingress Service (used by the Ingress to talk to NiFi) is configured >>with: >> >> sessionAffinity: ClientIP >>> >> >> We originally had a separate nginx reverse proxy (configured to the NiFi >> instance root "/"), but removed that once we fully configured security in >> the NiFi cluster instances directly and end-to-end through our >> ingress/service chain (i.e. all https, OIDC done directly by NiFi). >> >> We do still have a reverse proxy in front of our NiFi Registry instance >> though so it can do OIDC auth for us (as Registry doesn't have that >> capability baked in) - this, again, is configured to go directly to the >> Registry webroot "/". Bit different for Registry of course as it's not as >> complex as NiFi and isn't clustered. >> >> >> *Chris Sampson* >> IT Consultant >> *Tel:* 07867 843 675 >> chris.samp...@naimuri.com >> >> >> >> On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < >> wyllys.ingers...@keepertech.com> wrote: >> >>> >>> Yes, it would be easier to configure behind a single root context path, >>> but since nifi uses hardcoded root paths in the responses, it makes things >>> very challenging. I managed to get things working *mostly* with the >>> existing paths and some regular expression based path mapping in the k8s >>> ingress (i.e. nginx proxy), but I maintain that things would be MUCH easier >>> if NiFi wasn't so dependent on hardcoded path names and could unify things >>> behind a single root path. >>> >>> As I said before, if authentication is not involved, it's not a problem, >>> but once authentication and tokens are involved, it becomes more >>> challenging. Cookie and session affinity settings in the proxy are helpful >>> but in the most common k8s cluster configuration, the cluster of NiFi nodes >>> (i.e. stateful set of pods) is behind a single service name so the >>> proxy/ingress in front of everything doesn't actually know which one it's >>> talking to which make affinity harder to establish and maintain. >>> >>> >>> On Fri, Mar 20, 2020 at 9:31 AM Matt Gilman >>> wrote: >>> Wyllys, The guidance for the proxy configuration is to map it behind a single path. In order to work with any number of context paths that NiFi supports the proxying needs to be based on the root path. Let me know if I'm misunderstanding something. Unfort
Re: alternate web root directory?
Yup, I came up with a similar configuration for my NiFi cluster - fully secured using OIDC directly from NIFI. The issue I had with the nifi-registry was that it had to use different authentication (Kerberos vs OIDC) (aside: why aren't the 2 packages sharing the same authentication modules and configuration models??) but because they were both behind the same k8s ingress hostname, whenever one would authenticate (nifi vs nifi-registry) the browser would overwrite the authorization header so we could only be logged into one at a time. The solution was to add a DNS alias so we could still share the same k8s ingress but using different hostnames so that the browser would keep the authorization headers separate and we could stay authenticated to both within the same browser. Wyllys Ingersoll On Fri, Mar 20, 2020 at 10:15 AM Chris Sampson wrote: > We've found a clustered k8s NiFi with OIDC auth works OK once: > >- Session Affinity annotations added to the nginx Ingress (change >cookie name appropriately): > > ingress.kubernetes.io/affinity: cookie >> ingress.kubernetes.io/session-cookie-name: "my-nifi" >> > >- Ingress Service (used by the Ingress to talk to NiFi) is configured >with: > > sessionAffinity: ClientIP >> > > We originally had a separate nginx reverse proxy (configured to the NiFi > instance root "/"), but removed that once we fully configured security in > the NiFi cluster instances directly and end-to-end through our > ingress/service chain (i.e. all https, OIDC done directly by NiFi). > > We do still have a reverse proxy in front of our NiFi Registry instance > though so it can do OIDC auth for us (as Registry doesn't have that > capability baked in) - this, again, is configured to go directly to the > Registry webroot "/". Bit different for Registry of course as it's not as > complex as NiFi and isn't clustered. > > > *Chris Sampson* > IT Consultant > *Tel:* 07867 843 675 > chris.samp...@naimuri.com > > > > On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> >> Yes, it would be easier to configure behind a single root context path, >> but since nifi uses hardcoded root paths in the responses, it makes things >> very challenging. I managed to get things working *mostly* with the >> existing paths and some regular expression based path mapping in the k8s >> ingress (i.e. nginx proxy), but I maintain that things would be MUCH easier >> if NiFi wasn't so dependent on hardcoded path names and could unify things >> behind a single root path. >> >> As I said before, if authentication is not involved, it's not a problem, >> but once authentication and tokens are involved, it becomes more >> challenging. Cookie and session affinity settings in the proxy are helpful >> but in the most common k8s cluster configuration, the cluster of NiFi nodes >> (i.e. stateful set of pods) is behind a single service name so the >> proxy/ingress in front of everything doesn't actually know which one it's >> talking to which make affinity harder to establish and maintain. >> >> >> On Fri, Mar 20, 2020 at 9:31 AM Matt Gilman >> wrote: >> >>> Wyllys, >>> >>> The guidance for the proxy configuration is to map it behind a >>> single path. In order to work with any number of context paths that NiFi >>> supports the proxying needs to be based on the root path. Let me know if >>> I'm misunderstanding something. Unfortunately, I'm not super familiar with >>> nginx ingress in k8s environments but it sounds like that session affinity >>> may not working right. >>> >>> On Wed, Mar 18, 2020 at 5:37 PM Wyllys Ingersoll < >>> wyllys.ingers...@keepertech.com> wrote: >>> Yeah, I've read all of that and I have a semi-working configuration. The problem is that when using OpenID tokens (oidc) in a clustered configuration, the node that requests the authentication is the only one that can validate it. If a user authenticates to node-1, but then later node-2 gets a request with the token (because its clustered and the user has no control over which node handles the request), node-2 cannot verify the token and rejects it. Even configuring sticky-sessions and cookie affinity in the nginx ingress proxy don't solve the problem as far as I can tell. I don't even know if having it all behind a single root path would make a difference for the authentication issue, it just makes setting up the reverse proxy configuration easier since you only need to worry about 1 path instead of multiple. On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman wrote: > Wyllys, > > NiFi is comprised of any number of web applications. NiFi offers > extension points for custom processor configuration UIs and data type > viewers. These UIs can be bundled and discovered at runtime. These docs > [1] > detail the steps necessary for proxying a NiFi instance. > > Thanks >>
Re: alternate web root directory?
We've found a clustered k8s NiFi with OIDC auth works OK once: - Session Affinity annotations added to the nginx Ingress (change cookie name appropriately): ingress.kubernetes.io/affinity: cookie > ingress.kubernetes.io/session-cookie-name: "my-nifi" > - Ingress Service (used by the Ingress to talk to NiFi) is configured with: sessionAffinity: ClientIP > We originally had a separate nginx reverse proxy (configured to the NiFi instance root "/"), but removed that once we fully configured security in the NiFi cluster instances directly and end-to-end through our ingress/service chain (i.e. all https, OIDC done directly by NiFi). We do still have a reverse proxy in front of our NiFi Registry instance though so it can do OIDC auth for us (as Registry doesn't have that capability baked in) - this, again, is configured to go directly to the Registry webroot "/". Bit different for Registry of course as it's not as complex as NiFi and isn't clustered. *Chris Sampson* IT Consultant *Tel:* 07867 843 675 chris.samp...@naimuri.com On Fri, 20 Mar 2020 at 13:44, Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > Yes, it would be easier to configure behind a single root context path, > but since nifi uses hardcoded root paths in the responses, it makes things > very challenging. I managed to get things working *mostly* with the > existing paths and some regular expression based path mapping in the k8s > ingress (i.e. nginx proxy), but I maintain that things would be MUCH easier > if NiFi wasn't so dependent on hardcoded path names and could unify things > behind a single root path. > > As I said before, if authentication is not involved, it's not a problem, > but once authentication and tokens are involved, it becomes more > challenging. Cookie and session affinity settings in the proxy are helpful > but in the most common k8s cluster configuration, the cluster of NiFi nodes > (i.e. stateful set of pods) is behind a single service name so the > proxy/ingress in front of everything doesn't actually know which one it's > talking to which make affinity harder to establish and maintain. > > > On Fri, Mar 20, 2020 at 9:31 AM Matt Gilman > wrote: > >> Wyllys, >> >> The guidance for the proxy configuration is to map it behind a >> single path. In order to work with any number of context paths that NiFi >> supports the proxying needs to be based on the root path. Let me know if >> I'm misunderstanding something. Unfortunately, I'm not super familiar with >> nginx ingress in k8s environments but it sounds like that session affinity >> may not working right. >> >> On Wed, Mar 18, 2020 at 5:37 PM Wyllys Ingersoll < >> wyllys.ingers...@keepertech.com> wrote: >> >>> Yeah, I've read all of that and I have a semi-working configuration. >>> The problem is that when using OpenID tokens (oidc) in a clustered >>> configuration, the node that requests the authentication is the only one >>> that can validate it. If a user authenticates to node-1, but then later >>> node-2 gets a request with the token (because its clustered and the user >>> has no control over which node handles the request), node-2 cannot verify >>> the token and rejects it. Even configuring sticky-sessions and cookie >>> affinity in the nginx ingress proxy don't solve the problem as far as I can >>> tell. >>> >>> I don't even know if having it all behind a single root path would make >>> a difference for the authentication issue, it just makes setting up the >>> reverse proxy configuration easier since you only need to worry about 1 >>> path instead of multiple. >>> >>> >>> >>> On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman >>> wrote: >>> Wyllys, NiFi is comprised of any number of web applications. NiFi offers extension points for custom processor configuration UIs and data type viewers. These UIs can be bundled and discovered at runtime. These docs [1] detail the steps necessary for proxying a NiFi instance. Thanks Matt [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > I'm surprised you haven't had lots of requests for this already. As > it stands now, I cannot figure out how to configure a secure cluster > behind > a reverse proxy (for example, in a kubernetes environment behind an nginx > ingress) that also incorporates OpenID authentication from an external > service. I was thinking that if the NiFi nodes were able to operate under > a > single root path, it might make it easier to reverse proxy all of the > different paths that Nifi uses (/nifi, /nifi-api, for example) behind a > single ingress. I think having multiple ingress paths for the nifi > service > makes the reverse proxy configuration very complex when authentication > tokens are
Re: alternate web root directory?
Yes, it would be easier to configure behind a single root context path, but since nifi uses hardcoded root paths in the responses, it makes things very challenging. I managed to get things working *mostly* with the existing paths and some regular expression based path mapping in the k8s ingress (i.e. nginx proxy), but I maintain that things would be MUCH easier if NiFi wasn't so dependent on hardcoded path names and could unify things behind a single root path. As I said before, if authentication is not involved, it's not a problem, but once authentication and tokens are involved, it becomes more challenging. Cookie and session affinity settings in the proxy are helpful but in the most common k8s cluster configuration, the cluster of NiFi nodes (i.e. stateful set of pods) is behind a single service name so the proxy/ingress in front of everything doesn't actually know which one it's talking to which make affinity harder to establish and maintain. On Fri, Mar 20, 2020 at 9:31 AM Matt Gilman wrote: > Wyllys, > > The guidance for the proxy configuration is to map it behind a > single path. In order to work with any number of context paths that NiFi > supports the proxying needs to be based on the root path. Let me know if > I'm misunderstanding something. Unfortunately, I'm not super familiar with > nginx ingress in k8s environments but it sounds like that session affinity > may not working right. > > On Wed, Mar 18, 2020 at 5:37 PM Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> Yeah, I've read all of that and I have a semi-working configuration. The >> problem is that when using OpenID tokens (oidc) in a clustered >> configuration, the node that requests the authentication is the only one >> that can validate it. If a user authenticates to node-1, but then later >> node-2 gets a request with the token (because its clustered and the user >> has no control over which node handles the request), node-2 cannot verify >> the token and rejects it. Even configuring sticky-sessions and cookie >> affinity in the nginx ingress proxy don't solve the problem as far as I can >> tell. >> >> I don't even know if having it all behind a single root path would make a >> difference for the authentication issue, it just makes setting up the >> reverse proxy configuration easier since you only need to worry about 1 >> path instead of multiple. >> >> >> >> On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman >> wrote: >> >>> Wyllys, >>> >>> NiFi is comprised of any number of web applications. NiFi offers >>> extension points for custom processor configuration UIs and data type >>> viewers. These UIs can be bundled and discovered at runtime. These docs [1] >>> detail the steps necessary for proxying a NiFi instance. >>> >>> Thanks >>> >>> Matt >>> >>> [1] >>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration >>> >>> On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll < >>> wyllys.ingers...@keepertech.com> wrote: >>> I'm surprised you haven't had lots of requests for this already. As it stands now, I cannot figure out how to configure a secure cluster behind a reverse proxy (for example, in a kubernetes environment behind an nginx ingress) that also incorporates OpenID authentication from an external service. I was thinking that if the NiFi nodes were able to operate under a single root path, it might make it easier to reverse proxy all of the different paths that Nifi uses (/nifi, /nifi-api, for example) behind a single ingress. I think having multiple ingress paths for the nifi service makes the reverse proxy configuration very complex when authentication tokens are involved. Without authentication, it works fine. Thanks, Wyllys Ingersoll On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto wrote: > Hi Wyllys, > > As I started reading, I was going to suggest the proxy approach. > Unfortunately, at this time I am unaware of any way to change the paths > within NiFi itself - there would be substantial refactoring required to > make that an option. You can open a feature request Jira for that, or > perhaps the ability to inject a path prefix, but I expect it to be a high > level of effort to implement. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > > > Is there a way to configure nifi to use a different root directory for > web requests? > > We would like everything to be under a common root such as: > /XXX/nifi/... > /XXX/nifi-api/... > > Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult > to setup a reverse proxy that also can incorporate OpenID authenti
Re: alternate web root directory?
Wyllys, The guidance for the proxy configuration is to map it behind a single path. In order to work with any number of context paths that NiFi supports the proxying needs to be based on the root path. Let me know if I'm misunderstanding something. Unfortunately, I'm not super familiar with nginx ingress in k8s environments but it sounds like that session affinity may not working right. On Wed, Mar 18, 2020 at 5:37 PM Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > Yeah, I've read all of that and I have a semi-working configuration. The > problem is that when using OpenID tokens (oidc) in a clustered > configuration, the node that requests the authentication is the only one > that can validate it. If a user authenticates to node-1, but then later > node-2 gets a request with the token (because its clustered and the user > has no control over which node handles the request), node-2 cannot verify > the token and rejects it. Even configuring sticky-sessions and cookie > affinity in the nginx ingress proxy don't solve the problem as far as I can > tell. > > I don't even know if having it all behind a single root path would make a > difference for the authentication issue, it just makes setting up the > reverse proxy configuration easier since you only need to worry about 1 > path instead of multiple. > > > > On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman > wrote: > >> Wyllys, >> >> NiFi is comprised of any number of web applications. NiFi offers >> extension points for custom processor configuration UIs and data type >> viewers. These UIs can be bundled and discovered at runtime. These docs [1] >> detail the steps necessary for proxying a NiFi instance. >> >> Thanks >> >> Matt >> >> [1] >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration >> >> On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll < >> wyllys.ingers...@keepertech.com> wrote: >> >>> >>> I'm surprised you haven't had lots of requests for this already. As it >>> stands now, I cannot figure out how to configure a secure cluster behind a >>> reverse proxy (for example, in a kubernetes environment behind an nginx >>> ingress) that also incorporates OpenID authentication from an external >>> service. I was thinking that if the NiFi nodes were able to operate under a >>> single root path, it might make it easier to reverse proxy all of the >>> different paths that Nifi uses (/nifi, /nifi-api, for example) behind a >>> single ingress. I think having multiple ingress paths for the nifi service >>> makes the reverse proxy configuration very complex when authentication >>> tokens are involved. Without authentication, it works fine. >>> >>> Thanks, >>> Wyllys Ingersoll >>> >>> On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto >>> wrote: >>> Hi Wyllys, As I started reading, I was going to suggest the proxy approach. Unfortunately, at this time I am unaware of any way to change the paths within NiFi itself - there would be substantial refactoring required to make that an option. You can open a feature request Jira for that, or perhaps the ability to inject a path prefix, but I expect it to be a high level of effort to implement. Andy LoPresto alopre...@apache.org *alopresto.apa...@gmail.com * He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: Is there a way to configure nifi to use a different root directory for web requests? We would like everything to be under a common root such as: /XXX/nifi/... /XXX/nifi-api/... Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to setup a reverse proxy that also can incorporate OpenID authentication tokens to a secure backend cluster of nodes.
Re: alternate web root directory?
Yeah, I've read all of that and I have a semi-working configuration. The problem is that when using OpenID tokens (oidc) in a clustered configuration, the node that requests the authentication is the only one that can validate it. If a user authenticates to node-1, but then later node-2 gets a request with the token (because its clustered and the user has no control over which node handles the request), node-2 cannot verify the token and rejects it. Even configuring sticky-sessions and cookie affinity in the nginx ingress proxy don't solve the problem as far as I can tell. I don't even know if having it all behind a single root path would make a difference for the authentication issue, it just makes setting up the reverse proxy configuration easier since you only need to worry about 1 path instead of multiple. On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman wrote: > Wyllys, > > NiFi is comprised of any number of web applications. NiFi offers extension > points for custom processor configuration UIs and data type viewers. These > UIs can be bundled and discovered at runtime. These docs [1] detail the > steps necessary for proxying a NiFi instance. > > Thanks > > Matt > > [1] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration > > On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > >> >> I'm surprised you haven't had lots of requests for this already. As it >> stands now, I cannot figure out how to configure a secure cluster behind a >> reverse proxy (for example, in a kubernetes environment behind an nginx >> ingress) that also incorporates OpenID authentication from an external >> service. I was thinking that if the NiFi nodes were able to operate under a >> single root path, it might make it easier to reverse proxy all of the >> different paths that Nifi uses (/nifi, /nifi-api, for example) behind a >> single ingress. I think having multiple ingress paths for the nifi service >> makes the reverse proxy configuration very complex when authentication >> tokens are involved. Without authentication, it works fine. >> >> Thanks, >> Wyllys Ingersoll >> >> On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto >> wrote: >> >>> Hi Wyllys, >>> >>> As I started reading, I was going to suggest the proxy approach. >>> Unfortunately, at this time I am unaware of any way to change the paths >>> within NiFi itself - there would be substantial refactoring required to >>> make that an option. You can open a feature request Jira for that, or >>> perhaps the ability to inject a path prefix, but I expect it to be a high >>> level of effort to implement. >>> >>> >>> Andy LoPresto >>> alopre...@apache.org >>> *alopresto.apa...@gmail.com * >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>> On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll < >>> wyllys.ingers...@keepertech.com> wrote: >>> >>> >>> Is there a way to configure nifi to use a different root directory for >>> web requests? >>> >>> We would like everything to be under a common root such as: >>> /XXX/nifi/... >>> /XXX/nifi-api/... >>> >>> Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to >>> setup a reverse proxy that also can incorporate OpenID authentication >>> tokens to a secure backend cluster of nodes. >>> >>> >>> >>> >>>
Re: alternate web root directory?
Wyllys, NiFi is comprised of any number of web applications. NiFi offers extension points for custom processor configuration UIs and data type viewers. These UIs can be bundled and discovered at runtime. These docs [1] detail the steps necessary for proxying a NiFi instance. Thanks Matt [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > I'm surprised you haven't had lots of requests for this already. As it > stands now, I cannot figure out how to configure a secure cluster behind a > reverse proxy (for example, in a kubernetes environment behind an nginx > ingress) that also incorporates OpenID authentication from an external > service. I was thinking that if the NiFi nodes were able to operate under a > single root path, it might make it easier to reverse proxy all of the > different paths that Nifi uses (/nifi, /nifi-api, for example) behind a > single ingress. I think having multiple ingress paths for the nifi service > makes the reverse proxy configuration very complex when authentication > tokens are involved. Without authentication, it works fine. > > Thanks, > Wyllys Ingersoll > > On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto > wrote: > >> Hi Wyllys, >> >> As I started reading, I was going to suggest the proxy approach. >> Unfortunately, at this time I am unaware of any way to change the paths >> within NiFi itself - there would be substantial refactoring required to >> make that an option. You can open a feature request Jira for that, or >> perhaps the ability to inject a path prefix, but I expect it to be a high >> level of effort to implement. >> >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com * >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll < >> wyllys.ingers...@keepertech.com> wrote: >> >> >> Is there a way to configure nifi to use a different root directory for >> web requests? >> >> We would like everything to be under a common root such as: >> /XXX/nifi/... >> /XXX/nifi-api/... >> >> Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to >> setup a reverse proxy that also can incorporate OpenID authentication >> tokens to a secure backend cluster of nodes. >> >> >> >> >>
Re: alternate web root directory?
I'm surprised you haven't had lots of requests for this already. As it stands now, I cannot figure out how to configure a secure cluster behind a reverse proxy (for example, in a kubernetes environment behind an nginx ingress) that also incorporates OpenID authentication from an external service. I was thinking that if the NiFi nodes were able to operate under a single root path, it might make it easier to reverse proxy all of the different paths that Nifi uses (/nifi, /nifi-api, for example) behind a single ingress. I think having multiple ingress paths for the nifi service makes the reverse proxy configuration very complex when authentication tokens are involved. Without authentication, it works fine. Thanks, Wyllys Ingersoll On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto wrote: > Hi Wyllys, > > As I started reading, I was going to suggest the proxy approach. > Unfortunately, at this time I am unaware of any way to change the paths > within NiFi itself - there would be substantial refactoring required to > make that an option. You can open a feature request Jira for that, or > perhaps the ability to inject a path prefix, but I expect it to be a high > level of effort to implement. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll < > wyllys.ingers...@keepertech.com> wrote: > > > Is there a way to configure nifi to use a different root directory for web > requests? > > We would like everything to be under a common root such as: > /XXX/nifi/... > /XXX/nifi-api/... > > Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to > setup a reverse proxy that also can incorporate OpenID authentication > tokens to a secure backend cluster of nodes. > > > > >
Re: alternate web root directory?
Hi Wyllys, As I started reading, I was going to suggest the proxy approach. Unfortunately, at this time I am unaware of any way to change the paths within NiFi itself - there would be substantial refactoring required to make that an option. You can open a feature request Jira for that, or perhaps the ability to inject a path prefix, but I expect it to be a high level of effort to implement. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll > wrote: > > > Is there a way to configure nifi to use a different root directory for web > requests? > > We would like everything to be under a common root such as: > /XXX/nifi/... > /XXX/nifi-api/... > > Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to > setup a reverse proxy that also can incorporate OpenID authentication tokens > to a secure backend cluster of nodes. > >
alternate web root directory?
Is there a way to configure nifi to use a different root directory for web requests? We would like everything to be under a common root such as: /XXX/nifi/... /XXX/nifi-api/... Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to setup a reverse proxy that also can incorporate OpenID authentication tokens to a secure backend cluster of nodes.
