Re: 1.8.0 Clustering Timeouts
After more analysis for the log, it appears the issue is likely in Jetty as
the stacktrace shows the relevant threads during the 30 seconds timeout is
sitting in sun's EpollSelector in native code.
I've dug as deep as I can. It can be reliably recreated in nifi-1.8.0 and
in nifi-1.9.2 in my specific environment, though it can't be recreated
outside of the environment. It isn't eating 100% CPU during the 30 second
timeout, it seems to just loose track of the connection/request until the
client timeout closes the connection. I've tried increasing the timeout to
3 minutes with no success. I've tried different JDK versions Oracle 8u202,
8u212, 8u65, OpenJDK (Redhat flavor) 8u212.
I've boiled the essence of the log down to this... if anyone has any
suggestions i'm all ears. I do wish NIFI-5581 had the symptoms associated
with the issue instead of just the solution.
2019-05-21 03:05:30,416 The client request is initiated on thread "NiFi Web
Server-32".
2019-05-21 03:05:30,494 The request makes it way to "NiFi Web Server-209"
(how?) and initiates Request Replication
2019-05-21 03:05:30,496 we see the new replicated connection get accepted
and it lands on "NiFi Web Server-24" and "NiFi Web Server-210".
2019-05-21 03:05:30,567 shows the full replicated request headers.
2019-05-21 03:05:30,569 the threads start waiting with ManagedSelector
(3663b227)
30 seconds go by with the JVM doing some heartbeating between cluster nodes.
2019-05-21 03:06:00,563 the ManagedSelector (3663b227) wakes up because the
socket was closed.
2019-05-21 03:06:00,570 the client gets a timeout exception from okhttp.
https://gist.github.com/ruckc/61e2ef68f084e859e225319e0ba6c458
--
Curtis Ruck
On Mon, May 20, 2019 at 11:58 PM Curtis Ruck wrote:
> Alright, round two occurred today, and after much frustration, I believe I
> know what exactly is causing the problem, and the solution is probably just
> upgrade to nifi-1.9.2 which comes with an upgraded jetty.
>
> Java 8u202 - Nifi 1.8.0 - Jetty 9.4.11 (same Jetty version is in Nifi
> 1.9.2)
>
> BLUF, a single request to node1/nifi-api/controller/cluster (on thread
> NiFi Web Server-209) replicates (thread Replicate Request Thread-4) to
> itself (node1/nifi-api/controller/cluster) with additional headers
> indicating it's a replicated request. The replicated request is received
> by thread NiFi Web Server-210. The thread parks "waiting" This request to
> controller/cluster wants to run in a ReservedThreadExecutor. The
> replicated request is stuck "waiting" for 30 seconds until OkHttp timeouts
> in EpollSelector land.
>
> My nifi.properties has 200 threads assigned to nifi.web.jetty.threads. I
> also have read.timeout set to 30 sec.
>
> ~6500 lines of debugging is here.
>
> https://gist.githubusercontent.com/ruckc/df947e04ae4fb55bc37ecc116f747848/raw/06625f0b6169acf272edc648a7fbd2abd043e0a4/node1.log
>
> Wish I knew where to go from here...
>
> --
> Curtis Ruck
>
>
> On Mon, May 20, 2019 at 10:06 AM Curtis Ruck
> wrote:
>
>> I have a new 3 node cluster i'm trying to stand up.
>>
>> On most of the API calls (to create proxy the proxy user) and others for
>> debugging, i get a "java.net.SocketTimeoutException: timeout" returned. In
>> debugging the issue, the timeout occurs when the API replicates the request
>> to the same instance receiving the request. I.e when node1 replicates the
>> request to node1.
>>
>> I've attached a pruned debug logfile showing the /controller/config
>> request with the current request "nifi.cluster.node.read.timeout=10 sec".
>> From what I can tell in the attached log is, Thread "Nifi Web Server-24"
>> gets a request for /nifi-api/controller/cluster" it replicates the request
>> to itself, which arrives 10 secs (the request timeout) later and comes in
>> on thread "NiFi Web Server-161" at roughly the same moment (5-6 ms after)
>> okhttp3 timeouts the request. When 161 goes to respond the connection is
>> already closed. So I think what's happening is that node1 locks/blocks
>> something that the replicated request needs to read from, but can't until
>> the lock is released when the request timeouts. So why is the request
>> getting replicated to itself?
>>
>> In troubleshooting I tried finding relevant REST endpoints and i get
>> these results:
>>
>> /controller/config -> java.net.SocketTimeoutException: timeout
>> /flow/status -> java.net.SocketTimeoutException: timeout
>>
>> /flow/cluster/summary -> {
>> clusterSummary => {
>> connectedNodes => "3 / 3",
>> connectedNodeCount => 3,
>> totalNodeCount => 3,
>
Re: 1.8.0 Clustering Timeouts
Alright, round two occurred today, and after much frustration, I believe I
know what exactly is causing the problem, and the solution is probably just
upgrade to nifi-1.9.2 which comes with an upgraded jetty.
Java 8u202 - Nifi 1.8.0 - Jetty 9.4.11 (same Jetty version is in Nifi 1.9.2)
BLUF, a single request to node1/nifi-api/controller/cluster (on thread NiFi
Web Server-209) replicates (thread Replicate Request Thread-4) to itself
(node1/nifi-api/controller/cluster) with additional headers indicating it's
a replicated request. The replicated request is received by thread NiFi
Web Server-210. The thread parks "waiting" This request to
controller/cluster wants to run in a ReservedThreadExecutor. The
replicated request is stuck "waiting" for 30 seconds until OkHttp timeouts
in EpollSelector land.
My nifi.properties has 200 threads assigned to nifi.web.jetty.threads. I
also have read.timeout set to 30 sec.
~6500 lines of debugging is here.
https://gist.githubusercontent.com/ruckc/df947e04ae4fb55bc37ecc116f747848/raw/06625f0b6169acf272edc648a7fbd2abd043e0a4/node1.log
Wish I knew where to go from here...
--
Curtis Ruck
On Mon, May 20, 2019 at 10:06 AM Curtis Ruck wrote:
> I have a new 3 node cluster i'm trying to stand up.
>
> On most of the API calls (to create proxy the proxy user) and others for
> debugging, i get a "java.net.SocketTimeoutException: timeout" returned. In
> debugging the issue, the timeout occurs when the API replicates the request
> to the same instance receiving the request. I.e when node1 replicates the
> request to node1.
>
> I've attached a pruned debug logfile showing the /controller/config
> request with the current request "nifi.cluster.node.read.timeout=10 sec".
> From what I can tell in the attached log is, Thread "Nifi Web Server-24"
> gets a request for /nifi-api/controller/cluster" it replicates the request
> to itself, which arrives 10 secs (the request timeout) later and comes in
> on thread "NiFi Web Server-161" at roughly the same moment (5-6 ms after)
> okhttp3 timeouts the request. When 161 goes to respond the connection is
> already closed. So I think what's happening is that node1 locks/blocks
> something that the replicated request needs to read from, but can't until
> the lock is released when the request timeouts. So why is the request
> getting replicated to itself?
>
> In troubleshooting I tried finding relevant REST endpoints and i get these
> results:
>
> /controller/config -> java.net.SocketTimeoutException: timeout
> /flow/status -> java.net.SocketTimeoutException: timeout
>
> /flow/cluster/summary -> {
> clusterSummary => {
> connectedNodes => "3 / 3",
> connectedNodeCount => 3,
> totalNodeCount => 3,
> connectedToCluster => true,
> clustered => true
> }
> }
>
> /controller/cluster -> {
> cluster => {
> nodes => [
> {
> nodeId => "8d066e95-b254-4527-a214-79ca5359f078",
> address => "node2.example.com",
> apiPort => 8443,
> status => "CONNECTED",
> heartbeat => "05/20/2019 02:19:13 UTC",
> roles => [
> "Cluster Coordinator"
> ],
> activeThreadCount => 0,
> queued => "0 / 0 bytes",
> events => [
> {
> timestamp => "05/20/2019 02:17:35 UTC",
> category => "INFO",
> message => "Received first heartbeat from connecting node.
> Node connected."
> },
> {
> timestamp => "05/20/2019 02:17:20 UTC",
> category => "INFO",
> message => "Connection requested from existing node. Setting
> status to connecting."
> },
> {
> timestamp => "05/20/2019 02:17:20 UTC",
> category => "INFO",
> message => "Connection requested from existing node. Setting
> status to connecting."
> },
> {
> timestamp => "05/20/2019 02:17:14 UTC",
> category => "INFO",
> message => "Requesting that node connect to cluster"
> },
> {
> timestamp => "05/20/2019 02:17:14 UTC",
> category => "INFO",
> message => "Received heartbeat from node previously
> disconnected due to Has Not Yet Connected
> }
> ],
> nodeStartTime => "05/20/2019 02:17:08 U
1.8.0 Clustering Timeouts
r"
}
],
nodeStartTime => "05/20/2019 02:17:06 UTC"
},
{
nodeId => "334f49b5-d714-4013-8180-0ffca95aa829",
address => " node1.example.com ",
apiPort => 8443,
status => "CONNECTED",
heartbeat => "05/20/2019 02:19:10 UTC",
roles => [],
activeThreadCount => 0,
queued => "0 / 0 bytes",
events => [
{
timestamp => "05/20/2019 02:17:35 UTC",
category => "INFO",
message => "Received first heartbeat from connecting node. Node
connected."
},
{
timestamp => "05/20/2019 02:17:23 UTC",
category => "INFO",
message => "Connection requested from existing node. Setting
status to connecting."
},
{
timestamp => "05/20/2019 02:17:17 UTC",
category => "INFO",
message => "Connection requested from existing node. Setting
status to connecting."
},
{
timestamp => "05/20/2019 02:17:15 UTC",
category => "INFO",
message => "Received heartbeat from node previously
disconnected due to Has Not Yet Connected
},
{
timestamp => "05/20/2019 02:17:15 UTC",
category => "INFO",
message => "Requesting that node connect to cluster"
}
],
nodeStartTime => "05/20/2019 02:17:06 UTC"
}
],
generated => "02:19:15 UTC"
}
}
--
Curtis Ruck
2019-05-18 20:02:33,115 DEBUG [NiFi Web Server-24]
o.a.nifi.web.server.HostHeaderHandler HostHeaderHandler#doScope on
/nifi-api/controller/cluster
2019-05-18 20:02:33,115 DEBUG [NiFi Web Server-24]
o.a.nifi.web.server.HostHeaderHandler Received request
[/nifi-api/controller/cluster] with host header: node1.example.com:8443
2019-05-18 20:02:33,134 DEBUG [FileSystemRepository Workers Thread-2]
o.a.n.c.r.c.StandardResourceClaimManager Drained 0 destructable claims to []
2019-05-18 20:02:33,258 DEBUG [NiFi Web Server-24]
o.apache.nifi.util.ComponentIdGenerator Generating UUID
cc89f0aa-016a-1000-ec9a-4b7333661f42 for msb=1558209753258,
lsb=-1397721776189464766, ensureUnique=true
2019-05-18 20:02:33,259 DEBUG [NiFi Web Server-24]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Obtaining lock
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@1ea8b09f[Read locks
= 0] in order to replicate request GET
https://node1.example.com:8443/nifi-api/controller/cluster
2019-05-18 20:02:33,259 DEBUG [NiFi Web Server-24]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Lock
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@1ea8b09f[Read locks
= 1] obtained in order to replicate request GET
https://node1.example.com:8443/nifi-api/controller/cluster
2019-05-18 20:02:33,260 DEBUG [NiFi Web Server-24]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Replicating request GET
https://node1.example.com:8443/nifi-api/controller/cluster with entity {} to
[node1.example.com:8443]; response is null
2019-05-18 20:02:33,265 DEBUG [NiFi Web Server-24]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator For Request ID
0bbc58a4-683c-40de-9273-60abaa9707e9, response object is
StandardAsyncClusterResponse[id=0bbc58a4-683c-40de-9273-60abaa9707e9,
uri=https://node1.example.com:8443/nifi-api/controller/cluster, method=GET,
failure=false, responses=0/1]
2019-05-18 20:02:33,279 DEBUG [Replicate Request Thread-1]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Replicating request GET
/nifi-api/controller/cluster to node1.example.com:8443
2019-05-18 20:02:33,279 DEBUG [Replicate Request Thread-1]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Replicating request to GET
https://node1.example.com:8443/nifi-api/controller/cluster, request ID = null,
headers = {Accept=application/json, text/javascript,
X-Cluster-Id-Generation-Seed=cc89f0aa-016a-1000-ec9a-4b7333661f42,
Referer=https://127.0.0.1:8443/nifi/login, Connection=close,
X-Request-Replicated=true, X-ProxyPort=8443, X-ProxyHost=node1.example.com,
X-ProxiedEntitiesChain=,
X-RequestTransactionId=0bbc58a4-683c-40de-9273-60abaa9707e9,
Accept-Language=en-US,en;q=0.5, Content-Length=2,
Content-Type=application/json, X-ProxyScheme=https}
2019-05-18 20:02:33,280 DEBUG [Replicate Request Thread-1]
o.a.n.c.c.h.r.o.OkHttpReplicationClient Replicating request
OkHttpPreparedRequest[method=GET, headers={Accept=application/json,
text/javascript,
X-Cluster-Id-Generation-Seed=cc89f0aa-016a-1000-ec9a-4b7333661f42,
Referer=https://127.0.0.1:8443/nifi/login, Connection=close,
X-Request-Replicated=true, X-ProxyPort=8443, X-ProxyHost=node1.example.com,
X-Proxie
Re: X-Forwarded-Context whitelisting not working
Yes, went through that guide with a fine tooth comb. Then started enabling debug logging and comparing log outputs. The telling factor in the logs were these lines (/nifi1 in Sanitize and nothing in CatchAll). o.a.n.w.filter.SanitizeContextPathFilter - SanitizeContextPathFilter received provided whitelisted context paths from NiFi properties: /nifi1 o.a.n.w.filter.CatchAllFilter - CatchAllFilter [index.jsp] received provided whitelisted context paths from NiFi properties: Then investigation into how whitelistedContextPaths gets set revealed the likely culprit is the lack of a super.init(filterConfig) inside CatchAllFilter.init(). -- Curtis Ruck On Wed, May 15, 2019 at 8:50 AM Matt Gilman wrote: > Curtis, > > I haven't set this up recently but it was working the last time I tried > it. Just wanted to ensure you that were following the guidance in our admin > guide for standing up instances behind a proxy [1]. > > Matt > > [1] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration > > On Tue, May 14, 2019 at 6:17 PM Curtis Ruck wrote: > >> I am attempting (unsuccessfully) to configure multiple numbered >> unclustered nifi instances behind a single reverse proxy vhost (external >> limitations on single vhost and no clustering) >> >> In my reverse proxy I have X-Forwarded-Context set and in debug logging >> both CatchAllFilter and SanitizeContextPathFilter see the passed context >> (/nifi1 ... /nifiN). In debug logging though, CatchAllFilter isn't seeing >> any items from getWhitelistedContextPath() where as >> SanitizeContextPathFilter does show the items in the >> getWhitelistedContextPath(). >> >> Since CatchAllFilter extends SanitizeContextPathFilter, it should work >> except CatchAllFilter isn't calling super.init() which means the >> private whitelistedContextPaths never gets initialized. >> >> Has anyone gotten Nifi working at a nested context path i.e. >> (/nifi1/nifi, /nifi1/nifi-api)? >> >> -- >> Curtis Ruck >> >
Re: SAML based identity provider
I've been trying to figure out how to improve this area of NiFi. They support OpenID Direct Connect (OIDC), but when you combine it with a reverse proxy or their default/hardcoded PKI configuration, it's near impossible to use. Ideally the entire authn/z stack needs rearchitecting for better modularity for any decent SSO integration. The current APIs were built around having a writable authn/z store like LDAP/RDBMS. They are not designed for common SSO workflows where users connect to NiFi and inherit NiFi permissions based on their assertion/attributes. On Fri, Aug 31, 2018, 6:14 PM Vijay Chhipa wrote: > Hello, > > I am setting up NiFi in the company, but the out-of-the-box authentication > modules are not an option for me. > I would like to write a SAML based login identity provider, > Is there one out there already ? > > I am on NiFi 1.7.1, with Java 8, SAML 2.0, > > What do I need to get started with writing a new login identity provider? > Any examples, sample, or pointers are highly appreciated > > Vijay > >
Re:
I created NIFI-5506 for the wantClientAuth specific issue, and submitted a
WIP PR#2944 for review.
Besides issues with getting OIDC working (on the OIDC Server side), this
enables external providers. Potentially, this could be amended to include
X509 through reverse proxy by way of a request header, but considering that
wouldn't work with a reverse proxy without this PR, I considered it out of
scope of my near term issue.
--
Curtis Ruck
On Thu, Aug 9, 2018 at 3:47 PM Curtis Ruck wrote:
> The issue with Reverse Proxies and "certificates or other provider" is
> that if want=true, then the reverse proxy provides it's certificate, if a
> machine certificate is configured. In Apache HTTPD, this machine
> certificate can be set at a Server or VHost level, not individual proxy
> rules, so to remove it for NiFi, i have to remove it for our other apps
> that "require" X509 client auth, and then either do a SSO workflow, or
> consume Reverse Proxy provided "authentication" details. This turns
> "certificates or other provider" ends up at "reverse proxy certificate
> only". So if Bob and Tim visit the reverse proxy, Nifi believes they are
> "Reverse Proxy" not Bob or Tim.
>
> Ideally I need "other provider", because my "other provider" does PKI
> authentication as part of SSO. I could use "certificates or other
> provider", if NiFi could recognize Reverse Proxy validated certificates
> passed in via a request header. If the reverse proxy doesn't provide a
> certificate, then it uses "other provider". JBoss, Tomcat, all provide
> this functionality.
>
> I've already changed " else { setWantClientAuth(false); }" change in a
> fork and it got me closer to my customer's end goals. I'm trying to get
> this change into NiFi so we don't have to maintain this fork. I believe
> implementing this with a default want=true would not break any existing
> users of NiFi, and it would allow better integration with Reverse Proxies
> and Single Sign On.
>
> So In the near term i'd like a new nifi property setting to disable
> wantClientAuth, with the default enabled.
> In the long term it would be ideal to support external authn/z providers
> as first class.
>
> --
> My perspective comes from implementing Single Sign On in applications that
> don't always support it for over a decade for ~100 applications all sitting
> behind a Reverse Proxies, providing true single sign on without users
> having to do any special instructions to authenticate. I'm a true believer
> that the best security is when the security doesn't impact the users, and
> proper single sign on allows application developers focus on their
> application's logic and not their AuthN/AuthZ security model.
>
> --
> Curtis Ruck
>
>
>
> On Thu, Aug 9, 2018 at 3:00 PM Andy LoPresto wrote:
>
>> I think we agree in our assessment of what the code is doing and disagree
>> in our desire for how that should occur. If OIDC is enabled and
>> isClientAuthRequiredForRestApi() returns false, the result is:
>>
>> // Functionally equivalent to contextFactory.setNeedClientAuth(false);
>> contextFactory.setWantClientAuth(true);
>>
>> That means that the server will request a client certificate if
>> available, but will not require its presence to negotiate the TLS
>> handshake. You are asking to set contextFactory.setWantClientAuth(false);
>> as well, which will suppress the certificate selection dialog. If
>> needClientAuth and wantClientAuth are both false, client certificates
>> cannot be used to authenticate as they will never be sent from the browser.
>> This will effectively allow you to choose between “certificates only”,
>> “certificates or other provider”, and “other provider only (no
>> certificates)”.
>>
>> I am saying that core NiFi *always* accepts client certificates as an
>> authentication mechanism; there is no scenario in which need and want are
>> both set to false. This is by design. Again, I am not saying this can never
>> change, but because of the expectations, documentation, and shared
>> knowledge around this mechanism, changing it is (in my opinion) a major
>> change, and should not be done in a minor release. Other project members
>> may (and probably do) disagree with me.
>>
>> A property in nifi.properties which defaults to “off” but when manually
>> enabled can bypass this requirement is an option. I don’t think we disagree
>> on how to implement this specific change; I think we differ only on how
>> impactful it will be. My perspective comes from supporting a large number
>> of users with a broad variety of (often conflict
Re:
The issue with Reverse Proxies and "certificates or other provider" is that
if want=true, then the reverse proxy provides it's certificate, if a
machine certificate is configured. In Apache HTTPD, this machine
certificate can be set at a Server or VHost level, not individual proxy
rules, so to remove it for NiFi, i have to remove it for our other apps
that "require" X509 client auth, and then either do a SSO workflow, or
consume Reverse Proxy provided "authentication" details. This turns
"certificates or other provider" ends up at "reverse proxy certificate
only". So if Bob and Tim visit the reverse proxy, Nifi believes they are
"Reverse Proxy" not Bob or Tim.
Ideally I need "other provider", because my "other provider" does PKI
authentication as part of SSO. I could use "certificates or other
provider", if NiFi could recognize Reverse Proxy validated certificates
passed in via a request header. If the reverse proxy doesn't provide a
certificate, then it uses "other provider". JBoss, Tomcat, all provide
this functionality.
I've already changed " else { setWantClientAuth(false); }" change in a fork
and it got me closer to my customer's end goals. I'm trying to get this
change into NiFi so we don't have to maintain this fork. I believe
implementing this with a default want=true would not break any existing
users of NiFi, and it would allow better integration with Reverse Proxies
and Single Sign On.
So In the near term i'd like a new nifi property setting to disable
wantClientAuth, with the default enabled.
In the long term it would be ideal to support external authn/z providers as
first class.
--
My perspective comes from implementing Single Sign On in applications that
don't always support it for over a decade for ~100 applications all sitting
behind a Reverse Proxies, providing true single sign on without users
having to do any special instructions to authenticate. I'm a true believer
that the best security is when the security doesn't impact the users, and
proper single sign on allows application developers focus on their
application's logic and not their AuthN/AuthZ security model.
--
Curtis Ruck
On Thu, Aug 9, 2018 at 3:00 PM Andy LoPresto wrote:
> I think we agree in our assessment of what the code is doing and disagree
> in our desire for how that should occur. If OIDC is enabled and
> isClientAuthRequiredForRestApi() returns false, the result is:
>
> // Functionally equivalent to contextFactory.setNeedClientAuth(false);
> contextFactory.setWantClientAuth(true);
>
> That means that the server will request a client certificate if available,
> but will not require its presence to negotiate the TLS handshake. You are
> asking to set contextFactory.setWantClientAuth(false); as well, which will
> suppress the certificate selection dialog. If needClientAuth and
> wantClientAuth are both false, client certificates cannot be used to
> authenticate as they will never be sent from the browser. This will
> effectively allow you to choose between “certificates only”, “certificates
> or other provider”, and “other provider only (no certificates)”.
>
> I am saying that core NiFi *always* accepts client certificates as an
> authentication mechanism; there is no scenario in which need and want are
> both set to false. This is by design. Again, I am not saying this can never
> change, but because of the expectations, documentation, and shared
> knowledge around this mechanism, changing it is (in my opinion) a major
> change, and should not be done in a minor release. Other project members
> may (and probably do) disagree with me.
>
> A property in nifi.properties which defaults to “off” but when manually
> enabled can bypass this requirement is an option. I don’t think we disagree
> on how to implement this specific change; I think we differ only on how
> impactful it will be. My perspective comes from supporting a large number
> of users with a broad variety of (often conflicting) requirements, and
> sometimes (both they and I have) very little knowledge of the rest of their
> IT ecosystem. I believe your perspective comes from a specific user with
> specific requirements. That’s why I recommended making the localized change
> you need in a fork of the project, so you can achieve your objective in a
> timeframe that is not blocked by other parties.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Aug 9, 2018, at 11:48 AM, Curtis Ruck wrote:
>
> In my environment, by trying to enable OIDC, it returns false in that
> function you selected.
>
> My suggestion, is that in the } else { block, changing the
> setWantClientAuth(true) to
> setWantClientAu
Re:
In my environment, by trying to enable OIDC, it returns false in that
function you selected.
My suggestion, is that in the } else { block, changing the
setWantClientAuth(true) to
setWantClientAuth(nifiproperties.isWantClientAuth()) which can default to
true in the absence of the setting.
By allowing a property to disable this check, would neuter the current X509
Authentication, as it won't have a certificate to authenticate. It would
also address Shawn's concern of for having his users cancel the first
authentication popup.
It's not fixing AuthN/AuthZ, but its allowing us in special circumstances
to disable X509 easily. In my environment, it's even preferable because we
would use OIDC to redirect to Apereo CAS, which does X509 Authentication
itself.
--
Curtis Ruck
On Thu, Aug 9, 2018 at 2:43 PM Andy LoPresto wrote:
> Hi Curtis,
>
> There has definitely been some discussion about this and it is picking up
> recently. I understand the difficulty faced when using NiFi in conjunction
> with a reverse proxy or external identity provider. I am not saying the
> current way is perfect, or even the best.
>
> …but…
>
> Always allowing X.509 authentication has been the standard for years (due
> to NiFi’s original design requirements), and changing this will have
> far-reaching impact on the application. The additional identity providers
> have been added piece-meal as requirements arose, and because of the
> multiple contexts NiFi supports, there are additional changes required for
> each — issuing one time tokens for file downloads, etc.
>
> The code you identified returns a boolean value, but it is not simply
> reading a boolean flag from the properties file — it is calculating the
> presence/absence of the other (non-credential-based) identity providers:
>
> /**
> * Returns true if client certificates are required for REST API. Determined
> * if the following conditions are all true:
> *
> * - login identity provider is not populated
> * - Kerberos service support is not enabled
> * - openid connect is not enabled
> * - knox sso is not enabled
> *
> *
> * @return true if client certificates are required for access to the REST API
> */
> public boolean isClientAuthRequiredForRestApi() {
> return !isLoginIdentityProviderEnabled() &&
> !isKerberosSpnegoSupportEnabled() && !isOidcEnabled() && !isKnoxSsoEnabled();
> }
>
> There is active planning for a complete rewrite of the authentication
> mechanism ordering, but because this is such a wide-reaching change and
> will have substantial impact across the project, I strongly advocate for
> this to go in a major release. As the number of users of the project
> continues to grow, we have to balance improving the experience in edge
> scenarios against more common scenarios. While I don’t mean to negate the
> times a reverse proxy is required, it is not present in the majority of
> deployments, and the current model works sufficiently in those deployments.
> We have to keep those deployments in mind as well while we make these
> changes.
>
> If you have an immediate need to deploy NiFi behind a reverse proxy and
> disable the X.509 requests, my honest suggestion would be to fork the
> repository and apply a patch to return false from that method just as you
> specified. Unfortunately, that patch alone probably would not be accepted
> into the upstream NiFi project at this time. It can certainly be documented
> as a requirement for the rearchitected authentication mechanism moving
> forward.
>
> Thanks for sharing your expectations and needs from the project, and
> hopefully we can meet them soon.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Aug 9, 2018, at 11:28 AM, Curtis Ruck wrote:
>
> FYSA,
>
> This is where X509 is "always-on".
>
>
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L781-L785
>
> if (props.isClientAuthRequiredForRestApi()) {
>contextFactory.setNeedClientAuth(true);
> } else {
>contextFactory.setWantClientAuth(true);
> }
>
> I believe in the short term, modifying this section to use nifi.properties
> to allow us to provide a false to wantClientAuth, would address our
> concerns.
>
> --
> Curtis Ruck
>
>
> On Thu, Aug 9, 2018 at 12:54 PM Curtis Ruck wrote:
>
>> To support Shawn's statement even further. If my customer can't get NiFi
>> to operate behind our reverse proxy, it won't be in our system. I'm trying
>> to find the easiest approach, and NiFi's OIDC should be perfect, if X509
>> wasn't &quo
Re: Re:
FYSA,
This is where X509 is "always-on".
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L781-L785
if (props.isClientAuthRequiredForRestApi()) {
contextFactory.setNeedClientAuth(true);
} else {
contextFactory.setWantClientAuth(true);
}
I believe in the short term, modifying this section to use nifi.properties
to allow us to provide a false to wantClientAuth, would address our
concerns.
--
Curtis Ruck
On Thu, Aug 9, 2018 at 12:54 PM Curtis Ruck wrote:
> To support Shawn's statement even further. If my customer can't get NiFi
> to operate behind our reverse proxy, it won't be in our system. I'm trying
> to find the easiest approach, and NiFi's OIDC should be perfect, if X509
> wasn't "wanted" up front.
>
> I'd argue that all of the AuthN/AuthZ code should be abstracted out
> significantly more than it currently is, with the ability to completely
> configure it via nifi.properties, and mix-in custom AuthN/AuthZ solutions.
> The ability to manage users/groups in NiFi's UI should be a toggle.
> There should be an easy higher level API to use for group/role
> provisioning. If a new user "bob" open's NiFi and they have a "read-only"
> role, then they shouldn't need to be manually provisioned in NiFi, and we
> my customer tries to minimize the number of unique applications reaching
> into LDAP. Every application that implements LDAP support implements it
> differently, and they don't always scale up appropriately.
>
> For example, i'm trying to get Apereo CAS 5.x working with Apache NiFi.
> With CAS, it can provide SAML 2.0, SAML 1.1, OpenID Connect, or CAS's
> custom protocols, which can support Yubikey, Google Authentication, ADFS,
> Azure AD, etc. Sadly, because of the wantClientAuth(true) I can't use any
> of it.
>
> I'm even willing to assist in providing some PRs to move NiFi in the right
> direction, I just think we should figure out the higher level
> architecture/design a little better; especially since NiFi's job is to help
> things integrate together, it's not being a good team player.
>
> As much as I hate to say it, if NiFi was a proper Java EE project, I could
> just use a war overlay to modify the AuthN/AuthZ to success; even if it was
> just a self-executing .war.
>
> --
> Curtis Ruck
>
>
> On Thu, Aug 9, 2018 at 12:14 PM Shawn Weeks
> wrote:
>
>> I'll clarify my statement a little as well with a workflow.
>>
>>
>>
>>1. You open the NiFi UI Link
>>2. Chrome sees NiFi Asking for SSL and Prompts You for Cert
>>3. Then you get Prompts for Username and Password because of GSSAPI
>>even though your not on that REALM.
>>4. Then you get directed to the Identify Management Reverse Proxy URL
>>for Knox SSO
>>5. Then you get prompted for your Certificate which you should select.
>>6. Then you might get prompted for Kerberos Again which you cancel
>>7. Finally your in NiFi.
>>
>>
>> Painful doesn't even begin to describe it lol.
>>
>>
>> Thanks
>>
>> Shawn
>> --
>> *From:* Kevin Doran
>> *Sent:* Thursday, August 9, 2018 11:07:28 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Re:
>>
>>
>> *Explaining to your end users that you should skip the first Certificate
>> Prompt but accept the second but only when you haven't logged in the
>> current session is really painful*
>>
>>
>> Wow, that sounds terrible. Confusing, accident prone, and frustrating to
>> correct mistakes (at least in my experience, forcing a browser to forget
>> client certificate preferences is difficult).
>>
>> Thanks for sharing those details about your deployment scenario. This can
>> definitely be improved and I have some ideas for how to do it. I've cloned
>> the issue to NiFi to make sure we are tracking it for both projects [1][2]
>>
>> [1] https://issues.apache.org/jira/browse/NIFIREG-189
>> [2] https://issues.apache.org/jira/browse/NIFI-5504
>>
>> On Thu, Aug 9, 2018 at 11:54 AM, Shawn Weeks
>> wrote:
>>
>> The project I'm on is running into this issue as well and it gets
>> particularly painful when all of your server's are signed by the same root
>> ca that signs your smart card logins and your using something like KnoxSSO.
>> Explaining to your end users that you should skip the first Certificate
>> Prompt but accept the second but only when you haven't logged in the
>> current session is really painful and shows major shortcoming between the
>> back end authentication between servers and fr
Re: Re:
To support Shawn's statement even further. If my customer can't get NiFi to operate behind our reverse proxy, it won't be in our system. I'm trying to find the easiest approach, and NiFi's OIDC should be perfect, if X509 wasn't "wanted" up front. I'd argue that all of the AuthN/AuthZ code should be abstracted out significantly more than it currently is, with the ability to completely configure it via nifi.properties, and mix-in custom AuthN/AuthZ solutions. The ability to manage users/groups in NiFi's UI should be a toggle. There should be an easy higher level API to use for group/role provisioning. If a new user "bob" open's NiFi and they have a "read-only" role, then they shouldn't need to be manually provisioned in NiFi, and we my customer tries to minimize the number of unique applications reaching into LDAP. Every application that implements LDAP support implements it differently, and they don't always scale up appropriately. For example, i'm trying to get Apereo CAS 5.x working with Apache NiFi. With CAS, it can provide SAML 2.0, SAML 1.1, OpenID Connect, or CAS's custom protocols, which can support Yubikey, Google Authentication, ADFS, Azure AD, etc. Sadly, because of the wantClientAuth(true) I can't use any of it. I'm even willing to assist in providing some PRs to move NiFi in the right direction, I just think we should figure out the higher level architecture/design a little better; especially since NiFi's job is to help things integrate together, it's not being a good team player. As much as I hate to say it, if NiFi was a proper Java EE project, I could just use a war overlay to modify the AuthN/AuthZ to success; even if it was just a self-executing .war. -- Curtis Ruck On Thu, Aug 9, 2018 at 12:14 PM Shawn Weeks wrote: > I'll clarify my statement a little as well with a workflow. > > > >1. You open the NiFi UI Link >2. Chrome sees NiFi Asking for SSL and Prompts You for Cert >3. Then you get Prompts for Username and Password because of GSSAPI >even though your not on that REALM. >4. Then you get directed to the Identify Management Reverse Proxy URL >for Knox SSO >5. Then you get prompted for your Certificate which you should select. >6. Then you might get prompted for Kerberos Again which you cancel >7. Finally your in NiFi. > > > Painful doesn't even begin to describe it lol. > > > Thanks > > Shawn > -- > *From:* Kevin Doran > *Sent:* Thursday, August 9, 2018 11:07:28 AM > *To:* users@nifi.apache.org > *Subject:* Re: Re: > > > *Explaining to your end users that you should skip the first Certificate > Prompt but accept the second but only when you haven't logged in the > current session is really painful* > > > Wow, that sounds terrible. Confusing, accident prone, and frustrating to > correct mistakes (at least in my experience, forcing a browser to forget > client certificate preferences is difficult). > > Thanks for sharing those details about your deployment scenario. This can > definitely be improved and I have some ideas for how to do it. I've cloned > the issue to NiFi to make sure we are tracking it for both projects [1][2] > > [1] https://issues.apache.org/jira/browse/NIFIREG-189 > [2] https://issues.apache.org/jira/browse/NIFI-5504 > > On Thu, Aug 9, 2018 at 11:54 AM, Shawn Weeks > wrote: > > The project I'm on is running into this issue as well and it gets > particularly painful when all of your server's are signed by the same root > ca that signs your smart card logins and your using something like KnoxSSO. > Explaining to your end users that you should skip the first Certificate > Prompt but accept the second but only when you haven't logged in the > current session is really painful and shows major shortcoming between the > back end authentication between servers and front end ui authentication. > > > We can't even considering putting it behind our identify reverse proxies > because we can't turn off two way ssl. > > > Thanks > > Shawnk > -- > *From:* Kevin Doran > *Sent:* Thursday, August 9, 2018 10:47:56 AM > *To:* users@nifi.apache.org > *Subject:* Re: > > sorry forgot the link. here it is: > > [1] https://issues.apache.org/jira/projects/NIFIREG/issues/NIFIREG-189 > > On Thu, Aug 9, 2018 at 11:47 AM, Kevin Doran wrote: > > Hi Curtis, > > This has come up a few times. Unfortunately I don’t think there is > currently an easy way to disable X509-based identity extraction in NiFi > today. There is an open JIRA for the same issue in NiFi Registry [1]. NiFi > Registry follows the same AuthN/AuthZ design (and a fair amount of code) as > NiFi, so this ticket should apply to NiFi as well. > > Perhaps
[no subject]
I'm trying to setup OIDC authentication, but with Nifi service existing behind a reverse proxy, and for our other apps we use SSL Client Authentication between reverse proxy and application, Nifi is picking up the Reverse Proxy's SSL Certificate and falling into X509 Authentication instead of OIDC. Any idea how I can disable X509 authentication in Nifi? Connecting directly to nifi, it triggers the proper OIDC redirects. -- Curtis Ruck
