On 05/29/2014 03:14 PM, Liam Hoekenga wrote:
>
> Did this start recently? I wonder if the recent (local) DNS
> changes might have affected how Kerberos clients function.
>
> The splunk logs only go back 6mos, but during that time, we've seen an
> avg of 478k of these errors / mo.
That pretty much absolves the (local) DNS and networking changes.
They're more recent than that.
> Similarly, are there any differences between the two data centers?
> (There were recent router changes to fix DNS problems on the one
> of the subnets involved.)
>
>
> The cosign servers are running on two different varieties of hardware
> (whitebox and ibm), but I believe the OS images are identical. I
> don't know of any networking differences.
>
> Liam
Looking at the code for the version I think is running there, the
"cosign_login_krb()" code calls "cosign_login()" which calls
(eventually) "cosign_choose_conn()" in "cgi/network.c". I believe that
because this code is trying to establish a TCP connection with "cosignd"
(because the CGI is launched new for every web hit), AND because it
ISN'T possible to set a timeout on TCP connection establishment, this
can result in the combined weblogin server/cosignd server collapsing
under excess load under certain exceptional conditions - in particular,
triggered when the connection to the first attempted IP address of the
cosignd server takes a long time. (Since the CGI is invoked anew each
time, there's no way to mark a particular IP address as bad across
invocations of the CGI.)
"cosignd" uses a "-b" parameter to set the "backlog" number (defaults to
5) - this number is passed as the "backlog" (2nd) parameter to the
"listen()" call set up by the cosignd(). If the master cosign daemon is
busy or slow for some reason, this might fill up with incoming requests
from the web server filters on the back channel. If all the cosignd()s
are similarily busy, this might mean that "cosign_choose_conn()" will
fail, apparently causing Kerberos to fail (when it hasn't.)
This is all speculation.
> On Thu, May 29, 2014 at 2:58 PM, Richard Conto <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi Liam,
>
> Did this start recently? I wonder if the recent (local) DNS
> changes might have affected how Kerberos clients function.
>
> Similarly, are there any differences between the two data centers?
> (There were recent router changes to fix DNS problems on the one
> of the subnets involved.)
>
>
> --- Richard Conto
> ...
>
>
> On Thu, May 29, 2014 at 2:18 PM, Liam Hoekenga <[email protected]
> <mailto:[email protected]>> wrote:
>
> Our cosign installation uses kerberos and friend as the
> primary authenticators.
> We're getting instances of this error in our logs:
>
> Couldn't identify an authenticator for /USERNAME/
>
> I can find where this error occurs in the source code, but I
> can't really figure out what would trigger it. The problem
> usernames all seem to be valid, active kerberos principals,
> and our logs indicate that sometimes the problem usernames
> *are* able to authenticate.
>
> The failures are initiated by a wide variety of cosign service
> providers. They seem to happen in batches - we had ~360
> between 14:00 - 14:15 today, though that might be a bad
> observation.. Splunk shows that we see ~5k-20k of these a day.
>
> Any ideas?
> Liam
>
------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet
_______________________________________________
Cosign-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/cosign-discuss
