unsubscribe -----Original Message----- From: kerberos-boun...@mit.edu [mailto:kerberos-boun...@mit.edu] On Behalf Of kerberos-requ...@mit.edu Sent: Tuesday, August 09, 2011 12:40 PM To: kerberos@mit.edu Subject: EXT :Kerberos Digest, Vol 104, Issue 11
Send Kerberos mailing list submissions to kerberos@mit.edu To subscribe or unsubscribe via the World Wide Web, visit https://mailman.mit.edu/mailman/listinfo/kerberos or, via email, send a message with subject or body 'help' to kerberos-requ...@mit.edu You can reach the person managing the list at kerberos-ow...@mit.edu When replying, please edit your Subject line so it is more specific than "Re: Contents of Kerberos digest..." Today's Topics: 1. Re: Performance issues with krb5-1.9.1 (Jonathan Reams) 2. Re: Performance issues with krb5-1.9.1 (Jonathan Reams) 3. Problem after adding new realm in krb5.conf (Sanket Sangodkar) 4. Error while adding new realm to krb5.conf (Sangodkar, Sanket) 5. Re: Error while adding new realm to krb5.conf (Vipin Rathor) 6. Re: Performance issues with krb5-1.9.1 (Chris Hecker) ---------------------------------------------------------------------- Message: 1 Date: Mon, 8 Aug 2011 15:49:26 -0400 From: Jonathan Reams <jr3...@columbia.edu> Subject: Re: Performance issues with krb5-1.9.1 To: Greg Hudson <ghud...@mit.edu> Cc: "kerberos@mit.edu" <kerberos@mit.edu> Message-ID: <75985df2-f49b-40fb-9473-e856f6eeb...@columbia.edu> Content-Type: text/plain; charset=us-ascii Hi Greg, I applied this patch and saw a great improvement on our test KDC. I should follow up and say that the performance degradation appeared to be compounded by clients resubmitting their requests after they timed out, so the KDC wasn't only handling new requests, it was trying to fulfill old requests that were being re-submitted. Below is the results of my little performance test with the patch applied to krb5-1.9.1. [root@naiad ~]# for i in `seq 1 4`; do time for j in `seq 1 1000`; do kinit -k; kvno iprop_monitor | grep -v kvno; done; echo finished round $i; sleep 5; done real 0m20.312s user 0m4.513s sys 0m13.690s finished round 1 real 0m20.842s user 0m4.512s sys 0m13.896s finished round 2 real 0m21.123s user 0m4.605s sys 0m13.787s finished round 3 real 0m21.425s user 0m4.508s sys 0m13.642s finished round 4 This is much better than the 56 seconds we saw without the patch. We'll roll this out to our secondary KDC and see how it goes this week. Thanks for resolving this so quickly. Jonathan Reams CUIT Systems Engineering Columbia University On Aug 8, 2011, at 2:21 PM, Greg Hudson wrote: > > I found a regression which would affect these tests, but I'm not sure it > accounts for your global performance issues. > > The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache, > but due to a bug, it is actually opening and reading a replay cache for > every TGS request, which is significantly less efficient than the 1.8 > behavior (using a replay cache which stays open for the lifetime of the > KDC). > > In a test which runs in under five minutes, this regression produces > visible O(n^2) performance characteristics. This would not necessarily > account for performance degradation over hours, as the performance drag > of the replay cache should become stable after five minutes. It's > possible that the constant drag was enough to cause the KDC to fall > behind on the request load, but it's also possible that there's a second > problem which isn't so easily reproduced. > > I've attached a patch. Note that there is a second, in-memory > "lookaside" cache with O(n^2) performance characteristics in the short > term, which holds queries for up to two minutes. You may see a slight > degradation in performance in test cases due to this. You can > temporarily rebuild the kdc directory with "make clean; > CPPFLAGS=-DNOCACHE" if you want to remove this variable from your > performance tests. ------------------------------ Message: 2 Date: Tue, 9 Aug 2011 10:13:02 -0400 From: Jonathan Reams <jr3...@columbia.edu> Subject: Re: Performance issues with krb5-1.9.1 To: kerberos@mit.edu Message-ID: <931614f1-f874-4249-bf79-367398654...@columbia.edu> Content-Type: text/plain; charset=us-ascii Chris, We didn't actually see any problems either until the KDC was under heavy load. The unpatched version of 1.9.1 was and still is running on our secondary KDC without issue, and we had been using 1.9.1 in testing and development for months without issue as well. During the period where we saw the performance degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which means the KDC was handling roughly 43 requests per second (not counting lots of retransmits). That is typical of our primary production KDC's workload throughout the day, but we don't have any other KDC that gets that amount of traffic; by contrast, our secondary KDC gets a request once or twice a minute. So it would seem the performance problem only really comes into play when the KDC is under heavy load. Jonathan On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote: > > Just another data point: I'm not seeing this on my locally built (but > not with the attached patch) 1.9.1: > > real 0m41.409s > user 0m3.358s > sys 0m3.683s > finished round 1 > > real 0m35.036s > user 0m3.441s > sys 0m3.658s > finished round 2 > > real 0m44.344s > user 0m3.363s > sys 0m3.728s > finished round 3 > > real 0m40.930s > user 0m3.465s > sys 0m3.973s > finished round 4 > > I had to reduce the number of inner iterations to 300 because my machine > is slow. The variance in the above numbers is because there's a bunch > of stuff running on this machine. > > Chris > > On 2011/08/08 11:21, Greg Hudson wrote: >> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote: >>> I did some performance testing on our test KDC and was able to >>> reproduce the performance issue with 1.9.1. >> >> I found a regression which would affect these tests, but I'm not sure it >> accounts for your global performance issues. >> >> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache, >> but due to a bug, it is actually opening and reading a replay cache for >> every TGS request, which is significantly less efficient than the 1.8 >> behavior (using a replay cache which stays open for the lifetime of the >> KDC). >> >> In a test which runs in under five minutes, this regression produces >> visible O(n^2) performance characteristics. This would not necessarily >> account for performance degradation over hours, as the performance drag >> of the replay cache should become stable after five minutes. It's >> possible that the constant drag was enough to cause the KDC to fall >> behind on the request load, but it's also possible that there's a second >> problem which isn't so easily reproduced. >> >> I've attached a patch. Note that there is a second, in-memory >> "lookaside" cache with O(n^2) performance characteristics in the short >> term, which holds queries for up to two minutes. You may see a slight >> degradation in performance in test cases due to this. You can >> temporarily rebuild the kdc directory with "make clean; >> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your >> performance tests. >> >> >> >> >> ________________________________________________ >> Kerberos mailing list Kerberos@mit.edu >> https://mailman.mit.edu/mailman/listinfo/kerberos > ________________________________________________ > Kerberos mailing list Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos > ------------------------------ Message: 3 Date: Tue, 9 Aug 2011 12:21:37 +0000 (UTC) From: Sanket Sangodkar <sanket.sangod...@atos.net> Subject: Problem after adding new realm in krb5.conf To: kerberos@mit.edu Message-ID: <loom.20110809t140429-...@post.gmane.org> Content-Type: text/plain; charset=us-ascii Hi, We have configured SSO using Kerberos with Apache Http Server. The realms name are defined in krb5.conf. We needed to add one more domain to krb5.conf file - We have inserted new realm as - xxx.xxx.COM = { kdc = <dns_name> admin_server = <dns_name> } But after executing kinit command for respective domain xxx.xxx.COM it specifies follwing error message - kinit: Client not found in Kerberos database while getting initial credentials We wanted to know whether only specifying new name under [realm] content will add new domain/realm to Kerberos configuration. Or are we missing any more configuration to add new realm ? Regards, Sanket ------------------------------ Message: 4 Date: Tue, 9 Aug 2011 17:57:25 +0530 From: "Sangodkar, Sanket" <sanket.sangod...@atos.net> Subject: Error while adding new realm to krb5.conf To: <kerberos@mit.edu> Message-ID: <dc18c77d955e0447bccbfbd690781b99f87...@inicx002.in.int.atosorigin.com> Content-Type: text/plain; charset="us-ascii" Hi, We have configured SSO using Kerberos with Apache Http Server. The realms name are defined in krb5.conf. We needed to add one more domain to krb5.conf file - We have inserted new realm as - xxx.xxx.COM = { kdc = <dns_name> admin_server = <dns_name> } But after executing kinit command for respective domain xxx.xxx.COM it specifies following error message - kinit: Client not found in Kerberos database while getting initial credentials We wanted to know whether only specifying new name under [realm] content will add new domain/realm to Kerberos configuration. Or are we missing any more configuration to add new realm ? Regards, Sanket This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. ------------------------------ Message: 5 Date: Wed, 10 Aug 2011 00:20:32 +0530 From: Vipin Rathor <v.rat...@gmail.com> Subject: Re: Error while adding new realm to krb5.conf To: "Sangodkar, Sanket" <sanket.sangod...@atos.net> Cc: kerberos@mit.edu Message-ID: <CAN-7Vpkoi+yoeyKWTcVGOcbO6W93rXewrtvLzYPqTsvB=gy...@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 > kinit: Client not found in Kerberos database while getting initial > credentials Are you sure that you are using correct username (for kinit) which exist in the new realm? Please make sure that: 1. you are specifying correct username + realm name for getting ticket. (e.g. kinit u...@newrealm.com) 2. The new KDC & kadmin servers are reachable. > Or are we missing any more configuration to add new realm ? Can't think of any more configuration that is required. On Tue, Aug 9, 2011 at 5:57 PM, Sangodkar, Sanket <sanket.sangod...@atos.net> wrote: > > > Hi, > > > > We have configured SSO using Kerberos with Apache Http Server. > > The realms name are defined in krb5.conf. > > We needed to add one more domain to krb5.conf file - > > > > We have inserted new realm as - > > xxx.xxx.COM = { > > ? ? ? ? ? ? ? ?kdc = <dns_name> > > ? ? ? ? ? ? ? ?admin_server = <dns_name> > > ? ? ? ?} > > > > But after executing kinit command for respective domain xxx.xxx.COM it > > specifies following error message - > > kinit: Client not found in Kerberos database while getting initial > credentials > > > > We wanted to know whether only specifying new name under [realm] content > will > > add new domain/realm to Kerberos configuration. > > Or are we missing any more configuration to add new realm ? > > > > Regards, > > Sanket > > > > > > > > This e-mail and the documents attached are confidential and intended solely > for the addressee; it may also be privileged. If you receive this e-mail in > error, please notify the sender immediately and destroy it. As its integrity > cannot be secured on the Internet, the Atos group liability cannot be > triggered for the message content. Although the sender endeavors to maintain > a computer virus-free network, the sender does not warrant that this > transmission is virus-free and will not be liable for any damages resulting > from any virus transmitted. > ________________________________________________ > Kerberos mailing list ? ? ? ? ? Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos > -- -Rathor ------------------------------ Message: 6 Date: Tue, 09 Aug 2011 12:39:46 -0700 From: Chris Hecker <chec...@d6.com> Subject: Re: Performance issues with krb5-1.9.1 To: kerberos@mit.edu Message-ID: <4e418d02.7070...@d6.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Ah, yeah, my tests had krb5kdc at about 50% of one core (slapd was an additional 15%), but it wasn't completely saturating the machine. Glad the patch fixed it! Chris On 2011/08/09 07:13, Jonathan Reams wrote: > Chris, > > We didn't actually see any problems either until the KDC was under heavy > load. The unpatched version of 1.9.1 was and still is running on our > secondary KDC without issue, and we had been using 1.9.1 in testing and > development for months without issue as well. During the period where we saw > the performance degradation, the primary KDC handled 467000 distinct AS/TGS > requests. Which means the KDC was handling roughly 43 requests per second > (not counting lots of retransmits). That is typical of our primary production > KDC's workload throughout the day, but we don't have any other KDC that gets > that amount of traffic; by contrast, our secondary KDC gets a request once or > twice a minute. So it would seem the performance problem only really comes > into play when the KDC is under heavy load. > > Jonathan > > On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote: > >> >> Just another data point: I'm not seeing this on my locally built (but >> not with the attached patch) 1.9.1: >> >> real 0m41.409s >> user 0m3.358s >> sys 0m3.683s >> finished round 1 >> >> real 0m35.036s >> user 0m3.441s >> sys 0m3.658s >> finished round 2 >> >> real 0m44.344s >> user 0m3.363s >> sys 0m3.728s >> finished round 3 >> >> real 0m40.930s >> user 0m3.465s >> sys 0m3.973s >> finished round 4 >> >> I had to reduce the number of inner iterations to 300 because my machine >> is slow. The variance in the above numbers is because there's a bunch >> of stuff running on this machine. >> >> Chris >> >> On 2011/08/08 11:21, Greg Hudson wrote: >>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote: >>>> I did some performance testing on our test KDC and was able to >>>> reproduce the performance issue with 1.9.1. >>> >>> I found a regression which would affect these tests, but I'm not sure it >>> accounts for your global performance issues. >>> >>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache, >>> but due to a bug, it is actually opening and reading a replay cache for >>> every TGS request, which is significantly less efficient than the 1.8 >>> behavior (using a replay cache which stays open for the lifetime of the >>> KDC). >>> >>> In a test which runs in under five minutes, this regression produces >>> visible O(n^2) performance characteristics. This would not necessarily >>> account for performance degradation over hours, as the performance drag >>> of the replay cache should become stable after five minutes. It's >>> possible that the constant drag was enough to cause the KDC to fall >>> behind on the request load, but it's also possible that there's a second >>> problem which isn't so easily reproduced. >>> >>> I've attached a patch. Note that there is a second, in-memory >>> "lookaside" cache with O(n^2) performance characteristics in the short >>> term, which holds queries for up to two minutes. You may see a slight >>> degradation in performance in test cases due to this. You can >>> temporarily rebuild the kdc directory with "make clean; >>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your >>> performance tests. >>> >>> >>> >>> >>> ________________________________________________ >>> Kerberos mailing list Kerberos@mit.edu >>> https://mailman.mit.edu/mailman/listinfo/kerberos >> ________________________________________________ >> Kerberos mailing list Kerberos@mit.edu >> https://mailman.mit.edu/mailman/listinfo/kerberos >> > > > ________________________________________________ > Kerberos mailing list Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos > ------------------------------ _______________________________________________ Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos End of Kerberos Digest, Vol 104, Issue 11 ***************************************** ________________________________________________ Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos