Re: [Freeipa-users] Slow ipa performance -- why so many ldap lookups ?
On Tue, Mar 19, 2013 at 11:05:14PM +0100, Jan-Frode Myklebust wrote: On Tue, Mar 19, 2013 at 10:01:16PM +0100, Jakub Hrozek wrote: Hello Jan, I'm sorry you're seeing performance problems. We have been struggeling with performance and crashes for a while now. Have had one crash were a replica dropped it's whole database, and a couple of hangs probably caused by clients flooding 389ds. (ref ticket 00799719 and 00800931).. You should really use the ipa backend for better performance as it uses the memberof attribute (and a couple of other shortcuts to be able to tell if a missing member is a user or a group based on the format of the DN for example). Sure, we very much want to use the IPA backend, but the LDAP backend seems to have been working better for us. More robust. It might have been caused by one of the ipa-servers running with too high error-log-level, but we've too often seen users not getting their groups populated with the IPA backend -- while this has never happened with LDAP backend. We fixed the error-log-level today, and have moved our lab-servers over to ipa-backend. Will see in a few days if the problem is fixed now. This really sounds like a bug. If you encounter a situation like this, where a group does not show all its members, feel free to open a bug. What we find very strange in the trace is: - how many ldap searches are done (144!) The number really depends on the group structure and nesting levels. We have a few nested groups.. but sgallagh explained to me that this large number of lookups was caused by me testing using id, which calls 'initgroups()' followed by a loop of 'getgrgid()' for every group you are a member of and the getgrgid() needs to fetch all members of each group. Yes, on the other hand, with a cold cache, the SSSD should be able to just download all group members in one go using the dereference control. I'm not completely sure, but I would presume even this dereference lookup would show up as a single search in wireshark, not look them up individually. - that nesting is handled by the client, instead of using memberOf. I'm sorry, I don't quite understand the problem here. If ipa backend was used, then all groups would be resolved in a single search by fetching the objects the memberof attribute points at. I was expecting the same to be done with RFC2307bis, but found no way of telling it to use memberof instead of un-nesting all groups by itself. On the other hand, the RFC2307bis schema does not guarantee there is a memberof attribute at all, so the client has to perform multiple queries based on the member attribute. This is one of the prime reasons to stick to the ipa backend as opposed to the LDAP back end with the RFC2307bis schema. Ok. - that all group members are searched individually, and multiple times if they're members of multiple groups They shouldn't be fetched multiple times, sounds like a bug to me. How did you measure this metric? Wireshark lookups? Wireshark lookups. Ref. trace attached to my previous message. Yes, sorry, I saw the attachment after I sent my initial reply :-) Can you tell us a little bit about your nesting structure? How many users, how many groups, how deep is the nesting? 305 users. 58 groups. I belive there are max 2 levels of nesting group1(group2(group3)). By the way, the id command is not really a fair benchmark as, contrary to the initgroups() operation that happens during a login, also fetches all the group members. If you are seeing slow logins, then the best way to benchmark the initgroups is id -G, not id. sudo will typically hang for many seconds before giving the password-prompt, and this delay seems to have been approximately the same as the delay we see with id. Guess that's why I found it to be a good benchmark for the performance problems we see. OK, sudo might need the group members, fair point. I was thinking logins previously. Bottom line, if you are seeing inconsistent results with ipa backend, please open a bug. This is something that would need fixing right away. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Slow ipa performance -- why so many ldap lookups ?
On Wed, Mar 20, 2013 at 10:44:10AM +0100, Jakub Hrozek wrote: This really sounds like a bug. If you encounter a situation like this, where a group does not show all its members, feel free to open a bug. I have been experiencing this for quite some time, but I'm struggeling with how to give useful bugreports. Right now I tested a ssh-login to one of my ipa servers and failed to log in: Mar 20 12:55:13 ipa1 sshd[16112]: pam_access(sshd:account): access denied for user `janfrode' from `login2.example.net' then I immediatelty try again, and can successfully log in. The reason for pam_access denying access is most likely that my groups isn't populated on the first try, but on the second it works. I don't seem able to re-produce this issue by stopping/clearing/starting sssd, so I suspect it might be the connection between sssd and 389ds that has been broken by firewalls between them maybe. We have an evil firewall that breaks connections that's been idle for more than 30 minutes. Are there hearbeat or keepalive settings in IPA or 389ds that we should enable to keep connections alive ? Bottom line, if you are seeing inconsistent results with ipa backend, please open a bug. This is something that would need fixing right away. Don't know if I can call it inconsistent results with ipa backend, or just bad broken connection handling within sssd. Any hints for how I can provide better bugreports would be appreciated.. -jf ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Slow ipa performance -- why so many ldap lookups ?
On Wed, Mar 20, 2013 at 02:04:24PM +0100, Jan-Frode Myklebust wrote: On Wed, Mar 20, 2013 at 10:44:10AM +0100, Jakub Hrozek wrote: This really sounds like a bug. If you encounter a situation like this, where a group does not show all its members, feel free to open a bug. I have been experiencing this for quite some time, but I'm struggeling with how to give useful bugreports. Right now I tested a ssh-login to one of my ipa servers and failed to log in: Mar 20 12:55:13 ipa1 sshd[16112]: pam_access(sshd:account): access denied for user `janfrode' from `login2.example.net' then I immediatelty try again, and can successfully log in. The reason for pam_access denying access is most likely that my groups isn't populated on the first try, but on the second it works. I don't seem able to re-produce this issue by stopping/clearing/starting sssd, so I suspect it might be the connection between sssd and 389ds that has been broken by firewalls between them maybe. We have an evil firewall that breaks connections that's been idle for more than 30 minutes. Ah, I see. The SSSD *should* reconnect in that case, though. Are there hearbeat or keepalive settings in IPA or 389ds that we should enable to keep connections alive ? Bottom line, if you are seeing inconsistent results with ipa backend, please open a bug. This is something that would need fixing right away. Don't know if I can call it inconsistent results with ipa backend, or just bad broken connection handling within sssd. Any hints for how I can provide better bugreports would be appreciated.. I think pasting or attaching SSSD logs would be a good start. Can you put debug_level = 6 into your sssd.conf into the [pam] and [domain] sections restart the sssd and then attach /var/log/sssd/sssd_pam.log and /var/log/sssd/sssd_$domain.log ? ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] error setting up replication client
I'm trying to set up an ipa replica, and each time I try the install process fails at the same point. When I look in the ipareplica-install.log I see a 302 redirection which seems to be causing the issue. Any ideas why this is happening (or if something else is the issue)? Thanks -Patrick (http://fpaste.org/gbYz/) 2013-03-15T17:19:50Z DEBUG stderr= 2013-03-15T17:19:50Z DEBUG duration: 5 seconds 2013-03-15T17:19:50Z DEBUG [3/17]: configuring certificate server instance 2013-03-15T17:19:51Z DEBUG args=/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname i-d1579ba3.ipa-server.us-east-1.cloud.com -cs_port 9445 -client_certdb_dir /tmp/tmp-2l64F1 -client_certdb_pw d -preop_pin IWk44JzZT6A78Pha3SrM -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa - agent_cert_subject CN=ipa-ca-agent,O=CLOUD.COM -ldap_host i-d1579ba3.ipa-server.us-east-1.cloud.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password -base_dn o=ipaca -db_name ip aca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=CLOUD .COM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=CLOUD.COM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=CLOUD.COM -ca_server_cert_subject_name CN=i-d1579ba3.ipa-server.us-east-1.cloud.com,O= CLOUD.COM -ca_audit_signing_cert_subject_name CN=CA Audit,O=CLOUD.COM -ca_sign_cert_subject_name CN=Certificate Authority,O=CLOUD.COM -external false -clone true -clone_p12_file ca.p12 -clone_p12_pa ssword -sd_hostname i-6775b715.ipa-server.us-east-1.cloud.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password -clone_start_tls true -clone_uri https://i-6775b715.ipa-ser ver.us-east-1.cloud.com:443 2013-03-15T17:19:51Z DEBUG stdout=libpath=/usr/lib64 ### CRYPTO INIT WITH CERTDB:/tmp/tmp-2l64F1 tokenpwd: # Attempting to connect to: i-d1579ba3.ipa-server.us-east-1.cloud.com:9445 in TestCertApprovalCallback.approve() Peer cert details: subject: CN=i-d1579ba3.ipa-server.us-east-1.cloud.com,O=CLOUD.COM issuer: CN=Certificate Authority,O=CLOUD.COM serial: 3 item 1 reason=-8172 depth=1 cert details: subject: CN=Certificate Authority,O=CLOUD.COM issuer: CN=Certificate Authority,O=CLOUD.COM serial: 1 importing certificate. Connected. Posting Query = https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445//ca/admin/console/config/login?pin=IWk44JzZT6A78Pha3SrMxml=true RESPONSE STATUS: HTTP/1.1 200 OK RESPONSE HEADER: Server: Apache-Coyote/1.1 RESPONSE HEADER: Content-Type: text/html;charset=UTF-8 RESPONSE HEADER: Content-Length: 0 RESPONSE HEADER: Date: Fri, 15 Mar 2013 17:19:51 GMT RESPONSE HEADER: Connection: keep-alive xml returned: # Attempting to connect to: i-d1579ba3.ipa-server.us-east-1.cloud.com:9445 Connected. Posting Query = https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445//ca/admin/console/config/wizard?p=0op=nextxml=true RESPONSE STATUS: HTTP/1.1 302 Moved Temporarily RESPONSE HEADER: Server: Apache-Coyote/1.1 RESPONSE HEADER: Set-Cookie: JSESSIONID=A8B36AB92F386DB22B193215907C01AC; Path=/ca; Secure RESPONSE HEADER: Location: https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445/ca/admin/console/config/login RESPONSE HEADER: Content-Type: text/html;charset=UTF-8 RESPONSE HEADER: Content-Length: 0 RESPONSE HEADER: Date: Fri, 15 Mar 2013 17:19:51 GMT RESPONSE HEADER: Connection: keep-alive ERROR: unable to parse xml ERROR XML = ERROR: Tag=statushas no values Error in LoginPanel(): status value is null ERROR: ConfigureCA: LoginPanel() failure ERROR: unable to create CA ### 2013-03-15T17:19:51Z DEBUG stderr=[Fatal Error] :-1:-1: Premature end of file. org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.DOMParser.parse(DOMParser.java:239) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at ParseXML.parse(ParseXML.java:43) at ConfigureCA.getStatus(ConfigureCA.java:205) at ConfigureCA.checkStatus(ConfigureCA.java:221) at ConfigureCA.checkStatus(ConfigureCA.java:216) at ConfigureCA.LoginPanel(ConfigureCA.java:261) at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1157) at ConfigureCA.main(ConfigureCA.java:1672) 2013-03-15T17:19:51Z CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname i-d1579ba3.ipa-server.us-east-1.cloud.com -cs_port 9445 -client_certdb_dir
Re: [Freeipa-users] error setting up replication client
Patrick, Can you provide some log files? Looks like pkisilent is trying to get to the first configuration panel on the CA and is getting a 302. I would need to see the logs under /var/log/pki-ca for the replica subsystem. Thanks, Ade Lee On Wed, 2013-03-20 at 12:04 -0400, Patrick Hemmer wrote: I'm trying to set up an ipa replica, and each time I try the install process fails at the same point. When I look in the ipareplica-install.log I see a 302 redirection which seems to be causing the issue. Any ideas why this is happening (or if something else is the issue)? Thanks -Patrick (http://fpaste.org/gbYz/) 2013-03-15T17:19:50Z DEBUG stderr= 2013-03-15T17:19:50Z DEBUG duration: 5 seconds 2013-03-15T17:19:50Z DEBUG [3/17]: configuring certificate server instance 2013-03-15T17:19:51Z DEBUG args=/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname i-d1579ba3.ipa-server.us-east-1.cloud.com -cs_port 9445 -client_certdb_dir /tmp/tmp-2l64F1 -client_certdb_pw d -preop_pin IWk44JzZT6A78Pha3SrM -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa - agent_cert_subject CN=ipa-ca-agent,O=CLOUD.COM -ldap_host i-d1579ba3.ipa-server.us-east-1.cloud.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password -base_dn o=ipaca -db_name ip aca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=CLOUD .COM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=CLOUD.COM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=CLOUD.COM -ca_server_cert_subject_name CN=i-d1579ba3.ipa-server.us-east-1.cloud.com,O= CLOUD.COM -ca_audit_signing_cert_subject_name CN=CA Audit,O=CLOUD.COM -ca_sign_cert_subject_name CN=Certificate Authority,O=CLOUD.COM -external false -clone true -clone_p12_file ca.p12 -clone_p12_pa ssword -sd_hostname i-6775b715.ipa-server.us-east-1.cloud.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password -clone_start_tls true -clone_uri https://i-6775b715.ipa-ser ver.us-east-1.cloud.com:443 2013-03-15T17:19:51Z DEBUG stdout=libpath=/usr/lib64 ### CRYPTO INIT WITH CERTDB:/tmp/tmp-2l64F1 tokenpwd: # Attempting to connect to: i-d1579ba3.ipa-server.us-east-1.cloud.com:9445 in TestCertApprovalCallback.approve() Peer cert details: subject: CN=i-d1579ba3.ipa-server.us-east-1.cloud.com,O=CLOUD.COM issuer: CN=Certificate Authority,O=CLOUD.COM serial: 3 item 1 reason=-8172 depth=1 cert details: subject: CN=Certificate Authority,O=CLOUD.COM issuer: CN=Certificate Authority,O=CLOUD.COM serial: 1 importing certificate. Connected. Posting Query = https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445//ca/admin/console/config/login?pin=IWk44JzZT6A78Pha3SrMxml=true RESPONSE STATUS: HTTP/1.1 200 OK RESPONSE HEADER: Server: Apache-Coyote/1.1 RESPONSE HEADER: Content-Type: text/html;charset=UTF-8 RESPONSE HEADER: Content-Length: 0 RESPONSE HEADER: Date: Fri, 15 Mar 2013 17:19:51 GMT RESPONSE HEADER: Connection: keep-alive xml returned: # Attempting to connect to: i-d1579ba3.ipa-server.us-east-1.cloud.com:9445 Connected. Posting Query = https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445//ca/admin/console/config/wizard?p=0op=nextxml=true RESPONSE STATUS: HTTP/1.1 302 Moved Temporarily RESPONSE HEADER: Server: Apache-Coyote/1.1 RESPONSE HEADER: Set-Cookie: JSESSIONID=A8B36AB92F386DB22B193215907C01AC; Path=/ca; Secure RESPONSE HEADER: Location: https://i-d1579ba3.ipa-server.us-east-1.cloud.com:9445/ca/admin/console/config/login RESPONSE HEADER: Content-Type: text/html;charset=UTF-8 RESPONSE HEADER: Content-Length: 0 RESPONSE HEADER: Date: Fri, 15 Mar 2013 17:19:51 GMT RESPONSE HEADER: Connection: keep-alive ERROR: unable to parse xml ERROR XML = ERROR: Tag=statushas no values Error in LoginPanel(): status value is null ERROR: ConfigureCA: LoginPanel() failure ERROR: unable to create CA ### 2013-03-15T17:19:51Z DEBUG stderr=[Fatal Error] :-1:-1: Premature end of file. org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.DOMParser.parse(DOMParser.java:239) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at ParseXML.parse(ParseXML.java:43) at ConfigureCA.getStatus(ConfigureCA.java:205) at ConfigureCA.checkStatus(ConfigureCA.java:221) at
Re: [Freeipa-users] Mail Challenge Password Reset
We have a POC with PWM and a testIPA server running freeIPA v2.2.0. It is working very well and we plan to move it into production soon. I haven't written a how-to, but I have several notes on setting this up. What part of PWM are you having trouble with? -Mike - Original Message - From: John Moyer To: freeipa-users@redhat.com Sent: Tuesday, March 19, 2013 4:25 PM Subject: [Freeipa-users] Mail Challenge Password Reset Is there a mail challenge 3rd party tool that allows for users to change their own passwords if they don't know their password? Something like PWM for LDAP? https://code.google.com/p/pwm/ I've been looking around and no one seems to have done this yet, but wanted to yield to this group before giving up hope. Thanks, _ John Moyer ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Mail Challenge Password Reset
On 03/20/2013 07:23 PM, Michael ORourke wrote: We have a POC with PWM and a testIPA server running freeIPA v2.2.0. It is working very well and we plan to move it into production soon. I haven't written a how-to, but I have several notes on setting this up. What part of PWM are you having trouble with? I would be really awesome if you find a moment to write a HOWTO on the subj. Thanks Dmitri -Mike - Original Message - From: John Moyer To: freeipa-users@redhat.com Sent: Tuesday, March 19, 2013 4:25 PM Subject: [Freeipa-users] Mail Challenge Password Reset Is there a mail challenge 3rd party tool that allows for users to change their own passwords if they don't know their password? Something like PWM for LDAP? https://code.google.com/p/pwm/ I've been looking around and no one seems to have done this yet, but wanted to yield to this group before giving up hope. Thanks, _ John Moyer ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Mail Challenge Password Reset
On Wed, Mar 20, 2013 at 6:23 PM, Michael ORourke mrorou...@earthlink.net wrote: We have a POC with PWM and a testIPA server running freeIPA v2.2.0. It is working very well and we plan to move it into production soon. I haven't written a how-to, but I have several notes on setting this up. What part of PWM are you having trouble with? It's been a while, but IIRC when a user would request a reset via pwm and then set their password, it would require a further change because changing it through PWM was as-if an admin had done so. Something like that. Like I said, I didn't test that long with it. Like Dmitri said, if you could share your notes or write up a how-to the community would certainly appreciate it. Thanks, --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Mail Challenge Password Reset
On Wed, 2013-03-20 at 19:35 -0500, KodaK wrote: On Wed, Mar 20, 2013 at 6:23 PM, Michael ORourke mrorou...@earthlink.net wrote: We have a POC with PWM and a testIPA server running freeIPA v2.2.0. It is working very well and we plan to move it into production soon. I haven't written a how-to, but I have several notes on setting this up. What part of PWM are you having trouble with? It's been a while, but IIRC when a user would request a reset via pwm and then set their password, it would require a further change because changing it through PWM was as-if an admin had done so. You should have given the pwm user 'password sync' privileges. See this: http://www.freeipa.org/page/PasswordSynchronization Something like that. Like I said, I didn't test that long with it. Like Dmitri said, if you could share your notes or write up a how-to the community would certainly appreciate it. It would be very nice. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
