[Freeipa-users] dirsrv hangs, 0% CPU util
Hi! Today we started having problems with dirsrv hanging. We have observed the following symptoms (using EXAMPLE.COM instead of the real domain): /var/log/dirsrv/slapd-EXAMPLE-COM/errors: [15/Feb/2015:21:48:50 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected) [15/Feb/2015:21:48:50 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP server) /var/log/messages: Feb 15 21:49:02 ipa named[5545]: LDAP query timed out. Try to adjust timeout parameter Feb 15 21:49:03 ipa named[5545]: LDAP query timed out. Try to adjust timeout parameter (repeated) Trying to access the DS also with ldapsearch just hangs: ldapsearch -h localhost -x dc=example,dc=com And Kerberos is unavailable as well: # KRB5_TRACE=/dev/stdout kinit admin [6421] 1424029967.466519: Getting initial credentials for ad...@example.com [6421] 1424029967.467202: Sending request (172 bytes) to EXAMPLE.COM [6421] 1424029967.467736: Sending initial UDP request to dgram 10.1.1.1:88 [6421] 1424029968.469031: Initiating TCP connection to stream 10.1.1.1:88 [6421] 1424029968.469205: Sending TCP request to stream 10.1.1.1:88 [6421] 1424029971.472024: Sending retry UDP request to dgram 10.1.1.1:88 [6421] 1424029976.477340: Sending retry UDP request to dgram 10.1.1.1:88 kinit: Cannot contact any KDC for realm 'EXAMPLE.COM' while getting initial credentials Strange thing is that there is hardly any CPU utilization when the problem is occurring. In addition we have started to see the following entries in /var/log/messages: Feb 15 21:37:27 ipa kernel: possible SYN flooding on port 88. Sending cookies. Feb 15 21:39:37 ipa kernel: possible SYN flooding on port 88. Sending cookies. I'm not sure if this is related, but it's something we haven't seen before. We are running CentOS release 6.6 (Final) with the latest available packages: 389-ds-base-libs-1.2.11.15-48.el6_6.x86_64 389-ds-base-1.2.11.15-48.el6_6.x86_64 ipa-client-3.0.0-42.el6.centos.x86_64 ipa-server-selinux-3.0.0-42.el6.centos.x86_64 libipa_hbac-1.11.6-30.el6_6.3.x86_64 sssd-ipa-1.11.6-30.el6_6.3.x86_64 ipa-admintools-3.0.0-42.el6.centos.x86_64 ipa-python-3.0.0-42.el6.centos.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-server-3.0.0-42.el6.centos.x86_64 libipa_hbac-python-1.11.6-30.el6_6.3.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch krb5-workstation-1.10.3-33.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 sssd-krb5-common-1.11.6-30.el6_6.3.x86_64 python-krbV-1.0.90-3.el6.x86_64 krb5-server-1.10.3-33.el6.x86_64 sssd-krb5-1.11.6-30.el6_6.3.x86_64 pam_krb5-2.3.11-9.el6.x86_64 Killing the dirsrv processes and restarting them resolves the issue - until it happens again after about 15 minutes. Any idea what could have gone wrong? I can e-mail logs, if necessary. Thank you in advance! Best regards, Thomas -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] dirsrv hangs, 0% CPU util
Hi! On Sun, Feb 15, 2015 at 11:37 PM, Rich Megginson rmegg...@redhat.com wrote: Today we started having problems with dirsrv hanging. We have observed the following symptoms (using EXAMPLE.COM instead of the real domain): see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs Thanks! Please find the stacktrace attached. Best regards, Thomas stacktrace.1424039830.txt.gz Description: GNU Zip compressed data -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] dirsrv hangs, 0% CPU util
On 02/15/2015 03:41 PM, Thomas Raehalme wrote: Hi! On Sun, Feb 15, 2015 at 11:37 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: Today we started having problems with dirsrv hanging. We have observed the following symptoms (using EXAMPLE.COM http://EXAMPLE.COM instead of the real domain): see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs Thanks! Please find the stacktrace attached. Some info missing. Since this is IPA, you'll need some additional debuginfo packages: debuginfo-install ipa-server slapi_nis (or maybe it's slapi-nis) Also looks as though the nspr debuginfo does not match the nspr version. rpm -q nspr nspr-debuginfo Finally, do some stacktraces every couple of seconds over a period of a minute. For example, is the server really hung at the poll() in thread 32, or will the poll() eventually return write ready and proceed? Best regards, Thomas -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] dirsrv hangs, 0% CPU util
On Mon, Feb 16, 2015 at 12:57 AM, Rich Megginson rmegg...@redhat.com wrote: Some info missing. Since this is IPA, you'll need some additional debuginfo packages: debuginfo-install ipa-server slapi_nis (or maybe it's slapi-nis) Also looks as though the nspr debuginfo does not match the nspr version. rpm -q nspr nspr-debuginfo I installed only the minimum debuginfo packages mentioned on the link (with yum). Now I have added the ones you mentioned here. Finally, do some stacktraces every couple of seconds over a period of a minute. For example, is the server really hung at the poll() in thread 32, or will the poll() eventually return write ready and proceed? Will do. Unfortunately it'll probably not take too long until the next occurrence. Best regards, Thomas -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] dirsrv hangs, 0% CPU util
On Mon, 16 Feb 2015, Thomas Raehalme wrote: On Mon, Feb 16, 2015 at 1:04 AM, Thomas Raehalme thomas.raeha...@codecenter.fi wrote: Finally, do some stacktraces every couple of seconds over a period of a minute. For example, is the server really hung at the poll() in thread 32, or will the poll() eventually return write ready and proceed? Will do. Unfortunately it'll probably not take too long until the next occurrence. Here's a new set of stacktraces from a period of approx 1 minute. I hope the attachment is not too large. I suspect you've triggered https://fedorahosted.org/freeipa/ticket/4586 and https://fedorahosted.org/freeipa/ticket/4635 -- slapi-nis plugin configuration does not limit itself to $SUFFIX and listens to changes in cn=changelog too so it may deadlock with a replication traffic. We fixed these partly by changing slapi-nis configuration, partly by fixing bugs in 389-ds. I wonder if amending your slapi-nis config to avoid triggering internal searches on cn=changelog would be enough. If you have RHEL subscription, please open a case with Red Hat's support. -- / Alexander Bokovoy pgpXcPFjpJMge.pgp Description: PGP signature -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
