Thank you so much. It worked now, and now I can start ipa services. Thanks & Regards. ______________________________
-----Original Message----- From: Rob Crittenden <[email protected]> Sent: Friday, May 04, 2018 15:01 To: FreeIPA users list <[email protected]> Cc: Ludwig Krispenz <[email protected]>; thierry bordaz <[email protected]>; SOLER SANGUESA Miguel <[email protected]> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5) SOLER SANGUESA Miguel via FreeIPA-users wrote: > Hello, > > Thanks for your instructions, step to step seems it is improving. > Unfoutunately it is not solve yet, now I have this when I do an upgrade > following your instructions: > # ipa-server-upgrade > Upgrading IPA:. Estimated time: 1 minute 30 seconds > [1/8]: saving configuration > [2/8]: disabling listeners > [3/8]: enabling DS global lock > [4/8]: starting directory server > [5/8]: updating schema > [6/8]: upgrading server > [7/8]: stopping directory server > [8/8]: restoring configuration > Done. > Update complete > Upgrading IPA services > Upgrading the configuration of the IPA services > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command > ipa-server-upgrade manually. > Timeout exceeded > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more > information Ok, make sure dirsrv is not running (VERY important), then: 1. edit /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif 2. set nsslapd-port to 389 3. set nsslapd-security to on 4. set nsslapd-global-backend-lock to off (if you have this attribute at all) 5. ipa-server-upgrade rob > > > On ipaupgrade.log I have: > .... > 2018-05-04T06:54:01Z DEBUG wait_for_open_ports: localhost [389] timeout 300 > 2018-05-04T06:54:01Z DEBUG waiting for port: 389 > 2018-05-04T06:54:01Z DEBUG Failed to connect to port 389 tcp on 127.0.0.1 > 2018-05-04T06:59:01Z ERROR IPA server upgrade failed: Inspect > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > 2018-05-04T06:59:01Z DEBUG File > "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in > execute > return_value = self.run() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", > line 46, in run > server.upgrade() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line > 1919, in upgrade > upgrade_configuration() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line > 1586, in upgrade_configuration > ds.start(ds_serverid) > File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", > line 641, in start > super(DsInstance, self).start(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line > 401, in start > self.service.start(instance_name, capture_output=capture_output, > wait=wait) > File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", > line 161, in start > instance_name, capture_output=capture_output, wait=wait) > File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line > 300, in start > self.wait_for_open_ports(self.service_instance(instance_name)) > File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line > 270, in wait_for_open_ports > self.api.env.startup_timeout) > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1256, > in wait_for_open_ports > raise socket.timeout("Timeout exceeded") > > 2018-05-04T06:59:01Z DEBUG The ipa-server-upgrade command failed, exception: > timeout: Timeout exceeded > 2018-05-04T06:59:01Z ERROR Timeout exceeded > 2018-05-04T06:59:01Z ERROR The ipa-server-upgrade command failed. See > /var/log/ipaupgrade.log for more information > > And if I get the ports listening by ns-slapd, only upd port is listening: > # netstat -putan | grep 389 > udp6 0 0 :::389 :::* > 17469/ns-slapd > > Thanks & Regards. > ______________________________ > -----Original Message----- > From: Rob Crittenden <[email protected]> > Sent: Thursday, May 03, 2018 21:16 > To: thierry bordaz <[email protected]>; SOLER SANGUESA Miguel > <[email protected]>; FreeIPA users list <[email protected]> > Cc: Ludwig Krispenz <[email protected]> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 > (RHEL 7.4) to 4.5.4 (RHEL 7.5) > > thierry bordaz wrote: >> Hi Soler, >> >> Thanks for the information. >> So indexing is hanging because SC cache_init is running, the SC >> cache_init is hanging because SSSD is not started, SSSD is not started >> possibly because indexing prevents to get read access to the backend >> ("Backend is offline" TBC). >> >> An option would be to disable SC during upgrade phase ? Rob ? > > Sure, disabling SC would be a way to get things running again. > > To do this: > > 1. Be sure dirsrv is not running > 2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif > 3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config > 4. change nsslapd-pluginEnabled from on to off 5. try the upgrade again > > Assuming the upgrade is successful you can re-enable it post-install with: > > # ipa-compat-manage enable > > rob > >> >> regards >> thierry >> >> On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote: >>> Hello Thierry, >>> >>> The version is: slapi-nis-0.56.0-8.el7.x86_64 >>> >>> And the errors are: >>> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin - >>> group "[email protected]" does not exist because SSSD is offline. >>> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin - >>> waiting for SSSD to become online... >>> >>> SSSD service is up, but as the ipa services are not (all) running, >>> the ipa domain is down, logs when started sssd shows: >>> sssd[be[ipa.example.org]][823]: Backend is offline >>> >>> and logs sssd_nss.log & sssd_sudo.log have hundreds of: >>> [sss_dp_get_reply] (0x0010): The Data Provider returned an error >>> [org.freedesktop.sssd.Error.DataProvider.Offline] >>> >>> Thanks & Regards. >>> >>> -----Original Message----- >>> From: thierry bordaz <[email protected]> >>> Sent: Thursday, May 03, 2018 11:25 >>> To: SOLER SANGUESA Miguel <[email protected]>; Rob Crittenden >>> <[email protected]>; FreeIPA users list >>> <[email protected]> >>> Cc: Ludwig Krispenz <[email protected]> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5) >>> >>> me again ... >>> >>> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the >>> hanging period it could also indicate the reason what Schema compat >>> was waiting for (SSSD ?) >>> >>> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote: >>>> hello, >>>> >>>> Yesterday my ssh console closed the connection, so I had to start >>>> again the "ipa-server-upgrade", but the result is more or less the same: >>>> # ipa-server-upgrade >>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds >>>> [1/10]: stopping directory server >>>> [2/10]: saving configuration >>>> [3/10]: disabling listeners >>>> [4/10]: enabling DS global lock >>>> [5/10]: starting directory server >>>> [6/10]: updating schema >>>> [7/10]: upgrading server >>>> >>>> But now, the lines that are repeated on the access log are: >>>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH >>>> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config" >>>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode" >>>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0 >>>> tag=101 nentries=1 etime=1.0000687740 >>>> >>>> And it is on the same state than the other: >>>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket >>>> -b >>>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base >>>> SASL/EXTERNAL authentication started SASL username: >>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth >>>> SASL SSF: 0 >>>> # extended LDIF >>>> # >>>> # LDAPv3 >>>> # base >>>> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config> >>>> with scope baseObject # filter: (objectclass=*) # requesting: ALL # >>>> >>>> # indextask_l_137445500911864330_4055, index, tasks, config >>>> dn: >>>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config >>>> objectClass: top >>>> objectClass: extensibleObject >>>> nsindexattribute: l >>>> nsinstance: userRoot >>>> cn: indextask_l_137445500911864330_4055 >>>> nstaskcurrentitem: 0 >>>> nstasktotalitems: 1 >>>> >>>> # search result >>>> search: 3 >>>> result: 0 Success >>>> >>>> # numResponses: 2 >>>> # numEntries: 1 >>>> >>>> The pstack of the proces: /usr/bin/python2 >>>> /usr/sbin/ipa-server-upgrade # pstack 9805 >>>> #0 0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6 >>>> #1 0x00007fb8ff5e0000 in time_sleep () from >>>> /usr/lib64/python2.7/lib-dynload/timemodule.so >>>> #2 0x00007fb90730acf0 in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #3 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #4 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #5 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #6 0x00007fb90730d03d in PyEval_EvalCodeEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #7 0x00007fb90730a53c in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #8 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #9 0x00007fb90730d03d in PyEval_EvalCodeEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from >>>> /lib64/libpython2.7.so.1.0 >>>> #21 0x00007fb90730d142 in PyEval_EvalCode () from >>>> /lib64/libpython2.7.so.1.0 >>>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0 >>>> #23 0x00007fb90732773e in PyRun_FileExFlags () from >>>> /lib64/libpython2.7.so.1.0 >>>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from >>>> /lib64/libpython2.7.so.1.0 >>>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0 >>>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6 >>>> #27 0x000000000040066e in _start () >>>> >>>> >>>> And the pstack from the process: /usr/sbin/ns-slapd -D >>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i >>>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid >>>> # pstack 15536 >>>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49 >>>> (Thread 0x7f5bc02e8700 (LWP 15540)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48 >>>> (Thread 0x7f5bbfae7700 (LWP 15541)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47 >>>> (Thread 0x7f5bbf2e6700 (LWP 15542)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46 >>>> (Thread 0x7f5bbeae5700 (LWP 15543)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45 >>>> (Thread 0x7f5bbe2e4700 (LWP 15544)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44 >>>> (Thread 0x7f5bbdae3700 (LWP 15545)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43 >>>> (Thread 0x7f5bbd2e2700 (LWP 15546)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42 >>>> (Thread 0x7f5bbcae1700 (LWP 15547)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41 >>>> (Thread 0x7f5bbc2e0700 (LWP 15548)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40 >>>> (Thread 0x7f5bbbadf700 (LWP 15549)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39 >>>> (Thread 0x7f5bbb2de700 (LWP 15550)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38 >>>> (Thread 0x7f5bbaadd700 (LWP 15551)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37 >>>> (Thread 0x7f5bba2dc700 (LWP 15552)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36 >>>> (Thread 0x7f5bb9adb700 (LWP 15553)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35 >>>> (Thread 0x7f5bb92da700 (LWP 15554)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34 >>>> (Thread 0x7f5bb8ad9700 (LWP 15555)): >>>> #0 0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6 >>>> #1 0x00007f5bd1d9c803 in epoll_dispatch () at >>>> /lib64/libevent-2.0.so.5 >>>> #2 0x00007f5bd1d883ea in event_base_loop () at >>>> /lib64/libevent-2.0.so.5 >>>> #3 0x00007f5bd4d34eae in ns_event_fw_loop () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #4 0x00007f5bd4d33ac9 in event_loop_thread_func () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33 >>>> (Thread 0x7f5bb2b84700 (LWP 15559)): >>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6 >>>> #1 0x00007f5bd4acc599 in DS_Sleep () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc3e41e77 in deadlock_threadmain () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32 >>>> (Thread 0x7f5bb2383700 (LWP 15560)): >>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6 >>>> #1 0x00007f5bd4acc599 in DS_Sleep () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc3e45a87 in checkpoint_threadmain () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31 >>>> (Thread 0x7f5bb1b82700 (LWP 15561)): >>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6 >>>> #1 0x00007f5bd4acc599 in DS_Sleep () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc3e420c7 in trickle_threadmain () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30 >>>> (Thread 0x7f5bb1381700 (LWP 15562)): >>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6 >>>> #1 0x00007f5bd4acc599 in DS_Sleep () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc3e3ce47 in perf_threadmain () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29 >>>> (Thread 0x7f5bb0b80700 (LWP 15563)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f5bc920213e in cos_cache_wait_on_change () at >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28 >>>> (Thread 0x7f5bb037f700 (LWP 15564)): >>>> #0 0x00007f5bd189ae9d in poll () at /lib64/libc.so.6 >>>> #1 0x00007f5bc6d7e44c in ipa_cldap_worker () at >>>> /usr/lib64/dirsrv/plugins/libipa_cldap.so >>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27 >>>> (Thread 0x7f5bd5132700 (LWP 15566)): >>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so >>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #3 0x00007f5bc3572033 in _cl5TrimMain () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26 >>>> (Thread 0x7f5bafb7e700 (LWP 15567)): >>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so >>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #3 0x00007f5bc3587fa4 in protocol_sleep () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #4 0x00007f5bc358b75a in repl5_inc_run () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #5 0x00007f5bc358fbec in prot_thread_main () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #6 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #7 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #8 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25 >>>> (Thread 0x7f5baf37d700 (LWP 15568)): >>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so >>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #3 0x00007f5bc3587fa4 in protocol_sleep () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #4 0x00007f5bc358b75a in repl5_inc_run () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #5 0x00007f5bc358fbec in prot_thread_main () at >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so >>>> #6 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #7 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #8 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24 >>>> (Thread 0x7f5bae759700 (LWP 15569)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23 >>>> (Thread 0x7f5badf58700 (LWP 15570)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22 >>>> (Thread 0x7f5bad757700 (LWP 15571)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21 >>>> (Thread 0x7f5bacd4f700 (LWP 15572)): >>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so >>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #3 0x0000564591e10b08 in housecleaning () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20 >>>> (Thread 0x7f5bac54e700 (LWP 15573)): >>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so >>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #3 0x00007f5bd4a5ba63 in eq_loop () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19 >>>> (Thread 0x7f5babd4d700 (LWP 15574)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18 >>>> (Thread 0x7f5bab54c700 (LWP 15575)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17 >>>> (Thread 0x7f5baad4b700 (LWP 15576)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16 >>>> (Thread 0x7f5baa54a700 (LWP 15577)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15 >>>> (Thread 0x7f5ba9d49700 (LWP 15578)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14 >>>> (Thread 0x7f5ba9548700 (LWP 15579)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13 >>>> (Thread 0x7f5ba8d47700 (LWP 15580)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12 >>>> (Thread 0x7f5ba8546700 (LWP 15581)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11 >>>> (Thread 0x7f5ba7d45700 (LWP 15582)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10 >>>> (Thread 0x7f5ba7544700 (LWP 15583)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9 >>>> (Thread 0x7f5ba6d43700 (LWP 15584)): >>>> #0 0x00007f5bd189ae9d in poll () at /lib64/libc.so.6 >>>> #1 0x00007f5bd2854bc7 in _pr_poll_with_poll () at >>>> /lib64/libnspr4.so >>>> #2 0x0000564591e07049 in connection_read_operation () >>>> #3 0x0000564591e0765d in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8 >>>> (Thread 0x7f5ba6542700 (LWP 15585)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7 >>>> (Thread 0x7f5ba5d41700 (LWP 15586)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6 >>>> (Thread 0x7f5ba5540700 (LWP 15587)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5 >>>> (Thread 0x7f5ba4d3f700 (LWP 15588)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4 >>>> (Thread 0x7f5ba453e700 (LWP 15589)): >>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@@GLIBC_2.3.2 () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so >>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work >>>> () >>>> #3 0x0000564591e075d7 in connection_threadmain () >>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3 >>>> (Thread 0x7f5ba3900700 (LWP 15590)): >>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6 >>>> #1 0x00007f5bd4acc599 in DS_Sleep () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc1ea5c6e in backend_set_entry_from () at >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #3 0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #4 0x00007f5bd4aac614 in send_ldap_search_entry_ext () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f5bd4aace0c in send_ldap_search_entry () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f5bd4a87ca4 in send_entry.isra.0 () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #7 0x00007f5bd4a88208 in send_results_ext.constprop.4 () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #8 0x00007f5bd4a8a0d1 in op_shared_search () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #9 0x00007f5bd4a9b507 in search_internal_callback_pb () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () >>>> at /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so >>>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2 >>>> (Thread 0x7f5ba30ff700 (LWP 15606)): >>>> #0 0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f5bc3e921ac in instance_set_busy_and_readonly () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f5bd4ac5942 in task_index_thread () at >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so >>>> #6 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0 >>>> #7 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1 >>>> (Thread 0x7f5bd5135940 (LWP 15536)): >>>> #0 0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0 >>>> #1 0x00007f5bd4d34b2f in ns_thrpool_wait () at >>>> /usr/lib64/dirsrv/libnunc-stans.so.0 >>>> #2 0x0000564591e0d862 in slapd_daemon () >>>> #3 0x0000564591dfe86f in main () >>>> >>>> Thanks & Regards. >>>> ______________________________ >>>> >>>> -----Original Message----- >>>> From: thierry bordaz <[email protected]> >>>> Sent: Thursday, May 03, 2018 9:31 >>>> To: Rob Crittenden <[email protected]>; FreeIPA users list >>>> <[email protected]>; SOLER SANGUESA Miguel >>>> <[email protected]> >>>> Cc: Ludwig Krispenz <[email protected]> >>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating >>>> from >>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5) >>>> >>>> Hi, >>>> >>>> During indexing task we should see in the task status the periodic >>>> progression of the indexing. >>>> May be the indexing is hanging somewhere. When the problem occurs >>>> could you provide a pstack of the dirsrv server ? >>>> >>>> best regards >>>> thierry >>>> >>>> On 05/02/2018 10:27 PM, Rob Crittenden wrote: >>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote: >>>>>> Hello, >>>>>> >>>>>> This is the output of the command (seems that is not complete): >>>>>> >>>>>> # ldapsearch -H >>>>>> ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket >>>>>> -b >>>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks >>>>>> ,c >>>>>> n >>>>>> =config >>>>>> -s base >>>>>> SASL/EXTERNAL authentication started SASL username: >>>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth >>>>>> SASL SSF: 0 >>>>>> # extended LDIF >>>>>> # >>>>>> # LDAPv3 >>>>>> # base >>>>>> <cn=indextask_description_137444551994158920_5958,cn=index,cn=task >>>>>> s, >>>>>> c >>>>>> n=config> >>>>>> with scope baseObject >>>>>> # filter: (objectclass=*) >>>>>> # requesting: ALL >>>>>> # >>>>>> >>>>>> # indextask_description_137444551994158920_5958, index, tasks, >>>>>> config >>>>>> dn: >>>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks >>>>>> ,c >>>>>> n >>>>>> =conf >>>>>> ig >>>>>> objectClass: top >>>>>> objectClass: extensibleObject >>>>>> nsindexattribute: description >>>>>> nsinstance: userRoot >>>>>> cn: indextask_description_137444551994158920_5958 >>>>>> nstaskcurrentitem: 0 >>>>>> nstasktotalitems: 1 >>>>> IPA is looking for nstaskstatus and since that isn't there it is >>>>> looping forever. >>>>> >>>>> I've cc'd a couple of the 389-ds devs to see what they think about it. >>>>> >>>>> rob >>>>> >>>>>> # search result >>>>>> search: 3 >>>>>> result: 0 Success >>>>>> >>>>>> # numResponses: 2 >>>>>> # numEntries: 1 >>>>>> >>>>>> Thanks & Regards. >>>>>> ______________________________ >>>>>> -----Original Message----- >>>>>> From: Rob Crittenden <[email protected]> >>>>>> Sent: Tuesday, May 01, 2018 15:18 >>>>>> To: FreeIPA users list <[email protected]> >>>>>> Cc: SOLER SANGUESA Miguel <[email protected]> >>>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating >>>>>> from >>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5) >>>>>> >>>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote: >>>>>>> Hello, >>>>>>> >>>>>>> Thank you for your answer, now dirsrv can start, but after >>>>>>> running the " >>>>>>> [email protected] " it hungs: >>>>>>> >>>>>>> # ipa-server-upgrade >>>>>>> >>>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds >>>>>>> >>>>>>> [1/8]: saving configuration >>>>>>> >>>>>>> [2/8]: disabling listeners >>>>>>> >>>>>>> [3/8]: enabling DS global lock >>>>>>> >>>>>>> [4/8]: starting directory server >>>>>>> >>>>>>> [5/8]: updating schema >>>>>>> >>>>>>> [6/8]: upgrading server >>>>>>> >>>>>>> The latest messages on the log is (there are no logs on the lasts >>>>>>> 2h): >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG >>>>>>> --------------------------------------------- >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG dn: >>>>>>> cn=description,cn=index,cn=userroot,cn=ldbm >>>>>>> database,cn=plugins,cn=config >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG objectclass: >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG top >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG nsindex >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype: >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG eq >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG sub >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG cn: >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG description >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex: >>>>>>> >>>>>>> 2018-05-01T08:13:14Z DEBUG false >>>>>>> >>>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute: >>>>>>> description >>>>>>> >>>>>>> 2018-05-01T08:13:19Z DEBUG Task id: >>>>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=task >>>>>>> s, >>>>>>> c >>>>>>> n= >>>>>>> config >>>>>>> >>>>>>> If I check the ipa services, dirsrv is the only one running: >>>>>>> >>>>>>> # ipactl status >>>>>>> >>>>>>> Directory Service: RUNNING >>>>>>> >>>>>>> krb5kdc Service: STOPPED >>>>>>> >>>>>>> kadmin Service: STOPPED >>>>>>> >>>>>>> named Service: STOPPED >>>>>>> >>>>>>> httpd Service: STOPPED >>>>>>> >>>>>>> ipa-custodia Service: STOPPED >>>>>>> >>>>>>> ntpd Service: RUNNING >>>>>>> >>>>>>> pki-tomcatd Service: STOPPED >>>>>>> >>>>>>> smb Service: STOPPED >>>>>>> >>>>>>> winbind Service: STOPPED >>>>>>> >>>>>>> ipa-otpd Service: STOPPED >>>>>>> >>>>>>> ipa-dnskeysyncd Service: STOPPED >>>>>>> >>>>>>> ipa: INFO: The ipactl command was successful >>>>>>> >>>>>>> Should I stop the ipa-server-upgrade and start ipa services? >>>>>> It depends on how many entries you have. This is an indexing task >>>>>> and while 2 hours seems a bit excessive, it depends on how much >>>>>> work it has to do. >>>>>> >>>>>> The 389-ds access and/or error logs may provide details. >>>>>> >>>>>> rob >>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Alexander Bokovoy <[email protected]> >>>>>>> Sent: Tuesday, May 01, 2018 9:56 >>>>>>> To: FreeIPA users list <[email protected]> >>>>>>> Cc: SOLER SANGUESA Miguel <[email protected]> >>>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from >>>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5) >>>>>>> >>>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote: >>>>>>> >>>>>>> >hello, >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to >>>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the >>>>>>> unique replica I have, but after update dirsrv is not starting. >>>>>>> >>>>>>> >It says it is needed run "ipa-server-upgrade", but it also >>>>>>> fails: >>>>>>> >>>>>>> ># ipactl start >>>>>>> >>>>>>> >Upgrade required: please run ipa-server-upgrade command >>>>>>> Aborting ipactl >>>>>>> >>>>>>> > >>>>>>> >>>>>>> ># ipa-server-upgrade >>>>>>> >>>>>>> >Upgrading IPA:. Estimated time: 1 minute 30 seconds >>>>>>> >>>>>>> > [1/8]: saving configuration >>>>>>> >>>>>>> > [2/8]: disabling listeners >>>>>>> >>>>>>> > [3/8]: enabling DS global lock >>>>>>> >>>>>>> > [4/8]: starting directory server >>>>>>> >>>>>>> > [error] CalledProcessError: Command '/bin/systemctl start >>>>>>> >>>>>>> >[email protected] >>>>>>> <mailto:[email protected]>' returned non-zero exit >>>>>>> status >>>>>>> 1 >>>>>>> >>>>>>> > [cleanup]: stopping directory server >>>>>>> >>>>>>> > [cleanup]: restoring configuration >>>>>>> >>>>>>> >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log >>>>>>> and run command ipa-server-upgrade manually. >>>>>>> >>>>>>> >Unexpected error - see /var/log/ipaupgrade.log for details: >>>>>>> >>>>>>> >CalledProcessError: Command '/bin/systemctl start >>>>>>> >>>>>>> >[email protected] >>>>>>> <mailto:[email protected]>' returned non-zero exit >>>>>>> status >>>>>>> 1 The >>>>>>> >>>>>>> >ipa-server-upgrade command failed. See >>>>>>> /var/log/ipaupgrade.log for more >>>>>>> >>>>>>> >information >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >On the log I can see: >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG Starting external process >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active >>>>>>> >>>>>>> >[email protected] >>>>>>> <mailto:[email protected]> >>>>>>> 2018-04-30T14:36:15Z DEBUG Process >>>>>>> >>>>>>> >finished, return code=3 2018-04-30T14:36:15Z DEBUG >>>>>>> stdout=failed ... >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG [4/8]: starting directory >>>>>>> server >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG Starting external process >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start >>>>>>> >>>>>>> >[email protected] >>>>>>> <mailto:[email protected]> >>>>>>> 2018-04-30T14:36:15Z DEBUG Process >>>>>>> >>>>>>> >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout= >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG stderr=Job for >>>>>>> [email protected] >>>>>>> <mailto:[email protected]> >>>>>>> failed because the control process exited with error code. See >>>>>>> "systemctl status [email protected] >>>>>>> <mailto:[email protected]>" and "journalctl -xe" for >>>>>>> details. >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last): >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", >>>>>>> line 504, in start_creation >>>>>>> >>>>>>> > run_step(full_msg, method) >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", >>>>>>> line 494, in run_step >>>>>>> >>>>>>> > method() >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance. >>>>>>> py >>>>>>> ", >>>>>>> line 95, in __start >>>>>>> >>>>>>> > srv.start(self.serverid, ldapi=True) >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py" >>>>>>> , >>>>>>> line 161, in start >>>>>>> >>>>>>> > instance_name, capture_output=capture_output, wait=wait) >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", >>>>>>> line 294, in start >>>>>>> >>>>>>> > skip_output=not capture_output) >>>>>>> >>>>>>> > File >>>>>>> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", >>>>>>> line 542, in run >>>>>>> >>>>>>> > raise CalledProcessError(p.returncode, arg_string, >>>>>>> str(output)) >>>>>>> >>>>>>> >CalledProcessError: Command '/bin/systemctl start >>>>>>> >>>>>>> >[email protected] >>>>>>> <mailto:[email protected]>' returned non-zero exit >>>>>>> status >>>>>>> 1 >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >2018-04-30T14:36:15Z DEBUG [error] CalledProcessError: >>>>>>> Command '/bin/systemctl start [email protected] >>>>>>> <mailto:[email protected]>' returned non-zero exit >>>>>>> status >>>>>>> 1 >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show: >>>>>>> >>>>>>> >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - >>>>>>> slapd_bootstrap_config >>>>>>> - The default password storage scheme could not be read or was >>>>>>> not found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. >>>>>>> It is mandatory. >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >Checking on internet I show that "dse.ldif" could be >>>>>>> corrupted, so I changed with "dse.ldif.startOK" without any >>>>>>> change and then I changed with "dse.ldif.bak". The problem >>>>>>> persist but the error has >>>>>>> changed: >>>>>>> >>>>>>> >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - >>>>>>> config_set_port >>>>>>> - >>>>>>> >>>>>>> >Non-Secure Port Disabled >>>>>>> >>>>>>> >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - >>>>>>> symload_report_error >>>>>>> - >>>>>>> >>>>>>> >Netscape Portable Runtime error -5975: >>>>>>> >>>>>>> >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: >>>>>>> undefined >>>>>>> symbol: >>>>>>> >>>>>>> >replication_legacy_plugin_init >>>>>>> >>>>>>> >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - >>>>>>> symload_report_error >>>>>>> - >>>>>>> >>>>>>> >Could not load symbol "replication_legacy_plugin_init" from >>>>>>> >>>>>>> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for >>>>>>> plugin Legacy >>>>>>> >>>>>>> >Replication Plugin >>>>>>> >>>>>>> >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - >>>>>>> load_plugin_entry >>>>>>> - Unable to load plugin "cn=Legacy Replication >>>>>>> Plugin,cn=plugins,cn=config" >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >I saw a bug about this problem, but it is still opened: >>>>>>> >>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529 >>>>>>>> 44 >>>>>>>> 2 >>>>>>> > >>>>>>> >>>>>>> >Any idea how to fix the issue? >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >If it is not possible to fix it, can I remove the replica >>>>>>> from IPA and install it again with the same name? >>>>>>> >>>>>>> A quick fix could be to remove an entry for cn=Legacy Replication >>>>>>> Plugin,cn=plugins,cn=config from >>>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif >>>>>>> >>>>>>> when dirsrv is down. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> / Alexander Bokovoy >>>>>>> >>>>>>> Sr. Principal Software Engineer >>>>>>> >>>>>>> Security / Identity Management Engineering Red Hat Limited, >>>>>>> Finland >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> FreeIPA-users mailing list -- >>>>>>> [email protected] >>>>>>> To unsubscribe send an email to >>>>>>> [email protected] >>>>>>> >>>>>> _______________________________________________ >>>>>> FreeIPA-users mailing list -- [email protected] >>>>>> To unsubscribe send an email to >>>>>> [email protected] >>>>>> >> > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
