Thanks for your help. Details below. The problem 'moved' in I hope a
diagnositcally useful way, but the system remains broken.
On 8/8/23 08:54, Alexander Bokovoy wrote:
On Аўт, 08 жні 2023, Harry G Coin wrote:
On 8/8/23 02:43, Alexander Bokovoy wrote:
pstack $(pgrep ns-slapd) > ns-slapd log
Tried an upgrade from 4.9.10 to 4.9.11, the "writeback to ldap
failed" error moved from the primary instance (on which the dns
records were being added) to the replica which hung in the same
fashion. Here's the log you asked for from attempting 'systemctl
restart dirsrv@...' it just hangs at 100% cpu for about 10 minutes.
Thank you. Are you using schema compat for some legacy clients?
This is a fresh install of 4.9.10 about a week ago, upgraded to 4.9.11
yesterday, just two freeipa instances and no appreciable user load,
using the install defaults. The 'in house' system then starts loading
lots of dns records via the python ldap2 interface on the first of two
systems installed, the replica produced what you see in this post.
There is no 'private' information involved of any sort, it's supposed to
field DNS calls from the public but was so unreliable I had to implement
unbound on other servers, so all freeipa does is IXFR to unbound for the
heavy load. I suppose there may be <16 other in-house lab systems,
maybe 2 or 3 with any activity, that use it for dns. The only other
clue is these are running on VMs in older servers and have no other
software packages installed other than freeipa and what freeipa needs to
run, and the in-house program that loads the dns.
What's worse, every reboot attempt waits the full '9 min 29 secs' before
systemd forcibly terminates ns-slapd to finish the 'stop job'.
That's why I'm so troubled by all this, it's not like there is any
interference from anything other than what freeipa puts out there, and
it just locks with a message that gives no indication of what to do
about it, with nothing in any logs and 'systemctl is-system-running'
reports 'running'.
You could easily replicate this: imagine a simple validation test that
sets up two freeipa nodes, turns on dnssec, creates some domains, then
adds A AAAA and *.arpa records using the ldap2 api on one of the nodes.
Maybe limit the net speed between the nodes to a 1GB link typical, maybe
at most 4 processor cores of some older vintage and 5GB memory. It
takes less than 2 minutes after dns load start to lock up.
What's really odd is bind9 / named keeps blasting out change
notifications for some of the updated domains, then a few lines later,
with no intervening activity in any log or by any program affecting the
zone, will publish further change notifications with a new serial number
for the same zone. This happens for all the zones that get
modifications. I'm thinking 'rr' computations? I wonder if those
entries-- being auto-generated internally -- are creating a 'flow
control' issue between the primary and replica.
This looks like a compat tree stuck at some delete operation from the
retro changelog and locked up with multiple SASL bind operations in
parallel.
This sounds like a combination of existing bugs. In particular, thread
33 lockup is a problem we track for both RHEL 7 and RHEL 8 but those
bugs are all full of customer information and were made private so
linking them makes no good.
A one possible solution is to disable retro changelog trimming with
# dsconf IPA-TEST plugin retro-changelog set --max-age -1
Successfully changed the cn=Retro Changelog Plugin,cn=plugins,cn=config
(and restart the directory service)
But it did not help for some cases...
FYI on the replica prior
[root@registry2 ~]# ls -lh /var/lib/dirsrv/slapd-REALM/cldb/
total 29M
-rw-------. 1 dirsrv dirsrv 29M Aug 8 08:50
2701500c-321a11ee-b162de7a-fc13dab8_64cbd4a4000000040000.db
-rw-------. 1 dirsrv dirsrv 600K Aug 8 08:48
a68f1f2b-321a11ee-b162de7a-fc13dab8_64cbd580000000060000.db
-rw-------. 1 dirsrv dirsrv 30 Aug 3 11:24 DBVERSION
On the primary it was something like 69M.
On the replica I get:
[root@registry2 ~]# dsconf slapd-REALM plugin retro-changelog set
--max-age -1
Successfully changed the cn=Retro Changelog Plugin,cn=plugins,cn=config
On the primary I get:
dsconf slapd-REALM plugin retro-changelog set --max-age -1
Error: There is nothing to set in the cn=Retro Changelog
Plugin,cn=plugins,cn=config plugin entry
And when I reboot and re-run the test it.....
At boot, only on the replica, I get a new timeout:
Aug 08 10:08:39 registry2.1.adomain systemd[1]: ipa-dnskeysyncd.service:
Service RestartSec=1min expired, scheduling restart.
Aug 08 10:08:39 registry2.1.adomain systemd[1]: ipa-dnskeysyncd.service:
Scheduled restart job, restart counter is at 20.
Aug 08 10:08:39 registry2.1.adomain systemd[1]: Stopped IPA key daemon.
Aug 08 10:08:39 registry2.1.adomain systemd[1]: Started IPA key daemon.
Aug 08 10:08:42 registry2.1.adomain ipa-dnskeysyncd[27631]:
ipa-dnskeysyncd: INFO To increase debugging set debug=True in
dns.conf See default.conf(5) for details
Aug 08 10:08:46 registry2.1.adomain ns-slapd[1408]:
[08/Aug/2023:10:08:46.202605811 -0500] - ERR - slapd_poll - (110) -
Timed out
Aug 08 10:08:48 registry2.1.adomain ipa-dnskeysyncd[27631]:
ipa-dnskeysyncd: INFO LDAP bind...
Aug 08 10:08:48 registry2.1.adomain platform-python[27631]: GSSAPI
client step 1
Aug 08 10:08:48 registry2.1.adomain platform-python[27631]: GSSAPI
client step 1
Aug 08 10:08:48 registry2.1.adomain platform-python[27631]: GSSAPI
client step 1
Aug 08 10:08:48 registry2.1.adomain platform-python[27631]: GSSAPI
client step 2
Aug 08 10:08:48 registry2.1.adomain ipa-dnskeysyncd[27631]:
ipa-dnskeysyncd: INFO Commencing sync process
...
Aug 08 10:36:23 registry2.1.quietfountain.com ns-slapd[1408]: GSSAPI
client step 1
Aug 08 10:36:23 registry2.1.adomain ns-slapd[1408]: GSSAPI client step 1
Aug 08 10:36:23 registry2.1.adomain ns-slapd[1408]: GSSAPI client step 1
Aug 08 10:36:23 registry2.1.adomain ns-slapd[1408]: GSSAPI client step 2
Aug 08 10:36:24 registry2.1.adomain ns-slapd[1408]:
[08/Aug/2023:10:36:24.864167771 -0500] - ERR - slapd_poll - (110) -
Timed out
Aug 08 10:36:37 registry2.1.adomain named-pkcs11[1472]: 1.adomain/IN:
dns_diff_apply: update with no effect
Aug 08 10:36:39 registry2.1.adomain systemd[1]: ipa-dnskeysyncd.service:
Service RestartSec=1min expired, scheduling restart.
Aug 08 10:36:39 registry2.1.adomain systemd[1]: ipa-dnskeysyncd.service:
Scheduled restart job, restart counter is at 40.
Aug 08 10:36:39 registry2.1.adomain systemd[1]: Stopped IPA key daemon.
Aug 08 10:36:39 registry2.1.adomain systemd[1]: Started IPA key daemon.
Aug 08 10:36:42 registry2.1.adomain ipa-dnskeysyncd[55266]:
ipa-dnskeysyncd: INFO To increase debugging set debug=True in
dns.conf See default.conf(5) for details
Aug 08 10:36:47 registry2.1.adomain ipa-dnskeysyncd[55266]:
ipa-dnskeysyncd: INFO LDAP bind...
...
and bind9 can, but pkcs11 can't find one of the very first zones created,
Aug 08 10:00:38 registry2.1.adomain.com named-pkcs11[1472]: received
control channel command 'sign adomain.com.'
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
Traceback (most recent call last):
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/libexec/ipa/ipa-dnskeysyncd", line 130, in <module>
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search):
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib64/python3.6/site-packages/ldap/syncrepl.py", line 465, in
syncrepl_poll
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
self.syncrepl_refreshdone()
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib/python3.6/site-packages/ipaserver/dnssec/keysyncer.py", line
128, in syncrepl_refreshdone
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
self.bindmgr.sync(self.dnssec_zones)
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib/python3.6/site-packages/ipaserver/dnssec/bindmgr.py", line
231, in sync
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
self.sync_zone(zone)
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib/python3.6/site-packages/ipaserver/dnssec/bindmgr.py", line
215, in sync_zone
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
self.notify_zone(zone)
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib/python3.6/site-packages/ipaserver/dnssec/bindmgr.py", line 51,
in notify_zone
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
result = ipautil.run(cmd, capture_output=True)
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]: File
"/usr/lib/python3.6/site-packages/ipapython/ipautil.py", line 600, in run
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
p.returncode, arg_string, output_log, error_log
Aug 08 10:00:38 registry2.1.adomain.com ipa-dnskeysyncd[19580]:
ipapython.ipautil.CalledProcessError: CalledProcessError(Command
['rndc', 'sign', 'adomain.com.'] returned non-zero exit status 1: "rndc:
'sign' failed: not found\nno matching zone 'adomain
.com.' in any view\n")
Aug 08 10:00:40 registry2.1.adomain.com systemd[1]:
ipa-dnskeysyncd.service: Main process exited, code=exited, status=1/FAILURE
Aug 08 10:00:40 registry2.1.adomain.com systemd[1]:
ipa-dnskeysyncd.service: Failed with result 'exit-code'.
Aug 08 10:00:47 registry2.1.adomain.com named-pkcs11[1472]:
1.adomain.com/IN: dns_diff_apply: update with no effect
But the domain exists in the ldap, on the web-UI.
I added a temporary A record to the 'adomain' on the primary, to see if
it would result in it 'showing up' on the replica. While it did show up
in the ldap, the above key sync issue remained. The above errors
continue, all lead by the 'slapd_poll' (110) Timed out message.
So the result of the advised ds-conf change is -- ns-slapd doesn't hang
when A PTR and AAAA records are added on the primary or the replica,
but the ipa-dnskeysyncd aborts with a ns-slapd timeout. So perhaps the
threads that hang are fewer than before, but still there.
Here's a sample pstack
[root@registry2 ~]# cat /qflibrary/scratch/ns-slapd4.log
Thread 37 (Thread 0x7f0e4aff6700 (LWP 95390)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8a6d222c in repl5_inc_result_threadmain
(param=0x7f0e5eecdf60) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:285
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 36 (Thread 0x7f0e50279700 (LWP 2242)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed7041e5c in ps_send_results (arg=<optimized out>) at
ldap/servers/slapd/psearch.c:308
#2 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#3 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#4 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 35 (Thread 0x7f0e553fc700 (LWP 2235)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed7041e5c in ps_send_results (arg=<optimized out>) at
ldap/servers/slapd/psearch.c:308
#2 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#3 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#4 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 34 (Thread 0x7f0e55bfd700 (LWP 1442)):
#0 0x00007f0e9bf7af41 in poll () at target:/lib64/libc.so.6
#1 0x00007f0e9a247658 in _pr_poll_with_poll () at
target:/lib64/libnspr4.so
#2 0x0000564ed7039d70 in accept_thread (vports=0x7ffe886d3c40) at
ldap/servers/slapd/daemon.c:986
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 33 (Thread 0x7f0e563fe700 (LWP 1441)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e61c1a000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 32 (Thread 0x7f0e56dff700 (LWP 1440)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e66438580, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 31 (Thread 0x7f0e579fe700 (LWP 1439)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e883eeb20, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 30 (Thread 0x7f0e583ff700 (LWP 1438)):
#0 0x00007f0e9bf7d8af in fdatasync () at target:/lib64/libc.so.6
#1 0x00007f0e91cbe6b5 in __os_fsync () at target:/lib64/libdb-5.3.so
#2 0x00007f0e91ca598c in __log_flush_int () at target:/lib64/libdb-5.3.so
#3 0x00007f0e91ca7dd0 in __log_flush () at target:/lib64/libdb-5.3.so
#4 0x00007f0e91ca7f73 in __log_flush_pp () at target:/lib64/libdb-5.3.so
#5 0x00007f0e8afe1304 in bdb_txn_commit (li=<optimized out>,
txn=0x7f0e583fd028, use_lock=1) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2772
#6 0x00007f0e8af95515 in dblayer_txn_commit (be=0x7f0e88424f00,
txn=<optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:736
#7 0x00007f0e8afa7ebe in ldbm_back_add (pb=0x7f0e85748860) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:1242
#8 0x00007f0e9d7d7728 in op_shared_add (pb=pb@entry=0x7f0e85748860) at
ldap/servers/slapd/add.c:692
#9 0x00007f0e9d7d7bbe in add_internal_pb (pb=pb@entry=0x7f0e85748860)
at ldap/servers/slapd/add.c:407
#10 0x00007f0e9d7d8975 in slapi_add_internal_pb
(pb=pb@entry=0x7f0e85748860) at ldap/servers/slapd/add.c:331
#11 0x00007f0e8960f8bf in write_replog_db (newsuperior=0x0,
modrdn_mods=0x0, newrdn=0x0, post_entry=<optimized out>,
log_e=0x7f0e4df5b9c0, curtime=1691511446, flag=0, log_m=0x7f0e5ec0d440,
dn=0x7f0e57a40740
"idnsname=8.0.f.0.0.0.0.0.0.0.0.1.0.0.c.f.ip6.arpa.,cn=dns,dc=1,
dc=quietfountain,dc=com", optype=<optimized out>, pb=0x7f0e66a09580) at
ldap/servers/plugins/retrocl/retrocl_po.c:369
#12 0x00007f0e8960f8bf in retrocl_postob (pb=0x7f0e66a09580,
optype=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_po.c:697
#13 0x00007f0e9d83cc79 in plugin_call_func (list=0x7f0e924aae00,
operation=operation@entry=561, pb=pb@entry=0x7f0e66a09580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#14 0x00007f0e9d83cec4 in plugin_call_list (pb=0x7f0e66a09580,
operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#15 0x00007f0e9d83cec4 in plugin_call_plugins
(pb=pb@entry=0x7f0e66a09580, whichfunction=whichfunction@entry=561) at
ldap/servers/slapd/plugin.c:442
#16 0x00007f0e8afc3658 in ldbm_back_modify (pb=<optimized out>) at
ldap/servers/slapd/back-ldbm/ldbm_modify.c:1002
#17 0x00007f0e9d828300 in op_shared_modify (pb=pb@entry=0x7f0e66a09580,
pw_change=pw_change@entry=0, old_pw=0x0) at
ldap/servers/slapd/modify.c:1025
#18 0x00007f0e9d829a00 in do_modify (pb=pb@entry=0x7f0e66a09580) at
ldap/servers/slapd/modify.c:380
#19 0x0000564ed703475b in connection_dispatch_operation
(pb=0x7f0e66a09580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:651
#20 0x0000564ed703475b in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#21 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#22 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#23 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 29 (Thread 0x7f0e591ff700 (LWP 1437)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e68806580, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 28 (Thread 0x7f0e5a9ff700 (LWP 1436)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e69c05000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 27 (Thread 0x7f0e5b5ff700 (LWP 1435)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e563ff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 26 (Thread 0x7f0e5c1fd700 (LWP 1434)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e59fff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 25 (Thread 0x7f0e5c9fe700 (LWP 1433)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e587ff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 24 (Thread 0x7f0e5d1ff700 (LWP 1432)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e59800000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 23 (Thread 0x7f0e5dfff700 (LWP 1431)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e59a00000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 22 (Thread 0x7f0e5ebfe700 (LWP 1430)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e579ff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 21 (Thread 0x7f0e5f7ff700 (LWP 1429)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e5ebff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 20 (Thread 0x7f0e603fc700 (LWP 1428)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e59400000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 19 (Thread 0x7f0e60bfd700 (LWP 1427)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e5d5ff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 18 (Thread 0x7f0e613fe700 (LWP 1426)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed70323f7 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x7f0e5edff000, interval=interval@entry=0) at
ldap/servers/slapd/connection.c:989
#2 0x0000564ed7033a8a in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1538
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 17 (Thread 0x7f0e6295d700 (LWP 1424)):
#0 0x00007f0e99bec848 in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9d7fd0ee in eq_loop_rel (arg=<optimized out>) at
ldap/servers/slapd/eventq.c:316
#2 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#3 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#4 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 16 (Thread 0x7f0e6315e700 (LWP 1423)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9a245ba8 in PR_WaitCondVar () at target:/lib64/libnspr4.so
#2 0x00007f0e9d7fda23 in eq_loop (arg=<optimized out>) at
ldap/servers/slapd/eventq-deprecated.c:322
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 15 (Thread 0x7f0e6395f700 (LWP 1422)):
#0 0x00007f0e99bec848 in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x0000564ed703d2ba in housecleaning (cur_time=<optimized out>) at
ldap/servers/slapd/house.c:50
#2 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#3 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#4 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 14 (Thread 0x7f0e64160700 (LWP 1421)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9d86373d in slapi_wait_condvar_pt (cvar=0x7f0e864fd080,
mutex=0x7f0e864fcf00, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f0e89402af1 in roles_cache_wait_on_change
(arg=0x7f0e87553e80) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 13 (Thread 0x7f0e64961700 (LWP 1420)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9d86373d in slapi_wait_condvar_pt (cvar=0x7f0e864fcf90,
mutex=0x7f0e864fd0b0, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f0e89402af1 in roles_cache_wait_on_change
(arg=0x7f0e8640c980) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 12 (Thread 0x7f0e65162700 (LWP 1419)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9d86373d in slapi_wait_condvar_pt (cvar=0x7f0e864b9580,
mutex=0x7f0e864fd8f0, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f0e89402af1 in roles_cache_wait_on_change
(arg=0x7f0e8641d080) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 11 (Thread 0x7f0e673ff700 (LWP 1418)):
#0 0x00007f0e99bec848 in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e8a6d13be in protocol_sleep (prp=prp@entry=0x7f0e86408000,
duration=duration@entry=300) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1222
#2 0x00007f0e8a6d4fb5 in repl5_inc_run (prp=0x7f0e86408000) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:779
#3 0x00007f0e8a6d97e1 in prot_thread_main (arg=0x7f0e863e30b0) at
ldap/servers/plugins/replication/repl5_protocol.c:252
#4 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 10 (Thread 0x7f0e67ffe700 (LWP 1417)):
#0 0x00007f0e99be76cd in __pthread_timedjoin_ex () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9a24bd90 in PR_JoinThread () at target:/lib64/libnspr4.so
#2 0x00007f0e8a6d42a0 in repl5_inc_destroy_async_result_thread
(rd=0x7f0e5eecdf60) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:447
#3 0x00007f0e8a6d42a0 in send_updates (num_changes_sent=<synthetic
pointer>, remote_update_vector=<optimized out>, prp=0x7f0e86417f00) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1909
#4 0x00007f0e8a6d42a0 in repl5_inc_run (prp=0x7f0e86417f00) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1050
#5 0x00007f0e8a6d97e1 in prot_thread_main (arg=0x7f0e863e2840) at
ldap/servers/plugins/replication/repl5_protocol.c:252
#6 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#7 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#8 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 9 (Thread 0x7f0e9e0a7700 (LWP 1416)):
#0 0x00007f0e99bec7aa in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9a245673 in pt_TimedWait () at target:/lib64/libnspr4.so
#2 0x00007f0e9a245b3e in PR_WaitCondVar () at target:/lib64/libnspr4.so
#3 0x00007f0e8a6b991e in _cl5TrimMain (param=<optimized out>) at
ldap/servers/plugins/replication/cl5_api.c:3189
#4 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 8 (Thread 0x7f0e687ff700 (LWP 1415)):
#0 0x00007f0e9bf7af41 in poll () at target:/lib64/libc.so.6
#1 0x00007f0e8d3e1b8f in ipa_cldap_worker () at
target:/usr/lib64/dirsrv/plugins/libipa_cldap.so
#2 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#3 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 7 (Thread 0x7f0e695ff700 (LWP 1414)):
#0 0x00007f0e99bec45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f0e9d86373d in slapi_wait_condvar_pt (cvar=0x7f0e8755b2d0,
mutex=0x7f0e8755b1b0, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f0e90084659 in cos_cache_wait_on_change (arg=<optimized out>)
at ldap/servers/plugins/cos/cos_cache.c:404
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 6 (Thread 0x7f0e6a9cc700 (LWP 1413)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8afdbafc in locks_monitoring_threadmain
(param=0x7f0e96249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2991
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 5 (Thread 0x7f0e6b1cd700 (LWP 1412)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8afcf408 in perfctrs_wait
(milliseconds=milliseconds@entry=1000, priv=<optimized out>,
db_env=<optimized out>) at ldap/servers/slapd/back-ldbm/perfctrs.c:80
#3 0x00007f0e8afdb994 in perf_threadmain (param=0x7f0e96249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2916
#4 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 4 (Thread 0x7f0e6b9ce700 (LWP 1411)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=ticks@entry=250) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8afdb807 in trickle_threadmain (param=0x7f0e96249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3976
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 3 (Thread 0x7f0e6c1cf700 (LWP 1410)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=ticks@entry=2500) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8afddb43 in checkpoint_threadmain (param=0x7f0e96249700)
at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3824
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 2 (Thread 0x7f0e6c9d0700 (LWP 1409)):
#0 0x00007f0e9bf7d67f in select () at target:/lib64/libc.so.6
#1 0x00007f0e9d8785b4 in DS_Sleep (ticks=ticks@entry=100) at
ldap/servers/slapd/util.c:1060
#2 0x00007f0e8afdb2e7 in deadlock_threadmain (param=0x7f0e96249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3494
#3 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f0e99be61ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Thread 1 (Thread 0x7f0e9e2484c0 (LWP 1408)):
#0 0x00007f0e9bf7af41 in poll () at target:/lib64/libc.so.6
#1 0x00007f0e9a247658 in _pr_poll_with_poll () at
target:/lib64/libnspr4.so
#2 0x0000564ed7037f3f in slapd_daemon (ports=<optimized out>) at
ldap/servers/slapd/daemon.c:1631
#3 0x0000564ed702bded in main (argc=5, argv=0x7ffe886d4048) at
ldap/servers/slapd/main.c:1099
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
