To follow up on this. I think the issue is resolved. We have 8 IPA servers. And the primary server on which this error was occurring had 7 replication agreements! Ended up changing the replication agreements so that 2 servers had 4 agreements (3 + 1 amongst themselves) and all others with 2 agreements each.
This seems to have resolved the core-dump of kerberos. No upgrade was done. Hope this helps someone. On 29 January 2016 at 15:13, Prashant Bapat <prash...@apigee.com> wrote: > We will have to run with F21 for now. There are plans for moving to CentOS > 7.x in the near future. Until then, I'm afraid I will have to live with > this. > > Thanks much Sumit for all your help in identifying this. > > Regards. > --Prashant​ > > On 28 January 2016 at 23:24, Sumit Bose <sb...@redhat.com> wrote: > >> On Thu, Jan 28, 2016 at 09:36:55PM +0530, Prashant Bapat wrote: >> > Sure. Attached the stack trace with debuginfo installed. >> > >> > Thanks much! >> >> This looks very much like the issue Simo fixed recently, but >> unfortunately I think it is so recent that it is not available in any >> release package. Additionally it would be quite some effort for me the >> generate a F21 test build because as Lukas said F21 is already >> End-of-life and there is not infrastructure anymore to easily build F21 >> package. If it would be possible to upgrade to a newer version of Fedora >> I'd be happy to provide a test build with the patch. >> >> bye, >> Sumit >> >> > >> > On 28 January 2016 at 16:53, Sumit Bose <sb...@redhat.com> wrote: >> > >> > > On Thu, Jan 28, 2016 at 04:42:20PM +0530, Prashant Bapat wrote: >> > > > gdb stacktrace attached. >> > > >> > > Can you install the debuginfo with >> > > >> > > debuginfo-install krb5-server-1.12.2-19.fc21.x86_64 >> > > >> > > as suggested by gdb and then call 'bt full' again to get more details. >> > > Additionally the debuginfo of the freeipa package might be missing as >> > > well. >> > > >> > > bye, >> > > Sumit >> > > > >> > > > On 28 January 2016 at 16:27, Prashant Bapat <prash...@apigee.com> >> wrote: >> > > > >> > > > > Thanks Sumit. >> > > > > >> > > > > From the logs there is nothing unusual around the time of core >> dump. I >> > > > > found this one line odd though. >> > > > > >> > > > > *Jan 26 03:15:58 ipa.example.net <http://ipa.example.net> >> > > > > krb5kdc[4471](Error): worker 4473 exited with status 134* >> > > > > >> > > > > >> > > > > Let me try to get the full BT. >> > > > > >> > > > > On 28 January 2016 at 13:54, Sumit Bose <sb...@redhat.com> wrote: >> > > > > >> > > > >> On Thu, Jan 28, 2016 at 10:25:53AM +0530, Prashant Bapat wrote: >> > > > >> > Hi, >> > > > >> > >> > > > >> > We have a FreeIPA 4.1.4 setup on F21 servers. There is 1 >> master and >> > > 7 >> > > > >> > replicas in different regions. Earlier there was only 1 >> replica. >> > > Since I >> > > > >> > added new replicas, on the master node, once in a while the >> kerberos >> > > > >> > process dumps core and everything stops working - >> authentication, >> > > > >> > replication etc. If we restart everything using "ipactl >> restart" >> > > things >> > > > >> are >> > > > >> > back to normal. >> > > > >> > >> > > > >> > Attached is the output from journalctl for kerberos. >> > > > >> > >> > > > >> > Has anyone come across this ? Are there any pointers to >> > > troubleshooting >> > > > >> > this ? >> > > > >> >> > > > >> This might be fixed recently by a patch from Simo >> > > > >> (2144b1eeb789639b8a3df287b580aeb6196188a8). But to help to better >> > > > >> identify the issue the content of the kdc logs around the time >> of the >> > > > >> crash might be useful. Additionally a full backtrace which you >> can get >> > > > >> by calling >> > > > >> >> > > > >> coredumpclt gdb 4475 >> > > > >> >> > > > >> and then >> > > > >> >> > > > >> bt full >> > > > >> >> > > > >> bye, >> > > > >> Sumit >> > > > >> >> > > > >> > >> > > > >> > Any help is appreciated. >> > > > >> > >> > > > >> > Thanks. >> > > > >> > --Prashant >> > > > >> >> > > > >> > Jan 26 03:15:59 ipa.example.net systemd-coredump[5000]: >> Process >> > > 4475 >> > > > >> (krb5kdc) of user 0 dumped core. >> > > > >> > >> > > > >> > Stack >> trace >> > > of >> > > > >> thread 4475: >> > > > >> > #0 >> > > > >> 0x00007f99de8c18d7 raise (libc.so.6) >> > > > >> > #1 >> > > > >> 0x00007f99de8c353a abort (libc.so.6) >> > > > >> > #2 >> > > > >> 0x00007f99de8ba47d __assert_fail_base (libc.so.6) >> > > > >> > #3 >> > > > >> 0x00007f99de8ba532 __assert_fail (libc.so.6) >> > > > >> > #4 >> > > > >> 0x00007f99d783a78f ldap_get_values_len (libldap_r-2.4.so.2) >> > > > >> > #5 >> > > > >> 0x00007f99d7c8173e ipadb_ldap_attr_to_int (ipadb.so) >> > > > >> > #6 >> > > > >> 0x00007f99d7c83f9c ipadb_parse_ldap_entry (ipadb.so) >> > > > >> > #7 >> > > > >> 0x00007f99d7c849ab ipadb_get_principal (ipadb.so) >> > > > >> > #8 >> > > > >> 0x00007f99e0433b14 krb5_db_get_principal (libkdb5.so.7) >> > > > >> > #9 >> > > > >> 0x000055768457c230 process_tgs_req (krb5kdc) >> > > > >> > #10 >> > > > >> 0x0000557684579fe3 dispatch (krb5kdc) >> > > > >> > #11 >> > > > >> 0x000055768458d8a0 process_packet (krb5kdc) >> > > > >> > #12 >> > > > >> 0x00007f99dec4cc78 verto_fire (libverto.so.1) >> > > > >> > #13 >> > > > >> 0x00007f99d6fb72a3 epoll_event_loop_once (libtevent.so.0) >> > > > >> > #14 >> > > > >> 0x00007f99d6fb5787 std_event_loop_once (libtevent.so.0) >> > > > >> > #15 >> > > > >> 0x00007f99d6fb1fed _tevent_loop_once (libtevent.so.0) >> > > > >> > #16 >> > > > >> 0x00007f99dec4c3f7 verto_run (libverto.so.1) >> > > > >> > #17 >> > > > >> 0x00005576845795ab main (krb5kdc) >> > > > >> > #18 >> > > > >> 0x00007f99de8acfe0 __libc_start_main (libc.so.6) >> > > > >> > #19 >> > > > >> 0x00005576845798f0 _start (krb5kdc) >> > > > >> > >> > > > >> > Jan 26 03:15:59 ipa.example.net systemd-coredump[4999]: >> Process >> > > 4473 >> > > > >> (krb5kdc) of user 0 dumped core. >> > > > >> > >> > > > >> > Stack >> trace >> > > of >> > > > >> thread 4473: >> > > > >> > #0 >> > > > >> 0x00007f99de8c18d7 raise (libc.so.6) >> > > > >> > #1 >> > > > >> 0x00007f99de8c353a abort (libc.so.6) >> > > > >> > #2 >> > > > >> 0x00007f99de8ba47d __assert_fail_base (libc.so.6) >> > > > >> > #3 >> > > > >> 0x00007f99de8ba532 __assert_fail (libc.so.6) >> > > > >> > #4 >> > > > >> 0x00007f99d783a78f ldap_get_values_len (libldap_r-2.4.so.2) >> > > > >> > #5 >> > > > >> 0x00007f99d7c8173e ipadb_ldap_attr_to_int (ipadb.so) >> > > > >> > #6 >> > > > >> 0x00007f99d7c83f9c ipadb_parse_ldap_entry (ipadb.so) >> > > > >> > #7 >> > > > >> 0x00007f99d7c849ab ipadb_get_principal (ipadb.so) >> > > > >> > #8 >> > > > >> 0x00007f99e0433b14 krb5_db_get_principal (libkdb5.so.7) >> > > > >> > #9 >> > > > >> 0x000055768457c230 process_tgs_req (krb5kdc) >> > > > >> > #10 >> > > > >> 0x0000557684579fe3 dispatch (krb5kdc) >> > > > >> > #11 >> > > > >> 0x000055768458d8a0 process_packet (krb5kdc) >> > > > >> > #12 >> > > > >> 0x00007f99dec4cc78 verto_fire (libverto.so.1) >> > > > >> > #13 >> > > > >> 0x00007f99d6fb72a3 epoll_event_loop_once (libtevent.so.0) >> > > > >> > #14 >> > > > >> 0x00007f99d6fb5787 std_event_loop_once (libtevent.so.0) >> > > > >> > #15 >> > > > >> 0x00007f99d6fb1fed _tevent_loop_once (libtevent.so.0) >> > > > >> > #16 >> > > > >> 0x00007f99dec4c3f7 verto_run (libverto.so.1) >> > > > >> > #17 >> > > > >> 0x00005576845795ab main (krb5kdc) >> > > > >> > #18 >> > > > >> 0x00007f99de8acfe0 __libc_start_main (libc.so.6) >> > > > >> > #19 >> > > > >> 0x00005576845798f0 _start (krb5kdc) >> > > > >> >> > > > >> > -- >> > > > >> > Manage your subscription for the Freeipa-users mailing list: >> > > > >> > https://www.redhat.com/mailman/listinfo/freeipa-users >> > > > >> > Go to http://freeipa.org for more info on the project >> > > > >> >> > > > >> -- >> > > > >> Manage your subscription for the Freeipa-users mailing list: >> > > > >> https://www.redhat.com/mailman/listinfo/freeipa-users >> > > > >> Go to http://freeipa.org for more info on the project >> > > > >> >> > > > > >> > > > > >> > > >> > > >> > > >> > > >> >> >> >> >
-- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project