2 FreeIPA 4.1.4 servers running on CentOS 7. dc1 has a sync agreement to a windows server.
It has been running fine since June 5 when I re-initialized a sync agreement that had somehow uninitialized itself. Original issue report here : https://www.redhat.com/archives/freeipa-users/2015-June/msg00147.html Bug report here : https://fedorahosted.org/freeipa/ticket/5054 It appears the same thing may have happened again, one month later, but this time randomly, as we have not made any changes to our sync agreement since the initial change in June. it appears to have unitialized itself without us changing it and managed to crash the directory server in doing so. Note that during the last week I could still login to the web ui, but around the time the log entries change, I became unable to. I tried to login to the web server today and it would not let me login, so I went to the shell on the server and noticed that ipactl command would freeze up again. I looked at the logs (which I will paste below) and restarted the directory server service. I then found that my sync agreement had become uninitialized. --- output --- [root@dc1 slapd-IPADOMAIN-NET]# ldapsearch -xLLL -D "cn=directory manager" -W -b cn=config objectclass=nsDSWindowsReplicationAgreement Enter LDAP Password: dn: cn=meToofficedc2.office.addomain.net,cn=replica,cn=dc\3Dipadomain \2Cdc\3Dnet,cn=mapping tree,cn=config nsds7WindowsReplicaSubtree: OU=Staff,DC=office,DC=addomain,DC=net nsds7DirectoryReplicaSubtree: cn=users,cn=accounts,dc=ipadomain,dc=net cn: meToofficedc2.office.addomain.net nsds7NewWinGroupSyncEnabled: false objectClass: nsDSWindowsReplicationAgreement objectClass: top nsDS5ReplicaTransportInfo: TLS description: me to officedc2.office.addomain.net nsDS5ReplicaRoot: dc=ipadomain,dc=net nsDS5ReplicaHost: officedc2.office.addomain.net nsds5replicaTimeout: 120 nsDS5ReplicaBindDN: cn=freeipa syncuser,ou=Service Account,dc=office,dc=addomain,dc=net nsds7NewWinUserSyncEnabled: true nsDS5ReplicaPort: 389 nsds7WindowsDomain: ipadomain.net nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicaBindMethod: simple nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG RERBNEJDUmtOelUzTTJJNVlpMDBaV1EyTTJRMQ0KWXkwNU0yTm1aV05sTVMxbU5qRXpaak5oTlFBQ 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ2k0N0NxRGZFd2JIdm I0MFVFZVI3MA==}gWI9NIB8lbt9tmNszzbBFCAe4Vs/e0sMyn5+NZPJg9E= nsds7DirsyncCookie:: TVNEUwMAAABoJPGME7jQAQAAAAAAAAAAYAEAAPc1qQAAAAAAAAAAAAAAA AD3NakAAAAAAMUjuImqVZhBkOkdt24C0IsBAAAAAAAAAA4AAAAAAAAAY4GwFkVcvEmMMExrVon4d6 13PwAAAAAADGzFNzznrESIxHzA74fbsz4HUgAAAAAAOnFoO5OE2E27lR/g4EcjQTLbIwAAAAAAuEm PWjYok0qGS0HM/+TDmK7FgAMAAAAA6PTFXvAdnkaJSIkZT1lS+/FcJAAAAAAA4qTQaC46/Ua4KXgP /ixNcbK4dgAAAAAAWowbgYD1akibZ+sCul5C4VNsMQAAAAAAxSO4iapVmEGQ6R23bgLQi/c1qQAAA AAAogC6jFcyFUmhBp4B7FkaBcRHwwEAAAAAyhKMxsP0uUKGEnG2lsyA8eTUwgYAAAAA4n8Xx1bAlU mBUl3zhlZ9WBngDAAAAAAA71vM2ebFEkCJkBaLjB4CGU+4CQMAAAAAGfO+4ndZCkaVKnwZNlNsf90 NDAAAAAAAgD6n+M2bcUGkOwo5gPLx7IOjAwAAAAAA nsds50ruv: {replicageneration} 553fe9bb000000040000 nsds50ruv: {replica 4 ldap://dc1.ipadomain.net:389} 553fe9c9 000000040000 557f49fb000200040000 nsds50ruv: {replica 3 ldap://dc2.ipadomain.net:389} 553fe9c 4000000030000 557f3e4a000200030000 nsruvReplicaLastModified: {replica 4 ldap://dc1.ipadomain.ne t:389} 557f494a nsruvReplicaLastModified: {replica 3 ldap://dc2.ipadomain.n et:389} 557f3d95 oneWaySync: fromWindows nsds5ReplicaEnabled: on nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 0 nsds5replicaLastUpdateEnd: 0 nsds5replicaChangesSentSinceStartup: nsds5replicaLastUpdateStatus: -1 - LDAP error: Can't contact LDAP server nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 0 nsds5replicaLastInitEnd: 0 --- output --- Here are the error logs for the last month for the directory server. They are totally empty until July 2. ---output--- 389-Directory/1.3.3.8 B2015.040.128 dc1.ipadomain.net:636 (/etc/dirsrv/slapd-IPADOMAIN-NET) [02/Jul/2015:03:19:02 +0000] NSMMReplicationPlugin - windows sync - failed to send dirsync search request: 2 [02/Jul/2015:06:10:29 +0000] - Entry "uid=jenkinsdev,cn=users,cn=accounts,dc=ipadomain,dc=net" missing attribute "sn" required by object class "person" [03/Jul/2015:02:04:02 +0000] NSMMReplicationPlugin - windows sync - failed to send dirsync search request: 2 [03/Jul/2015:05:39:01 +0000] NSMMReplicationPlugin - windows sync - failed to send dirsync search request: 2 [03/Jul/2015:17:09:00 +0000] NSMMReplicationPlugin - windows sync - failed to send dirsync search request: 2 [03/Jul/2015:22:41:32 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot contact any KDC for realm 'IPADOMAIN.NET')) errno 115 (Operation now in progress) [03/Jul/2015:22:41:32 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [03/Jul/2015:22:41:32 +0000] NSMMReplicationPlugin - agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot contact any KDC for realm 'IPADOMAIN.NET')) [03/Jul/2015:22:41:36 +0000] NSMMReplicationPlugin - agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI auth resumed [05/Jul/2015:19:24:00 +0000] NSMMReplicationPlugin - windows sync - failed to send dirsync search request: 2 [06/Jul/2015:02:46:50 +0000] - Entry "uid=accounting,cn=users,cn=accounts,dc=ipadomain,dc=net" missing attribute "sn" required by object class "person" [06/Jul/2015:17:47:04 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [06/Jul/2015:17:47:04 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.addomain.net" (officedc2:389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [06/Jul/2015:17:47:07 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [06/Jul/2015:17:47:13 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [06/Jul/2015:17:47:25 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) ... repeats for 7 days ... [13/Jul/2015:21:49:21 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such file or directory) [13/Jul/2015:21:49:45 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such file or directory) [13/Jul/2015:21:50:33 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such file or directory) [13/Jul/2015:21:52:09 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such file or directory) [13/Jul/2015:21:54:00 +0000] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such file or directory) [13/Jul/2015:23:04:05 +0000] set_krb5_creds - Could not get initial credentials for principal [ldap/dc1.ipadomain....@ipadomain.net] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text)) [13/Jul/2015:23:04:05 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired)) errno 2 (No such file or directory) [13/Jul/2015:23:04:10 +0000] set_krb5_creds - Could not get initial credentials for principal [ldap/dc1.ipadomain....@ipadomain.net] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text)) [13/Jul/2015:23:04:10 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired)) errno 2 (No such file or directory) [13/Jul/2015:23:04:10 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [13/Jul/2015:23:04:10 +0000] NSMMReplicationPlugin - agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired)) ---output--- -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project