Re: [Freeipa-users] more replication issues
On 05/13/2015 06:34 PM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. your original message is very clear: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) this means that you have replication agreement wth SIMPLE auth which uses a nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config which does not exist on the target server of the agreement. Now you say it was never deleted, so it was probably never added, but used in the replication agreements. How do you manage and setup replication agreements ? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 5/15/15 3:30 AM, Ludwig Krispenz wrote: On 05/13/2015 06:34 PM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. your original message is very clear: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) this means that you have replication agreement wth SIMPLE auth which uses a nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config which does not exist on the target server of the agreement. Now you say it was never deleted, so it was probably never added, but used in the replication agreements. How do you manage and setup replication agreements ? All replicas are configred simply: ipa-replica-prepare hostname... scp .. ipa-replica-install --no-ntp --setup-ca Replica-file That is it. NTP is not set because internal NTP servers are used. All replicas are CA replicas for safety (no certs are managed) After a few days to a week the message starts popping up in logs. ~J -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 05/15/2015 02:45 PM, Janelle wrote: On 5/15/15 3:30 AM, Ludwig Krispenz wrote: On 05/13/2015 06:34 PM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. your original message is very clear: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) this means that you have replication agreement wth SIMPLE auth which uses a nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config which does not exist on the target server of the agreement. Now you say it was never deleted, so it was probably never added, but used in the replication agreements. How do you manage and setup replication agreements ? All replicas are configred simply: ipa-replica-prepare hostname... scp .. ipa-replica-install --no-ntp --setup-ca Replica-file That is it. NTP is not set because internal NTP servers are used. All replicas are CA replicas for safety (no certs are managed) ok, I was a bit puzzled because ipa uses ldapprincipals and gssapi for the main suffix replication. But I just verified that after ipa-replica-install --setup-ca CA replication is setup with users in ou=csusers,cn=config and uses it as replica binddn, I have no idea why it would disappear. when Rich asked to search for a DEL, did you check this on the server that logged the message or on the endpoint of the replication agreement (it should be there), and you may have to check in the rotated access logs access.timestamp as well After a few days to a week the message starts popping up in logs. ~J -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 05/15/2015 09:53 AM, Janelle wrote: On May 15, 2015, at 08:57, Ludwig Krispenz lkris...@redhat.com wrote: On 05/15/2015 02:45 PM, Janelle wrote: On 5/15/15 3:30 AM, Ludwig Krispenz wrote: On 05/13/2015 06:34 PM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. your original message is very clear: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) this means that you have replication agreement wth SIMPLE auth which uses a nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config which does not exist on the target server of the agreement. Now you say it was never deleted, so it was probably never added, but used in the replication agreements. How do you manage and setup replication agreements ? All replicas are configred simply: ipa-replica-prepare hostname... scp .. ipa-replica-install --no-ntp --setup-ca Replica-file That is it. NTP is not set because internal NTP servers are used. All replicas are CA replicas for safety (no certs are managed) ok, I was a bit puzzled because ipa uses ldapprincipals and gssapi for the main suffix replication. But I just verified that after ipa-replica-install --setup-ca CA replication is setup with users in ou=csusers,cn=config and uses it as replica binddn, I have no idea why it would disappear. when Rich asked to search for a DEL, did you check this on the server that logged the message or on the endpoint of the replication agreement (it should be there), and you may have to check in the rotated access logs access.timestamp as well Checked it on ALL servers just to be sure. ~J If it is present at some point, then is missing, it must be some internal operation that is removing it. Please enable access logging of internal operations: ldapmodify -x -h consumer.host -D cn=directory manager -w password EOF dn: cn=config changetype: modify replace: nsslapd-accesslog-level nsslapd-accesslog-level: 4 EOF Then you will have to wait until the problem reoccurs Is or was the server ipa01.example.com the target of a host delete, replica delete, or cleanallruv operation? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On May 15, 2015, at 08:57, Ludwig Krispenz lkris...@redhat.com wrote: On 05/15/2015 02:45 PM, Janelle wrote: On 5/15/15 3:30 AM, Ludwig Krispenz wrote: On 05/13/2015 06:34 PM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. your original message is very clear: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) this means that you have replication agreement wth SIMPLE auth which uses a nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config which does not exist on the target server of the agreement. Now you say it was never deleted, so it was probably never added, but used in the replication agreements. How do you manage and setup replication agreements ? All replicas are configred simply: ipa-replica-prepare hostname... scp .. ipa-replica-install --no-ntp --setup-ca Replica-file That is it. NTP is not set because internal NTP servers are used. All replicas are CA replicas for safety (no certs are managed) ok, I was a bit puzzled because ipa uses ldapprincipals and gssapi for the main suffix replication. But I just verified that after ipa-replica-install --setup-ca CA replication is setup with users in ou=csusers,cn=config and uses it as replica binddn, I have no idea why it would disappear. when Rich asked to search for a DEL, did you check this on the server that logged the message or on the endpoint of the replication agreement (it should be there), and you may have to check in the rotated access logs access.timestamp as well Checked it on ALL servers just to be sure. ~J -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager results from the first ldapsearch. however, the parent is there: dn: ou=csusers,cn=config objectClass: top objectClass: organizationalUnit ou: csusers -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] more replication issues
Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) more and more and more. When it happens, I have to re-initialize from one of the good servers and go around in a circle (I have replication in a ring, as shown in documentation examples). The list-ruv on every server matches. And yet, out of 18 masters, thisis occuring now on about half of them. Once again I am beginning to question the robustness of 389-ds and the replication problems that many of us continue to report. How do we get this to be more solid? I love this product. It really is something that RH can push, but it really needs to be rock solid and with all the replication issues, well, it seems like it is not commercially ready? Any ideas/thoughts/comments? thank you Janelle -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config more and more and more. When it happens, I have to re-initialize from one of the good servers and go around in a circle (I have replication in a ring, as shown in documentation examples). The list-ruv on every server matches. And yet, out of 18 masters, thisis occuring now on about half of them. Once again I am beginning to question the robustness of 389-ds and the replication problems that many of us continue to report. How do we get this to be more solid? I love this product. It really is something that RH can push, but it really needs to be rock solid and with all the replication issues, well, it seems like it is not commercially ready? Any ideas/thoughts/comments? thank you Janelle -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) results from the first ldapsearch. however, the parent is there: dn: ou=csusers,cn=config objectClass: top objectClass: organizationalUnit ou: csusers -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 05/13/2015 10:34 AM, Janelle wrote: On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. Either there is some internal op that is deleting it, or there is a bug that is causing it to be removed. To see what internal operation could be doing this, you could enable internal access logging: ldapmodify -x -h consumer.host -D cn=directory manager -w password EOF dn: cn=config changetype: modify replace: nsslapd-accesslog-level nsslapd-accesslog-level: 4 EOF Then you will have to wait until the problem reoccurs Is or was the server ipa01.example.com the target of a host delete, replica delete, or cleanallruv operation? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] more replication issues
On 5/13/15 9:13 AM, Rich Megginson wrote: On 05/13/2015 10:04 AM, Janelle wrote: On 5/13/15 8:49 AM, Rich Megginson wrote: On 05/13/2015 09:40 AM, Janelle wrote: Recently I started seeing these crop up across my servers: slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) Does that entry exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config Does the parent exist? ldapsearch -xLLL -h consumer.host -D cn=directory manager -W -s base -b ou=csusers,cn=config I am finding that there does seem to be a relation to the above error and a possible CSN issue: Can't locate CSN 555131e500020019 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. I guess what concerns me is what could be causing this. We don't do a lot of changes all the time. And in answer to the question above - we seem to have last the agreement somehow: No such object (32) Is there a DEL operation in the access log for cn=Replication Manager masterAgreement1-ipa01.example.com-pki-tomcat,ou=csusers,cn=config? maybe something like # grep DEL /var/log/dirsrv/slapd-INST/access|grep -i Replication Manager nope -- none of the servers have it. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
