Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
On 05/06/2016 03:29 PM, Devin Acosta wrote: I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. Is this the page you looked at: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html If it is the same process, what exactly failed? Thanks, Mark I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base with scope subtree # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict # # 7ad08581-059911e6-b55c83a4-93228cdf + ldapsearch, sysaccounts, etc, rsinc.loc al dn: nsuniqueid=7ad08581-059911e6-b55c83a4-93228cdf+uid=ldapsearch,cn=sysaccoun ts,cn=etc,dc=rsinc,dc=local userPassword:: e1NTSEF9M3krdTh5TkdYV= = uid: ldapsearch objectClass: account objectClass: simplesecurityobject objectClass: top nsds5ReplConflict: namingConflict uid=ldapsearch,cn=sysaccounts,cn=etc,dc=rsin c,dc=local # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [dacosta@ipa1-i2x ~]$ ./ipa_check_consistency -H "ipa1-i2x.local ipa01-aws.rsinc.local" -d RSINC.LOCAL Directory Manager password: FreeIPA servers: ipa1-i2x ipa01-aws STATE === Active Users ERROR 33 FAIL Stage Users ERROR 0 FAIL Preserved Users ERROR 0 FAIL User Groups ERROR 7 FAIL Hosts ERROR 82 FAIL Host Groups ERROR 1 FAIL HBAC Rules ERROR 2 FAIL SUDO Rules ERROR 4 FAIL DNS Zones ERROR 14 FAIL LDAP Conflicts ERROR YES FAIL Anonymous BIND ERROR on FAIL Replication Status ipa02-aws 0 ipa1-i2x 0 === [dacosta@ipa1-i2x ~]$ ipa-replica-manage list ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: master ipa01-aws.rsinc.local: master ipa1-i2x.rsinc.local: master Devin Acosta Linux Certified Engineer e: de...@linuxguru.co -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
I did try to resync idm1-i2x from ipa01-aws, probably was a bad idea..
Is there any way to basically have it resync and get a fresh copy from
the other nodes that are ok?
Well it initially started when I noticed errors in the logs about having
a conflict on a record. So i was trying to get that record cleaned up. I
then though oh maybe I should just have it reload everything from
another server, and i wonder if now that's why the box is just giving
strange results.
i had ipa1-i2x.rsinc.local reload from ipa01-aws.rsinc.local, you can
see the output of the commands below about replication status. I can
still log into ipa1-i2x.rsinc.local,
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa02-aws.rsinc.local
ipa: WARNING: session memcached servers not running
ipa01-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update
started
last update ended: 1970-01-01 00:00:00+00:00
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa01-aws.rsinc.local
ipa: WARNING: session memcached servers not running
ipa02-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update
succeeded
last update ended: 2016-05-06 19:47:26+00:00
ipa1-i2x.rsinc.local: replica
last init status: 0 Total update succeeded
last init ended: 2016-05-06 18:46:29+00:00
last update status: 0 Replica acquired successfully: Incremental update
succeeded
last update ended: 2016-05-06 19:46:59+00:00
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa1-i2x.rsinc.local
ipa: WARNING: session memcached servers not running
ipa01-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 1 Can't acquire busy replica
last update ended: 1970-01-01 00:00:00+00:00
I do have these errors on (idm1-i2x) in the errors:
[06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv:
RUV [changelog max RUV] does not contain element [{replica 4
ldap://ipa01-aws.rsinc.local:389} 56e2f9e70004
572ce68100020004] which is present in RUV [database RUV]
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica dc=rsinc,dc=local
there were some differences between the changelog max RUV and the
database RUV. If there are obsolete elements in the database RUV, you
should remove them using the CLEANALLRUV task. If they are not
obsolete, you should check their status to see why there are no changes
from those servers in the changelog.
[06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv:
RUV [changelog max RUV] does not contain element [{replica 91
ldap://ipa1-i2x.rsinc.local:389} 56f02d3b005b
56f02d67005b] which is present in RUV [database RUV]
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica o=ipaca there were
some differences between the changelog max RUV and the database RUV. If
there are obsolete elements in the database RUV, you should remove them
using the CLEANALLRUV task. If they are not obsolete, you should check
their status to see why there are no changes from those servers in the
changelog.
[06/May/2016:18:48:46 +] set_krb5_creds - Could not get initial
credentials for principal [ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see
e-text))
[06/May/2016:18:48:46 +] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[06/May/2016:18:48:46 +] slapi_ldap_bind - Error: could not perform
interactive bind for id [] authentication mechanism [GSSAPI]: error -2
(Local error)
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind
with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
more information (No Kerberos credentials available))
[06/May/2016:18:48:46 +] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
[06/May/2016:18:48:46 +] - Listening on All Interfaces port 636 for
LDAPS requests
[06/May/2016:18:48:46 +] - Listening on
/var/run/slapd-RSINC-LOCAL.socket for LDAPI requests
[06/May/2016:18:48:50 +] NSMMReplicationPlugin -
agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind
with GSSAPI auth resumed
[06/May/2016:18:49:18 +] - Retry count exceeded in delete
[06/May/2016:18:49:18 +] DSRetroclPlugin - delete_changerecord:
could not delete change record 436145 (rc: 51)
Thanks
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
Please keep freeipa-users in loop
Well indeed something bad is happening with replication, did you tried
reinitialize replica? Maybe guys from DS will know what is happening.
Martin
On 06.05.2016 21:51, Devin Acosta wrote:
Martin,
Well it initially started when I noticed errors in the logs about
having a conflict on a record. So i was trying to get that record
cleaned up. I then though oh maybe I should just have it reload
everything from another server, and i wonder if now that's why the box
is just giving strange results.
i had ipa1-i2x.rsinc.local reload from ipa01-aws.rsinc.local, you can
see the output of the commands below about replication status. I can
still log into ipa1-i2x.rsinc.local,
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa02-aws.rsinc.local
ipa: WARNING: session memcached servers not running
ipa01-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental
update started
last update ended: 1970-01-01 00:00:00+00:00
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa01-aws.rsinc.local
ipa: WARNING: session memcached servers not running
ipa02-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental
update succeeded
last update ended: 2016-05-06 19:47:26+00:00
ipa1-i2x.rsinc.local: replica
last init status: 0 Total update succeeded
last init ended: 2016-05-06 18:46:29+00:00
last update status: 0 Replica acquired successfully: Incremental
update succeeded
last update ended: 2016-05-06 19:46:59+00:00
[dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa1-i2x.rsinc.local
ipa: WARNING: session memcached servers not running
ipa01-aws.rsinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 1 Can't acquire busy replica
last update ended: 1970-01-01 00:00:00+00:00
I do have these errors on (idm1-i2x) in the errors:
[06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv:
RUV [changelog max RUV] does not contain element [{replica 4
ldap://ipa01-aws.rsinc.local:389} 56e2f9e70004
572ce68100020004] which is present in RUV [database RUV]
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica dc=rsinc,dc=local
there were some differences between the changelog max RUV and the
database RUV. If there are obsolete elements in the database RUV, you
should remove them using the CLEANALLRUV task. If they are not
obsolete, you should check their status to see why there are no
changes from those servers in the changelog.
[06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv:
RUV [changelog max RUV] does not contain element [{replica 91
ldap://ipa1-i2x.rsinc.local:389} 56f02d3b005b
56f02d67005b] which is present in RUV [database RUV]
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica o=ipaca there were
some differences between the changelog max RUV and the database RUV.
If there are obsolete elements in the database RUV, you should remove
them using the CLEANALLRUV task. If they are not obsolete, you should
check their status to see why there are no changes from those servers
in the changelog.
[06/May/2016:18:48:46 +] set_krb5_creds - Could not get initial
credentials for principal [ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see
e-text))
[06/May/2016:18:48:46 +] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[06/May/2016:18:48:46 +] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[06/May/2016:18:48:46 +] NSMMReplicationPlugin -
agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind
with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1):
generic failure: GSSAPI Error: Unspecified GSS failure. Minor code
may provide more information (No Kerberos credentials available))
[06/May/2016:18:48:46 +] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
[06/May/2016:18:48:46 +] - Listening on All Interfaces port 636
for LDAPS requests
[06/May/2016:18:48:46 +] - Listening on
/var/run/slapd-RSINC-LOCAL.socket for LDAPI requests
[06/May/2016:18:48:50 +] NSMMReplicationPlugin -
agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind
with GSSAPI auth resumed
[06/May/2016:18:49:18 +] - Retry count exceeded in delete
[06/May/2016:18:49:18 +] DSRetroclPlugin - delete_changerecord:
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
On 06.05.2016 21:29, Devin Acosta wrote: I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base with scope subtree # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict # # 7ad08581-059911e6-b55c83a4-93228cdf + ldapsearch, sysaccounts, etc, rsinc.loc al dn: nsuniqueid=7ad08581-059911e6-b55c83a4-93228cdf+uid=ldapsearch,cn=sysaccoun ts,cn=etc,dc=rsinc,dc=local userPassword:: e1NTSEF9M3krdTh5TkdYV= = uid: ldapsearch objectClass: account objectClass: simplesecurityobject objectClass: top nsds5ReplConflict: namingConflict uid=ldapsearch,cn=sysaccounts,cn=etc,dc=rsin c,dc=local # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [dacosta@ipa1-i2x ~]$ ./ipa_check_consistency -H "ipa1-i2x.local ipa01-aws.rsinc.local" -d RSINC.LOCAL Directory Manager password: FreeIPA servers: ipa1-i2x ipa01-aws STATE === Active Users ERROR 33 FAIL Stage Users ERROR 0 FAIL Preserved Users ERROR 0 FAIL User Groups ERROR 7 FAIL Hosts ERROR 82 FAIL Host Groups ERROR 1 FAIL HBAC Rules ERROR 2 FAIL SUDO Rules ERROR 4 FAIL DNS Zones ERROR 14 FAIL LDAP Conflicts ERROR YES FAIL Anonymous BIND ERROR on FAIL Replication Status ipa02-aws 0 ipa1-i2x 0 === [dacosta@ipa1-i2x ~]$ ipa-replica-manage list ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: master ipa01-aws.rsinc.local: master ipa1-i2x.rsinc.local: master Devin Acosta Linux Certified Engineer e: de...@linuxguru.co hello, it is not clear to me what is wrong, do you have there conflicts? The output of command is not tool supported by freeIPA, I have no idea what is wrong. to check replication status for each IPA server run ipa-replica-manage -v list can you kinit on all replicas? can you do ldapsearch as directory manager on each server? Martin -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] nsds5ReplConflict / Replication issue!
I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base with scope subtree # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict # # 7ad08581-059911e6-b55c83a4-93228cdf + ldapsearch, sysaccounts, etc, rsinc.loc al dn: nsuniqueid=7ad08581-059911e6-b55c83a4-93228cdf+uid=ldapsearch,cn=sysaccoun ts,cn=etc,dc=rsinc,dc=local userPassword:: e1NTSEF9M3krdTh5TkdYV= = uid: ldapsearch objectClass: account objectClass: simplesecurityobject objectClass: top nsds5ReplConflict: namingConflict uid=ldapsearch,cn=sysaccounts,cn=etc,dc=rsin c,dc=local # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [dacosta@ipa1-i2x ~]$ ./ipa_check_consistency -H "ipa1-i2x.local ipa01-aws.rsinc.local" -d RSINC.LOCAL Directory Manager password: FreeIPA servers: ipa1-i2x ipa01-aws STATE === Active Users ERROR 33 FAIL Stage Users ERROR 0 FAIL Preserved Users ERROR 0 FAIL User Groups ERROR 7 FAIL Hosts ERROR 82 FAIL Host Groups ERROR 1 FAIL HBAC Rules ERROR 2 FAIL SUDO Rules ERROR 4 FAIL DNS Zones ERROR 14 FAIL LDAP Conflicts ERROR YES FAIL Anonymous BIND ERROR on FAIL Replication Status ipa02-aws 0 ipa1-i2x 0 === [dacosta@ipa1-i2x ~]$ ipa-replica-manage list ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: master ipa01-aws.rsinc.local: master ipa1-i2x.rsinc.local: master Devin Acosta Linux Certified Engineer e: de...@linuxguru.co -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
