Hi Rich,
Thanks for your reply.
Please find the logging from the problems below.
The serverb55 is one of 2 servers in a multiple masters configuration that
consists of serverb55 and serverb05.
The problem I inititially had was that I had 2 entries that could not be
deleted serverb55.
Here's logging from the access file.
=======================================================================
access.20100614-092820:[15/Jun/2010:09:20:49 +0200] conn=342177 op=7 SRCH
base="uid=dbeijk, ou=people, dc=directory,dc=intern" scope=0
filter="(objectClass=*)" attrs=ALL
access.20100614-092820:[15/Jun/2010:09:20:49 +0200] conn=342177 op=7 RESULT
err=0 tag=101 nentries=1 etime=0
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=8 SRCH
base="uid=dbeijk, ou=people, dc=directory,dc=intern" scope=1
filter="(objectClass=*)" attrs="objectClass"
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=8 RESULT
err=0 tag=101 nentries=0 etime=0 notes=U
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=9 DEL
dn="uid=dbeijk, ou=people, dc=directory,dc=intern"
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=9 RESULT
err=1 tag=107 nentries=0 etime=0 csn=4c172a21000000370000
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=10 SRCH
base="uid=dbeijk, ou=people, dc=directory,dc=intern" scope=1
filter="(objectClass=*)" attrs="objectClass"
access.20100614-092820:[15/Jun/2010:09:22:08 +0200] conn=342177 op=10 RESULT
err=0 tag=101 nentries=0 etime=0 notes=U
=======================================================================
LDAP error 1 i found means 'unwiling to perform'. First I thought something
might be wrong with the entry itself.
The error log found in the error log from the serverb55 I've added below
seemed to point in that direction.
When I logged on the the other ldapserver, serverb05, I tried to delete the
same entry to see if this slapd had the same issue but here it worked.
Replicating the delete didn't. The following error was logged to the
errorlog of this :
========================================================================
[15/Jun/2010:09:35:17 +0200] NSMMReplicationPlugin - agmt="cn=serverb55"
(serverb55:636): Consumer failed to replay change (uniqueid
a276337c-5dc511df-852cfef8-667fa4d4, CSN 4c172d36000000050000): Operations
error. Will retry later.
=======================================================================
So there seemed to be a problem with the serverb55 only.
Since I assumed the database got somehow corrupt or inconsistent I've tried
the following steps to try and recreate the database or had it checked in
order to get it right again.
First there's the errors from the account that could not be deleted.
I 'reinitialised the consumer' from the working serverb05 to the problematic
serverb55.
Then I restarted the slapd.
Made an export of the database and imported that.
Slapd stopped the database.
Please find the logging from /var/log/dirsrv/slapd-serverb55/errors from the
actions leading to the problem of the fatal server stop.
======================================================================
CentOS-Directory/8.1.0
B2009.134.1334
serverb55:636
(/etc/dirsrv/slapd-serverb55)
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "uidNumber" required by object
class "posixAccount"
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "gidNumber" required by object
class "posixAccount"
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" -- attribute "uidNumber" not allowed
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "uid" required by object class
"posixAccount"
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "cn" required by object class
"posixAccount"
[15/Jun/2010:09:22:58 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "homeDirectory" required by object
class "posixAccount"
[15/Jun/2010:09:23:04 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "uidNumber" required by object
class "posixAccount"
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "uidNumber" required by object
class "posixAccount"
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "gidNumber" required by object
class "posixAccount"
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" required attribute "objectclass" missing
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" required attribute "objectclass" missing
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "uid" required by object class
"posixAccount"
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "cn" required by object class
"posixAccount"
[15/Jun/2010:09:23:18 +0200] - Entry "uid=dbeijk, ou=People,
dc=directory,dc=intern" missing attribute "homeDirectory" required by object
class "posixAccount"
[15/Jun/2010:09:24:56 +0200] - Entry "uid=DEL *.*, ou=People,
dc=directory,dc=intern" missing attribute "homeDirectory" required by object
class "posixAccount"
[15/Jun/2010:09:50:20 +0200] - Entry "cn=wchiman, ou=people,
dc=directory,dc=intern" -- attribute "uidNumber" not allowed
[15/Jun/2010:10:12:43 +0200] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=directory,dc=intern is going
offline; disabling replication
[15/Jun/2010:10:12:43 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:10:12:43 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:10:12:43 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:10:12:43 +0200] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database
[15/Jun/2010:10:12:48 +0200] - import userRoot: Workers finished; cleaning
up...
[15/Jun/2010:10:12:49 +0200] - import userRoot: Workers cleaned up.
[15/Jun/2010:10:12:49 +0200] - import userRoot: Indexing complete.
Post-processing...
[15/Jun/2010:10:12:49 +0200] - import userRoot: Flushing caches...
[15/Jun/2010:10:12:49 +0200] - import userRoot: Closing files...
[15/Jun/2010:10:12:49 +0200] - import userRoot: Import complete. Processed
4849 entries in 6 seconds. (808.17 entries/sec)
[15/Jun/2010:10:12:49 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:10:12:49 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:10:12:49 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:10:12:49 +0200] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=directory,dc=intern is coming
online; enabling replication
[15/Jun/2010:10:12:49 +0200] NSMMReplicationPlugin - replica_reload_ruv:
Warning: new data for replica dc=directory,dc=intern does not match the data
in the changelog.
Recreating the changelog file. This could affect replication with
replica's consumers in which case the consumers should be reinitialized.
[15/Jun/2010:10:12:49 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:10:55:43 +0200] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=directory,dc=intern is going
offline; disabling replication
[15/Jun/2010:10:55:43 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:10:55:43 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:10:55:43 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:10:55:43 +0200] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database
[15/Jun/2010:10:55:49 +0200] - import userRoot: Workers finished; cleaning
up...
[15/Jun/2010:10:55:49 +0200] - import userRoot: Workers cleaned up.
[15/Jun/2010:10:55:49 +0200] - import userRoot: Indexing complete.
Post-processing...
[15/Jun/2010:10:55:49 +0200] - import userRoot: Flushing caches...
[15/Jun/2010:10:55:49 +0200] - import userRoot: Closing files...
[15/Jun/2010:10:55:49 +0200] - import userRoot: Import complete. Processed
4850 entries in 5 seconds. (970.00 entries/sec)
[15/Jun/2010:10:55:49 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:10:55:49 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:10:55:49 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:10:55:49 +0200] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=directory,dc=intern is coming
online; enabling replication
[15/Jun/2010:10:55:49 +0200] NSMMReplicationPlugin - replica_reload_ruv:
Warning: new data for replica dc=directory,dc=intern does not match the data
in the changelog.
Recreating the changelog file. This could affect replication with
replica's consumers in which case the consumers should be reinitialized.
[15/Jun/2010:10:55:49 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:10:59:57 +0200] - slapd shutting down - signaling operation
threads
[15/Jun/2010:10:59:57 +0200] - slapd shutting down - waiting for 26 threads
to terminate
[15/Jun/2010:10:59:57 +0200] - slapd shutting down - closing down internal
subsystems and plugins
[15/Jun/2010:10:59:58 +0200] - Waiting for 4 database threads to stop
[15/Jun/2010:10:59:59 +0200] - All database threads now stopped
[15/Jun/2010:10:59:59 +0200] - slapd stopped.
CentOS-Directory/8.1.0 B2009.134.1334
<host>:<port> (/etc/dirsrv/slapd-serverb55)
[15/Jun/2010:11:00:01 +0200] - Entry "cn=schema" single-valued attribute
"modifyTimestamp" has multiple values
CentOS-Directory/8.1.0 B2009.134.1334
serverb55:636 (/etc/dirsrv/slapd-serverb55)
[15/Jun/2010:11:00:01 +0200] - CentOS-Directory/8.1.0 B2009.134.1334
starting up
[15/Jun/2010:11:00:01 +0200] - I'm resizing my cache now...cache was
20000000 and is now 8000000
[15/Jun/2010:11:00:01 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:11:00:01 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:11:00:01 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:11:00:01 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:11:00:01 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:11:00:01 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:11:00:01 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:11:00:01 +0200] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: data for replica
dc=directory,dc=intern was reloaded and it no longer matches the data in the
changelog (replica data > changelog). Recreating the changelog file. This
could affect replication with replica's consumers in which case the
consumers should be reinitialized.
[15/Jun/2010:11:00:01 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:11:00:01 +0200] - slapd started. Listening on All Interfaces
port 389 for LDAP requests
[15/Jun/2010:11:00:01 +0200] - Listening on All Interfaces port 636 for
LDAPS requests
[15/Jun/2010:11:29:59 +0200] - slapd shutting down - signaling operation
threads
[15/Jun/2010:11:29:59 +0200] - slapd shutting down - closing down internal
subsystems and plugins
[15/Jun/2010:11:30:00 +0200] - Waiting for 4 database threads to stop
[15/Jun/2010:11:30:00 +0200] - All database threads now stopped
[15/Jun/2010:11:30:00 +0200] - slapd stopped.
CentOS-Directory/8.1.0 B2009.134.1334
<host>:<port> (/etc/dirsrv/slapd-serverb55)
[15/Jun/2010:11:30:03 +0200] - Entry "cn=schema" single-valued attribute
"modifyTimestamp" has multiple values
CentOS-Directory/8.1.0 B2009.134.1334
serverb55:636 (/etc/dirsrv/slapd-serverb55)
[15/Jun/2010:11:30:03 +0200] - CentOS-Directory/8.1.0 B2009.134.1334
starting up
[15/Jun/2010:11:30:03 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:11:30:03 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:11:30:03 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:11:30:03 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:11:30:03 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:11:30:03 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:11:30:03 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:11:30:03 +0200] - skipping cos definition
cn=nsAccountInactivation_cos,dc=directory,dc=intern--no templates found
[15/Jun/2010:11:30:03 +0200] - slapd started. Listening on All Interfaces
port 389 for LDAP requests
[15/Jun/2010:11:30:03 +0200] - Listening on All Interfaces port 636 for
LDAPS requests
[15/Jun/2010:11:40:44 +0200] - Beginning export of 'userroot'
[15/Jun/2010:11:40:44 +0200] - export userRoot: Processed 139 entries
(100%).
[15/Jun/2010:11:40:44 +0200] - Export finished.
[15/Jun/2010:11:46:12 +0200] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=directory,dc=intern is going
offline; disabling replication
[15/Jun/2010:11:46:12 +0200] - attrcrypt_unwrap_key: failed to unwrap key
for cipher AES
[15/Jun/2010:11:46:12 +0200] - Failed to retrieve key for cipher AES in
attrcrypt_cipher_init
[15/Jun/2010:11:46:12 +0200] - Failed to initialize cipher AES in
attrcrypt_init
[15/Jun/2010:11:46:12 +0200] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database
[15/Jun/2010:11:46:14 +0200] - libdb: page 1: illegal page type or format
[15/Jun/2010:11:46:14 +0200] - libdb: PANIC: Invalid argument
[15/Jun/2010:11:46:14 +0200] - FATAL ERROR at by MCC ou=people dc=directory
dc=intern (77); server stopping as database recovery needed.
CentOS-Directory/8.1.0 B2009.134.1334
<host>:<port> (/etc/dirsrv/slapd-serverb55
======================================================================
Finally my questions :
What could be the cause of the problem ?
What would be the best procedure to get the serverb55 up and running again ?
Thanks for any advise.
Regards,
Mark
=======
On Tue, Jun 15, 2010 at 7:04 PM, Rich Megginson <rmegg...@redhat.com> wrote:
> mark benschop wrote:
> > Hi All,
> >
> > I'm having a problem on a CentOs Directory Server 8.1 multiple master
> > setup.
> > The database of one of the servers has been marked as corrupt and has
> > been brought offline by the Directory Server.
> Can you post any relevant error messages from the error log of the server?
> > Ldapclients querying the ldapserver for e.g. loggin in of users get an
> > errormessage, effectively disabling users to log in.
> What error message?
> >
> > I'm wondering what the best method is to recover from this situation.
> > I can think of a few :
> > 1) Starting the ldapserver, deleting the database, recreating it and
> > restoring a backup.
>
> > 2) Starting the ldapserver, deleting the database and reinitialising
> > the server from the other master.
> If you reinitialize the problem server from another server, you don't
> need to delete the database, reinit will do that for you.
> >
> > Can anyone give me some hints if this wil work or would another
> > approach be better ?
> >
> > Thanks for your advise,
> > Mark
> > ------------------------------------------------------------------------
> >
> > --
> > 389 users mailing list
> > 389-users@lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users
>
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
