Re: [OF-users] Problems with OF clustering

Sat, 29 Mar 2008 17:03:19 -0700

Writig ths mail is a really nice job because I feel f...ed in the whole line! I'm so angry! 

Due to your posting I did a manually upgrade of the modules:
drbd, heartbeat, kernel, drbd-kernel (all this modules found in repository) All went without errors, except the update drbd-kernel... 

[EMAIL PROTECTED] ~]# conary update drbd
no new troves were found

conary update drbd-kernel
Including extra troves to resolve dependencies:
    kernel:runtime=2.6.19.7-0.1.1-1Applying update job 1 of 2:
    Install drbd-kernel(:lib)=0.7.23-1-1[~!drbd-kernel.smp]
    Install kernel:runtime=2.6.19.7-0.1.1-1[~!kernel.smp]
Creating database transaction (3 of 3)...changeset cannot be applied:
applying update would cause errors:/lib/modules/ 2.6.19.7-0.1.x86.i686.cmov/kernel/drivers/block/drbd.ko conflicts with a file owned by kernel:runtime=/[EMAIL PROTECTED]:devel//1// [EMAIL PROTECTED]:devel/2.6.19.7-0.1.1-1[~!kernel.debug,~! kernel.debugdata,~!kernel.numa,~!kernel.smp is: x86 (cmov,i486,i586,i686)] 

I did a conary update kernel and got something like this at start:

drbd not loadeable
Ok, with kernel of 2.6.22.x drbd doesn't start anymore, because of "no drbd found". 
insmod drbd doesn`t work.
So, now what the heck is going wrong???????? Am I totally silly or is there anyone trying to drive me loco?????
I think, there is more than one error! Now I have a totally rotten system!
Before updating this things nothing went really well, but now its all died! Please can anyone tell me whats wrong? So, when you tell me, that my kernel and tools are too old please could you give me a little useful help to get this solved? If the system gives me a clear and loud "no new troves" when make a conary updateall, I think there is all at newest as told.
Searcing for drbd tools gives absolutely nothing. Which tools please? drbd tells something about "drbd userland tools". Didn't find any of them. On linux-ha.org I read only about drbd, there is nothing about drbd-tools. Is this a special openfiler-thing which I have to know about? Sorry for asking this but I'm not programmer and don't know about all this things! Documentation about all this is a bit small at all...
So is the only thing to do install all new, maybe OF 2.3beta? Is this version running???? My opinion is that this version (2.2) is more buggy than my car! I'm totally disappointed! Can anyone help me with anything useable? 

uname -a:
Linux SAN-01.local 2.6.19.7-0.3.smp.pae.gcc3.4.x86.i686 #1 SMP Thu Apr 12 01:53:56 EDT 2007 i686 i686 i386 GNU/Linux 

[EMAIL PROTECTED] ~]# conary updateall
no new troves were found

And to make all complete, system-log when starting:

drbd is updated to v8.0.7, but at start telling wrong version!
When doing the update it did't tell any errors!
Next is that 2 of the drbds are possibly broken due to "out of memory" Whatever this means! Its really exciting all! 2 Weeks of working for abolutely nothing!
Oh yes, the astonishing "No usable activity log found" and kicking all syncing back to 0,1%! Yeah, it was about 5% before the update!
At last, my all loved "BUG! md_sync_timer expired! Worker calls drbd_md_sync()."
Mar 30 00:06:11 SAN-01 kernel: drbd: initialised. Version: 8.0.2 (api: 86/proto:86) Mar 30 00:06:11 SAN-01 kernel: drbd: SVN Revision: 2844 build by [EMAIL PROTECTED], 2007-04-27 17:08:12 Mar 30 00:06:11 SAN-01 kernel: drbd: registered as block device major 147 
Mar 30 00:06:11 SAN-01 kernel: drbd: minor_table @ 0xf73ba5c0
Mar 30 00:06:11 SAN-01 kernel: drbd0: disk( Diskless -> Attaching )
Mar 30 00:06:11 SAN-01 kernel: klogd 1.4.1, ---------- state change ---------- Mar 30 00:06:11 SAN-01 kernel: drbd0: Found 6 transactions (57 active extents) in activity log. Mar 30 00:06:11 SAN-01 kernel: drbd0: max_segment_size ( = BIO size ) = 32768 Mar 30 00:06:11 SAN-01 kernel: drbd0: drbd_bm_resize called with capacity == 626432 Mar 30 00:06:11 SAN-01 kernel: drbd0: resync bitmap: bits=78304 words=2448 
Mar 30 00:06:11 SAN-01 kernel: drbd0: size = 305 MB (313216 KB)
Mar 30 00:06:11 SAN-01 kernel: drbd0: reading of bitmap took 1 jiffies
Mar 30 00:06:11 SAN-01 kernel: drbd0: recounting of set bits took additional 0 jiffies Mar 30 00:06:11 SAN-01 kernel: drbd0: 0 KB marked out-of-sync by on disk bit-map. Mar 30 00:06:11 SAN-01 kernel: drbd0: Marked additional 157 MB as out- of-sync based on AL. 
Mar 30 00:06:11 SAN-01 kernel: drbd0: disk( Attaching -> UpToDate )
Mar 30 00:06:11 SAN-01 kernel: drbd0: Writing meta data super block now.
Mar 30 00:06:11 SAN-01 kernel: drbd1: disk( Diskless -> Attaching )
Mar 30 00:06:11 SAN-01 kernel: drbd1: Found 4 transactions (65 active extents) in activity log. Mar 30 00:06:11 SAN-01 kernel: drbd1: max_segment_size ( = BIO size ) = 32768 Mar 30 00:06:11 SAN-01 kernel: drbd1: drbd_bm_resize called with capacity == 4294827920 Mar 30 00:06:11 SAN-01 kernel: drbd1: resync bitmap: bits=536853490 words=16776672 
Mar 30 00:06:11 SAN-01 kernel: drbd1: size = 2047 GB (2147413960 KB)
Mar 30 00:06:11 SAN-01 kernel: drbd1: reading of bitmap took 121 jiffies
Mar 30 00:06:11 SAN-01 kernel: drbd1: recounting of set bits took additional 34 jiffies Mar 30 00:06:11 SAN-01 kernel: drbd1: 1823 GB marked out-of-sync by on disk bit-map. Mar 30 00:06:11 SAN-01 kernel: drbd1: Marked additional 256 MB as out- of-sync based on AL. Mar 30 00:06:12 SAN-01 kernel: drbd1: disk( Attaching -> UpToDate ) pdsk( DUnknown -> Outdated ) 
Mar 30 00:06:12 SAN-01 kernel: drbd1: Writing meta data super block now.
Mar 30 00:06:12 SAN-01 kernel: drbd2: disk( Diskless -> Attaching )
Mar 30 00:06:12 SAN-01 kernel: drbd2: No usable activity log found.
Mar 30 00:06:12 SAN-01 kernel: drbd2: max_segment_size ( = BIO size ) = 32768 Mar 30 00:06:12 SAN-01 kernel: drbd2: drbd_bm_resize called with capacity == 4294827920 Mar 30 00:06:12 SAN-01 kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size. Mar 30 00:06:12 SAN-01 kernel: drbd2: bitmap: failed to vmalloc 67106692 bytes Mar 30 00:06:12 SAN-01 kernel: drbd2: OUT OF MEMORY! Could not allocate bitmap! Set device size => 0 
Mar 30 00:06:12 SAN-01 kernel: drbd2: size = 0 KB (0 KB)
Mar 30 00:06:12 SAN-01 kernel: drbd2: Marked additional 0 KB as out- of-sync based on AL. 
Mar 30 00:06:12 SAN-01 kernel: drbd2: disk( Attaching -> UpToDate )
Mar 30 00:06:12 SAN-01 kernel: drbd2: Writing meta data super block now.
Mar 30 00:06:12 SAN-01 kernel: drbd3: disk( Diskless -> Attaching )
Mar 30 00:06:12 SAN-01 kernel: drbd3: No usable activity log found.
Mar 30 00:06:12 SAN-01 kernel: drbd3: max_segment_size ( = BIO size ) = 32768 Mar 30 00:06:12 SAN-01 kernel: drbd3: drbd_bm_resize called with capacity == 4294827920 Mar 30 00:06:12 SAN-01 kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size. Mar 30 00:06:12 SAN-01 kernel: drbd3: bitmap: failed to vmalloc 67106692 bytes Mar 30 00:06:12 SAN-01 kernel: drbd3: OUT OF MEMORY! Could not allocate bitmap! Set device size => 0 
Mar 30 00:06:12 SAN-01 kernel: drbd3: size = 0 KB (0 KB)
Mar 30 00:06:12 SAN-01 kernel: drbd3: Marked additional 0 KB as out- of-sync based on AL. 
Mar 30 00:06:12 SAN-01 kernel: drbd3: disk( Attaching -> UpToDate )
Mar 30 00:06:12 SAN-01 kernel: drbd3: Writing meta data super block now.
Mar 30 00:06:12 SAN-01 kernel: drbd4: disk( Diskless -> Attaching )
Mar 30 00:06:12 SAN-01 kernel: drbd4: No usable activity log found.
Mar 30 00:06:12 SAN-01 kernel: drbd4: max_segment_size ( = BIO size ) = 32768 Mar 30 00:06:12 SAN-01 kernel: drbd4: drbd_bm_resize called with capacity == 786788800 Mar 30 00:06:12 SAN-01 kernel: drbd4: resync bitmap: bits=98348600 words=3073394 
Mar 30 00:06:12 SAN-01 kernel: drbd4: size = 375 GB (393394400 KB)
Mar 30 00:06:12 SAN-01 kernel: drbd4: reading of bitmap took 21 jiffies
Mar 30 00:06:12 SAN-01 kernel: drbd4: recounting of set bits took additional 7 jiffies Mar 30 00:06:12 SAN-01 kernel: drbd4: 264 GB marked out-of-sync by on disk bit-map. Mar 30 00:06:12 SAN-01 kernel: drbd4: Marked additional 0 KB as out- of-sync based on AL. Mar 30 00:06:12 SAN-01 kernel: drbd4: disk( Attaching -> UpToDate ) pdsk( DUnknown -> Outdated ) 
Mar 30 00:06:12 SAN-01 kernel: drbd4: Writing meta data super block now.
Mar 30 00:06:12 SAN-01 kernel: drbd0: conn( StandAlone -> Unconnected )
Mar 30 00:06:12 SAN-01 kernel: drbd0: receiver (re)started
Mar 30 00:06:12 SAN-01 kernel: drbd0: conn( Unconnected -> WFConnection ) 
Mar 30 00:06:12 SAN-01 kernel: drbd1: conn( StandAlone -> Unconnected )
Mar 30 00:06:12 SAN-01 kernel: drbd1: receiver (re)started
Mar 30 00:06:12 SAN-01 kernel: drbd1: conn( Unconnected -> WFConnection ) 
Mar 30 00:06:12 SAN-01 kernel: drbd2: conn( StandAlone -> Unconnected )
Mar 30 00:06:12 SAN-01 kernel: drbd2: receiver (re)started
Mar 30 00:06:12 SAN-01 kernel: drbd2: conn( Unconnected -> WFConnection ) 
Mar 30 00:06:12 SAN-01 kernel: drbd3: conn( StandAlone -> Unconnected )
Mar 30 00:06:12 SAN-01 kernel: drbd3: receiver (re)started
Mar 30 00:06:12 SAN-01 kernel: drbd3: conn( Unconnected -> WFConnection ) 
Mar 30 00:06:13 SAN-01 kernel: drbd4: conn( StandAlone -> Unconnected )
Mar 30 00:06:13 SAN-01 kernel: drbd4: receiver (re)started
Mar 30 00:06:13 SAN-01 kernel: drbd4: conn( Unconnected -> WFConnection ) Mar 30 00:16:18 SAN-01 sshd(pam_unix)[3096]: session opened for user root by root(uid=0) 
Mar 30 00:20:35 SAN-01 kernel: drbd0: role( Secondary -> Primary )
Mar 30 00:20:40 SAN-01 kernel: drbd1: role( Secondary -> Primary )
Mar 30 00:20:45 SAN-01 kernel: drbd2: role( Secondary -> Primary )
Mar 30 00:20:49 SAN-01 kernel: drbd3: role( Secondary -> Primary )
Mar 30 00:20:53 SAN-01 kernel: drbd4: role( Secondary -> Primary )
Mar 30 00:23:50 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Down Mar 30 00:23:51 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex Mar 30 00:23:51 SAN-01 kernel: e1000: eth1: e1000_watchdog: 10/100 speed: disabling TSO Mar 30 00:23:55 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Down Mar 30 00:23:57 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex Mar 30 00:25:21 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Down Mar 30 00:25:24 SAN-01 kernel: e1000: eth1: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex Mar 30 00:25:48 SAN-01 kernel: drbd0: conn( WFConnection -> WFReportParams ) Mar 30 00:25:48 SAN-01 kernel: drbd0: Handshake successful: DRBD Network Protocol version 86 Mar 30 00:25:48 SAN-01 kernel: drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate ) 
Mar 30 00:25:48 SAN-01 kernel: drbd0: Writing meta data super block now.
Mar 30 00:25:48 SAN-01 kernel: drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent ) Mar 30 00:25:48 SAN-01 kernel: drbd0: Began resync as SyncSource (will sync 161664 KB [40416 bits set]). 
Mar 30 00:25:48 SAN-01 kernel: drbd0: Writing meta data super block now.
Mar 30 00:25:49 SAN-01 kernel: drbd2: conn( WFConnection -> WFReportParams ) Mar 30 00:25:51 SAN-01 kernel: drbd2: Handshake successful: DRBD Network Protocol version 86 Mar 30 00:25:51 SAN-01 kernel: drbd3: conn( WFConnection -> WFReportParams ) Mar 30 00:25:51 SAN-01 kernel: drbd3: Handshake successful: DRBD Network Protocol version 86 Mar 30 00:25:51 SAN-01 kernel: drbd4: conn( WFConnection -> WFReportParams ) Mar 30 00:25:51 SAN-01 kernel: drbd1: conn( WFConnection -> WFReportParams ) Mar 30 00:25:51 SAN-01 kernel: drbd1: Handshake successful: DRBD Network Protocol version 86 Mar 30 00:25:51 SAN-01 kernel: drbd3: drbd_bm_resize called with capacity == 4294827920 Mar 30 00:25:51 SAN-01 kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size. Mar 30 00:25:51 SAN-01 kernel: drbd3: bitmap: failed to vmalloc 67106692 bytes Mar 30 00:25:51 SAN-01 kernel: drbd3: OUT OF MEMORY! Could not allocate bitmap! Set device size => 0 
Mar 30 00:25:51 SAN-01 kernel: drbd3: size = 0 KB (0 KB)
Mar 30 00:25:51 SAN-01 kernel: drbd3: Becoming sync source due to disk states. 
Mar 30 00:25:51 SAN-01 kernel: drbd3: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd4: Handshake successful: DRBD Network Protocol version 86 Mar 30 00:25:51 SAN-01 kernel: drbd3: drbd_bm_set_all: (!b->bm) in / home/buildof/conary/openfiler/builds/kernel/drbd-8.0.2/drbd/ drbd_bitmap.c:617 
Mar 30 00:25:51 SAN-01 kernel: drbd3: writing of bitmap took 0 jiffies
Mar 30 00:25:51 SAN-01 kernel: drbd3: 0 KB marked out-of-sync by on disk bit-map. Mar 30 00:25:51 SAN-01 kernel: drbd3: 0 KB now marked out-of-sync by on disk bit-map. 
Mar 30 00:25:51 SAN-01 kernel: drbd3: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd3: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Inconsistent ) 
Mar 30 00:25:51 SAN-01 kernel: drbd3: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd3: conn( WFBitMapS -> SyncSource )
Mar 30 00:25:51 SAN-01 kernel: drbd3: Began resync as SyncSource (will sync 0 KB [0 bits set]). Mar 30 00:25:51 SAN-01 kernel: drbd3: Resync done (total 1 sec; paused 0 sec; 0 K/sec) Mar 30 00:25:51 SAN-01 kernel: drbd3: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) 
Mar 30 00:25:51 SAN-01 kernel: drbd3: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd2: drbd_bm_resize called with capacity == 4294827920 Mar 30 00:25:51 SAN-01 kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size. Mar 30 00:25:51 SAN-01 kernel: drbd2: bitmap: failed to vmalloc 67106692 bytes Mar 30 00:25:51 SAN-01 kernel: drbd2: OUT OF MEMORY! Could not allocate bitmap! Set device size => 0 
Mar 30 00:25:51 SAN-01 kernel: drbd2: size = 0 KB (0 KB)
Mar 30 00:25:51 SAN-01 kernel: drbd2: Becoming sync source due to disk states. 
Mar 30 00:25:51 SAN-01 kernel: drbd2: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd2: drbd_bm_set_all: (!b->bm) in / home/buildof/conary/openfiler/builds/kernel/drbd-8.0.2/drbd/ drbd_bitmap.c:617 
Mar 30 00:25:51 SAN-01 kernel: drbd2: writing of bitmap took 0 jiffies
Mar 30 00:25:51 SAN-01 kernel: drbd2: 0 KB marked out-of-sync by on disk bit-map. Mar 30 00:25:51 SAN-01 kernel: drbd2: 0 KB now marked out-of-sync by on disk bit-map. 
Mar 30 00:25:51 SAN-01 kernel: drbd2: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd2: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Inconsistent ) 
Mar 30 00:25:51 SAN-01 kernel: drbd2: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 kernel: drbd1: Becoming sync source due to disk states. Mar 30 00:25:51 SAN-01 kernel: drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Inconsistent ) Mar 30 00:25:51 SAN-01 kernel: drbd4: Becoming sync source due to disk states. Mar 30 00:25:51 SAN-01 kernel: drbd4: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Inconsistent ) 
Mar 30 00:25:51 SAN-01 kernel: drbd2: conn( WFBitMapS -> SyncSource )
Mar 30 00:25:51 SAN-01 kernel: drbd2: Began resync as SyncSource (will sync 0 KB [0 bits set]). Mar 30 00:25:51 SAN-01 kernel: drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec) Mar 30 00:25:51 SAN-01 kernel: drbd2: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) 
Mar 30 00:25:51 SAN-01 kernel: drbd2: Writing meta data super block now.
Mar 30 00:25:51 SAN-01 rc: Starting drbd:  succeeded
Mar 30 00:25:52 SAN-01 logd: [3212]: info: logd started with default configuration. Mar 30 00:25:52 SAN-01 logd: [3213]: info: G_main_add_SignalHandler: Added signal handler for signal 15 Mar 30 00:25:52 SAN-01 logd: [3212]: info: G_main_add_SignalHandler: Added signal handler for signal 15 
Mar 30 00:25:52 SAN-01 heartbeat: [3292]: info: Version 2 support: false
Mar 30 00:25:52 SAN-01 heartbeat: [3292]: WARN: Logging daemon is disabled --enabling logging daemon is recommended Mar 30 00:25:52 SAN-01 heartbeat: [3292]: info: ************************** Mar 30 00:25:52 SAN-01 heartbeat: [3292]: info: Configuration validated. Starting heartbeat 2.1.1 
Mar 30 00:25:52 SAN-01 heartbeat: [3293]: info: heartbeat: version 2.1.1
Mar 30 00:25:53 SAN-01 smartd[3299]: smartd version 5.33 [i686-pc- linux-gnu] Copyright (C) 2002-4 Bruce Allen Mar 30 00:25:53 SAN-01 smartd[3299]: Home page is http:// smartmontools.sourceforge.net/ Mar 30 00:25:53 SAN-01 smartd[3299]: Opened configuration file /etc/ smartd.conf Mar 30 00:25:53 SAN-01 smartd[3299]: Configuration file /etc/ smartd.conf parsed. 
Mar 30 00:25:53 SAN-01 smartd[3299]: Device: /dev/hda, opened
Mar 30 00:25:53 SAN-01 smartd[3299]: Device: /dev/hda, not found in smartd database. Mar 30 00:25:53 SAN-01 smartd[3299]: Device: /dev/hda, is SMART capable. Adding to "monitor" list. 
Mar 30 00:25:53 SAN-01 smartd[3299]: Monitoring 1 ATA and 0 SCSI devices
Mar 30 00:25:54 SAN-01 smartd: smartd startup succeeded
Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: Heartbeat generation: 8
Mar 30 00:25:54 SAN-01 smartd[3301]: smartd has fork()ed into background mode. New PID=3301. Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: G_main_add_TriggerHandler: Added signal manual handler Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: G_main_add_TriggerHandler: Added signal manual handler Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: Removing /var/run/ heartbeat/rsctmp failed, recreating. Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1 Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1 Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: G_main_add_SignalHandler: Added signal handler for signal 17 
Mar 30 00:25:54 SAN-01 xinetd: xinetd startup succeeded
Mar 30 00:25:54 SAN-01 heartbeat: [3293]: info: Local status now set to: 'up'
Mar 30 00:25:56 SAN-01 heartbeat: [3293]: info: Comm_now_up(): updating status to active Mar 30 00:25:56 SAN-01 heartbeat: [3293]: info: Local status now set to: 'active' Mar 30 00:25:56 SAN-01 heartbeat: [3293]: WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 580 ms (> 50 ms) (GSource: 0x80f27f0) Mar 30 00:25:56 SAN-01 heartbeat: [3293]: info: Status update for node san-02.local: status active Mar 30 00:25:56 SAN-01 heartbeat: [3293]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 580 ms (> 100 ms) before being called (GSource: 0x80ee080) Mar 30 00:25:56 SAN-01 heartbeat: [3293]: info: G_SIG_dispatch: started at 1718186851 should have started at 1718186793 Mar 30 00:25:56 SAN-01 harc[3419]: info: Running /etc/ha.d/rc.d/ status status
ar 30 00:26:06 SAN-01 heartbeat: [3293]: info: local resource transition completed. Mar 30 00:26:06 SAN-01 heartbeat: [3293]: info: Initial resource acquisition complete (T_RESOURCES(us)) Mar 30 00:26:06 SAN-01 heartbeat: [3293]: info: remote resource transition completed. Mar 30 00:26:06 SAN-01 heartbeat: [3747]: ERROR: pclose(/usr/lib/ heartbeat/ResourceManager listkeys medpacs-san-01.local) exited with return code 127 Mar 30 00:26:06 SAN-01 heartbeat: [3747]: ERROR: [/usr/lib/heartbeat/ ResourceManager listkeys medpacs-san-01.local] exited with return code 127 Mar 30 00:26:06 SAN-01 heartbeat: [3747]: info: No local resources [/ usr/lib/heartbeat/ResourceManager listkeys medpacs-san-01.local] to acquire. 
Mar 30 00:26:13 SAN-01 kernel: drbd4: Writing meta data super block now.
Mar 30 00:26:13 SAN-01 kernel: drbd4: BUG! md_sync_timer expired! Worker calls drbd_md_sync(). 
Mar 30 00:26:25 SAN-01 kernel: drbd4: conn( WFBitMapS -> SyncSource )
Mar 30 00:26:25 SAN-01 kernel: drbd4: Began resync as SyncSource (will sync 277215456 KB [69303864 bits set]). 
Mar 30 00:26:25 SAN-01 kernel: drbd4: Writing meta data super block now.
Mar 30 00:27:15 SAN-01 kernel: drbd1: Writing meta data super block now.
Mar 30 00:27:15 SAN-01 kernel: drbd1: BUG! md_sync_timer expired! Worker calls drbd_md_sync(). 


igor ([EMAIL PROTECTED])
--------------------------------


Am 28.03.2008 um 18:32 schrieb Rafiu Fakunle:
Igor, you're using a very old version of the kernel and drbd tools. You need to update your system. (sorry for top posting) 

R.

Igor wrote:

New things on drbd:

[EMAIL PROTECTED] ~]# drbdsetup /dev/drbd0 pri
Mary -o
[EMAIL PROTECTED] ~]# drbdsetup /dev/drbd1 pri
Mary -o
[EMAIL PROTECTED] ~]# drbdsetup /dev/drbd2 pri
Mary -o
No response from the DRBD driver! Is the module loaded?Error code 946605981 unknown.You should updated the drbd userland tools. 
[EMAIL PROTECTED] ~]# drbdsetup /dev/drbd3 pri
Mary -o
No response from the DRBD driver! Is the module loaded?Error code 110620099 unknown.You should updated the drbd userland tools. 
[EMAIL PROTECTED] ~]# drbdsetup /dev/drbd4 pri
Mary -o
No response from the DRBD driver! Is the module loaded?Error code 162624575 unknown.You should updated the drbd userland tools. 
[EMAIL PROTECTED] ~]# service drbd status
drbd driver loaded OK; device status:version: 8.0.2 (api:86/proto: 86)SVN Revision: 2844 build by [EMAIL PROTECTED], 2007-04-27 17:08:12 
0: cs:SyncSource st:Pri
Mary/Secondary ds:UpToDate/Inconsistent C r---
        ns:66608 nr:0 dw:0 dr:74592 al:0 bm:4 lo:1 pe:7 ua:250 ap:0
        [====>...............] sync'ed: 22.1% (246816/313216)K
        finish: 0:03:25 speed: 1,168 (884) K/sec
resync: used:1/31 hits:4401 misses:5 starving:0 dirty:0 changed:5 act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0 
1: cs:SyncSource st:Pri
Mary/Secondary ds:UpToDate/Inconsistent C r---
ns:117824 nr:0 dw:0 dr:131072 al:0 bm:90 lo:1 pe:5 ua:414 ap:0 
        [>...................] sync'ed:  0.1% (1983593/1983707)M
        finish: 705:16:39 speed: 512 (972) K/sec
resync: used:4/31 hits:7743 misses:31 starving:0 dirty:0 changed:31 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 2: cs:WFBitMapS st:Secondary/Secondary ds:UpToDate/Inconsistent C rap- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 3: cs:WFBitMapS st:Secondary/Secondary ds:UpToDate/Inconsistent C rap- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 4: cs:PausedSyncS st:Secondary/Secondary ds:UpToDate/Inconsistent C ra-- ns:229068 nr:0 dw:0 dr:269568 al:0 bm:13 lo:1 pe:6 ua:1266 ap:0 resync: used:4/31 hits:15560 misses:17 starving:0 dirty:0 changed:17 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 

On the second machine I get this starting drbd:
[EMAIL PROTECTED] ~]# service drbd startStarting DRBD resources: [ d0 d1 d2 d3 d4 s0 s1 s2 s3 n0 n1 n2 n3 Message from [EMAIL PROTECTED] at Fri 
Mar 28 16:21:54 2008 ...SAN-02 kernel: Oops: 0000 [#1]
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: SMP
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: CPU:    0
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EIP: 0060: [<f8aa5816>] Not tainted VLI 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EFLAGS: 00010046 (2.6.19.7-0.3.smp.pae.gcc3.4.x86.i686 #1) 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EIP is at drbd_bm_reset_find +0x53/0x114 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: eax: 00000000 ebx: f78d24b4 ecx: 00000286 edx: 00000000 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: esi: f6dd83c0 edi: 00000011 ebp: f78d2400 esp: f6e13ec0n4 ]. 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: Process drbd3_receiver (pid: 3803, ti=f6e12000 task=f7f04d30 task.ti=f6e12000) 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: ds: 007b   es: 007b   ss: 0068
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: Stack: 00000003 f78d2400 0000001f 00000011 00000286 f8ab5318 00000000 f8ac1cfe 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: f78d24b4 f78d2400 00000011 00000000 f8aa8549 00000282 c012ca70 f78d24b4 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: c0467580 00000282 c012cb17 00000282 f78d24b4 f78d2400 857f5ac6 00000000 [EMAIL PROTECTED] ~]# Message from [EMAIL PROTECTED] at Fri 
Mar 28 16:21:54 2008 ...SAN-02 kernel: Call Trace:
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8ab5318>] drbd_rs_cancel_all+0x149/0x14f [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aa8549>] drbd_start_resync+0x76/0x247 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<c012ca70>] lock_timer_base+0x15/0x2f 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<c012cb17>] __mod_timer +0x8d/0x95 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aaddba>] receive_sync_uuid+0x158/0x167 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aadf91>] receive_bitmap +0x1c8/0x1d4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aaa042>] drbd_recv_header+0x14/0xac [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aadc62>] receive_sync_uuid+0x0/0x167 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aae19d>] drbdd +0x84/0x161 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8aaefff>] drbdd_init +0xb1/0x1a6 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8ab7fe2>] drbd_thread_setup+0x9e/0xe4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<f8ab7f44>] drbd_thread_setup+0x0/0xe4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: [<c010497b>] kernel_thread_helper+0x7/0x10 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel:  =======================
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: Code: 0d ac f8 8b 80 ec 03 00 00 c7 04 24 7d 15 ac f8 89 44 24 04 e8 8b fb 67 c7 e9 c6 00 00 00 8d 46 04 e8 17 41 87 c7 8b 56 14 8b 06 <81> 3c 90 67 02 74 83 74 26 c7 44 24 0c 99 03 00 00 c7 44 24 08 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EIP: [<f8aa5816>] drbd_bm_reset_find+0x53/0x114 [drbd] SS:ESP 0068:f6e13ec0 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: Oops: 0000 [#2]
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: SMP
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: CPU:    0
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EIP: 0060: [<f8aa5816>] Not tainted VLI 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EFLAGS: 00010046 (2.6.19.7-0.3.smp.pae.gcc3.4.x86.i686 #1) 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: EIP is at drbd_bm_reset_find +0x53/0x114 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: eax: 00000000 ebx: f78e10b4 ecx: 00000286 edx: 00000000 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:54 2008 ...SAN-02 kernel: esi: f6dd8740 edi: 00000011 ebp: f78e1000 esp: f6f59ec0 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: ds: 007b   es: 007b   ss: 0068
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: Process drbd2_receiver (pid: 3795, ti=f6f58000 task=f7f29350 task.ti=f6f58000) 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: Stack: 00000003 f78e1000 0000001f 00000011 00000286 f8ab5318 00000000 f8ac1cfe 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: f78e10b4 f78e1000 00000011 00000000 f8aa8549 00000282 c012ca70 f78e10b4 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: c0467580 00000282 c012cb17 00000282 f78e10b4 f78e1000 5b9ab18b 00000000 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: Call Trace:
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8ab5318>] drbd_rs_cancel_all+0x149/0x14f [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aa8549>] drbd_start_resync+0x76/0x247 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<c012ca70>] lock_timer_base+0x15/0x2f 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<c012cb17>] __mod_timer +0x8d/0x95 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aaddba>] receive_sync_uuid+0x158/0x167 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aadf91>] receive_bitmap +0x1c8/0x1d4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aaa042>] drbd_recv_header+0x14/0xac [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aadc62>] receive_sync_uuid+0x0/0x167 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aae19d>] drbdd +0x84/0x161 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8aaefff>] drbdd_init +0xb1/0x1a6 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8ab7fe2>] drbd_thread_setup+0x9e/0xe4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<f8ab7f44>] drbd_thread_setup+0x0/0xe4 [drbd] 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: [<c010497b>] kernel_thread_helper+0x7/0x10 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel:  =======================
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: Code: 0d ac f8 8b 80 ec 03 00 00 c7 04 24 7d 15 ac f8 89 44 24 04 e8 8b fb 67 c7 e9 c6 00 00 00 8d 46 04 e8 17 41 87 c7 8b 56 14 8b 06 <81> 3c 90 67 02 74 83 74 26 c7 44 24 0c 99 03 00 00 c7 44 24 08 
Message from [EMAIL PROTECTED] at Fri
Mar 28 16:21:55 2008 ...SAN-02 kernel: EIP: [<f8aa5816>] drbd_bm_reset_find+0x53/0x114 [drbd] SS:ESP 0068:f6f59ec0 [EMAIL PROTECTED] ~]# service drbd statusdrbd driver loaded OK; device status:version: 8.0.2 (api:86/proto:86)SVN Revision: 2844 build by [EMAIL PROTECTED], 2007-04-27 17:08:12 0: cs:Connected st:Secondary/Secondary ds:Inconsistent/ Inconsistent C r--- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0 1: cs:WFBitMapT st:Secondary/Secondary ds:Inconsistent/UpToDate C r--- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 2: cs:WFSyncUUID st:Secondary/Secondary ds:Inconsistent/UpToDate C r--- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 3: cs:WFSyncUUID st:Secondary/Secondary ds:Inconsistent/UpToDate C r--- 
        ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 4: cs:SyncTarget st:Secondary/Secondary ds:Inconsistent/UpToDate C r--- 
        ns:0 nr:26752 dw:26688 dr:0 al:0 bm:0 lo:4 pe:8064 ua:2 ap:0
        [>...................] sync'ed:  0.1% (156500/156526)M
        finish: 27:49:20 speed: 1,560 (1,480) K/sec
resync: used:17/31 hits:9716 misses:18 starving:0 dirty:0 changed:18 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
apart of the strange start of drbd it seems to work for the moment... But what happens? 

Stopping drbd does this:

[EMAIL PROTECTED] ~]# service drbd stop
Stopping all DRBD resources
Child process does not terminate!
Exiting.
ERROR: Module drbd is in use.
[EMAIL PROTECTED] ~]# No response from the DRBD driver! Is the module loaded? 
No response from the DRBD driver! Is the module loaded?
Error code 134520849 unknown.You should updated the drbd userland tools. 
service drbd stop
Stopping all DRBD resourceslock on /var/lock/drbd-147-0 currently held by pid:3934 Command '/sbin/drbdsetup /dev/drbd0 down' terminated with exit code 20 
drbdsetup exited with code 20ERROR: Module drbd is in use.
[EMAIL PROTECTED] ~]# No response from the DRBD driver! Is the module loaded? Error code 134520849 unknown.You should updated the drbd userland tools. 

[EMAIL PROTECTED] ~]# service drbd stop
Stopping all DRBD resources
Child process does not terminate!
Exiting.
ERROR: Module drbd is in use.

[EMAIL PROTECTED] ~]# service drbd stop
Stopping all DRBD resources
No response from the DRBD driver! Is the module loaded?
Error code 134520849 unknown.You should updated the drbd userland tools. Child process does not terminate!Exiting.ERROR: Module drbd is in use.
[EMAIL PROTECTED] ~]# No response from the DRBD driver! Is the module loaded? Error code 134520849 unknown.You should updated the drbd userland tools. No response from the DRBD driver! Is the module loaded?Error code 134520849 unknown.You should updated the drbd userland tools.
I did a conary update conary and then a conary updateall. Worked all without any errors on both machines. 

A conary updateall tells this on both machines:

no new troves were found

uname -a:
Linux SAN-01.local 2.6.19.7-0.3.smp.pae.gcc3.4.x86.i686 #1 SMP Thu Apr 12 01:53:56 EDT 2007 i686 i686 i386 GNU/Linux Linux SAN-02.local 2.6.19.7-0.3.smp.pae.gcc3.4.x86.i686 #1 SMP Thu Apr 12 01:53:56 EDT 2007 i686 i686 i386 GNU/Linux 


What shall I do????????

igor ([EMAIL PROTECTED])
--------------------------------



_______________________________________________
Openfiler-users mailing list
[email protected]
https://lists.openfiler.com/mailman/listinfo/openfiler-users




_______________________________________________
Openfiler-users mailing list
[email protected]
https://lists.openfiler.com/mailman/listinfo/openfiler-users

Reply via email to