Are you using the IPAddr2 primitive? Maybe post your configuration?

Nick.

On Sun, Oct 30, 2011 at 4:35 PM, James Smith <james.sm...@m247.com> wrote:
> Ipv4.
>
> Regards,
>
> James
>
>
> -----Original Message-----
> From: linux-ha-boun...@lists.linux-ha.org 
> [mailto:linux-ha-boun...@lists.linux-ha.org] On Behalf Of Nick Khamis
> Sent: 30 October 2011 12:28
> To: General Linux-HA mailing list
> Subject: Re: [Linux-HA] SANs falling over, don't know why!
>
> Are you using IPV4 or 6?
>
> Nick.
>
> On Sun, Oct 30, 2011 at 4:29 AM, James Smith <james.sm...@m247.com> wrote:
>> Well fileio hasn't solved the underlying issue, the SAN broke this morning 
>> at 6AM:
>>
>> Oct 30 06:01:19 iscsi1cl6 kernel: iscsi_trgt: Abort Task (01) issued
>> on tid:1 lun:6 by sid:4222124721766912 (Function Complete) Oct 30
>> 06:01:20 iscsi1cl6 lrmd: [3770]: info: RA output:
>> (ClusterIP:monitor:stderr) Converted dotted-quad netmask to CIDR as:
>> 24 Oct 30 06:01:47 iscsi1cl6 last message repeated 27 times Oct 30
>> 06:01:48 iscsi1cl6 kernel: iscsi_trgt: Abort Task (01) issued on tid:1
>> lun:6 by sid:5066549651898880 (Function Complete) Oct 30 06:01:48
>> iscsi1cl6 kernel: iscsi_trgt: Abort Task (01) issued on tid:1 lun:6 by
>> sid:5066549651898880 (Function Complete) Oct 30 06:01:48 iscsi1cl6
>> lrmd: [3770]: info: RA output: (ClusterIP:monitor:stderr) Converted
>> dotted-quad netmask to CIDR as: 24 Oct 30 06:02:19 iscsi1cl6 last
>> message repeated 30 times Oct 30 06:02:48 iscsi1cl6 last message
>> repeated 28 times Oct 30 06:02:49 iscsi1cl6 kernel: iscsi_trgt: Abort
>> Task (01) issued on tid:1 lun:6 by sid:5066549651898880 (Function
>> Complete) Oct 30 06:02:49 iscsi1cl6 lrmd: [3770]: info: RA output:
>> (ClusterIP:monitor:stderr) Converted dotted-quad netmask to CIDR as:
>> 24 Oct 30 06:02:51 iscsi1cl6 last message repeated 2 times Oct 30
>> 06:02:52 iscsi1cl6 kernel: iscsi_trgt: Abort Task (01) issued on tid:1
>> lun:6 by sid:4222124721766912 (Function Complete) Oct 30 06:02:52
>> iscsi1cl6 lrmd: [3770]: info: RA output: (ClusterIP:monitor:stderr)
>> Converted dotted-quad netmask to CIDR as: 24 Oct 30 06:03:17 iscsi1cl6
>> last message repeated 24 times Oct 30 06:03:18 iscsi1cl6 kernel:
>> iscsi_trgt: cmnd_rx_start(1849) 1 3b000030 -7 Oct 30 06:03:18
>> iscsi1cl6 kernel: iscsi_trgt: cmnd_skip_pdu(459) 3b000030 1 2a 4096
>> Oct 30 06:03:18 iscsi1cl6 lrmd: [3770]: info: RA output:
>> (ClusterIP:monitor:stderr) Converted dotted-quad netmask to CIDR as:
>> 24 Oct 30 06:03:49 iscsi1cl6 last message repeated 30 times Oct 30
>> 06:04:32 iscsi1cl6 last message repeated 42 times Oct 30 06:04:33
>> iscsi1cl6 cib: [3769]: info: cib_stats: Processed 1 operations
>> (10000.00us average, 0% utilization) in the last 10min Oct 30 06:04:33
>> iscsi1cl6 lrmd: [3770]: info: RA output: (ClusterIP:monitor:stderr)
>> Converted dotted-quad netmask to CIDR as: 24 Oct 30 06:05:04 iscsi1cl6
>> last message repeated 30 times Oct 30 06:05:41 iscsi1cl6 last message
>> repeated 36 times Oct 30 06:05:42 iscsi1cl6 kernel: iscsi_trgt: Abort
>> Task (01) issued on tid:1 lun:6 by sid:5629499605320192 (Function
>> Complete)
>>
>>
>> Regards,
>>
>> James
>>
>> -----Original Message-----
>> From: linux-ha-boun...@lists.linux-ha.org
>> [mailto:linux-ha-boun...@lists.linux-ha.org] On Behalf Of James Smith
>> Sent: 30 October 2011 00:25
>> To: General Linux-HA mailing list
>> Subject: Re: [Linux-HA] SANs falling over, don't know why!
>>
>> Hi,
>>
>> Changed nothing to my knowledge :p
>>
>> These boxes don't currently have fencing enabled.  I imagine the reboot is 
>> caused by a kernel panic, sysctl is set to reboot on this.
>>
>> There is one big 4TB LUN, used by several VMs on XenServer, each with 
>> multiple disks.
>>
>> In my quest to resolve, I have changed iet to use fileio instead of blockio 
>> and fiddled with some drbd performance related bits 
>> (http://www.drbd.org/users-guide/s-latency-tuning.html).
>>
>> If I'm woken up again tonight with this thing breaking it's going in the 
>> bin.  I'll probably also ditch ietd and try open-iscsi or iscsi-scst.  
>> Monday morning I'll be shifting some load off this cluster also.
>>
>> Regards,
>>
>> James
>>
>> -----Original Message-----
>> From: linux-ha-boun...@lists.linux-ha.org
>> [mailto:linux-ha-boun...@lists.linux-ha.org] On Behalf Of Andreas Kurz
>> Sent: 29 October 2011 22:36
>> To: linux-ha@lists.linux-ha.org
>> Subject: Re: [Linux-HA] SANs falling over, don't know why!
>>
>> Hello,
>>
>> On 10/29/2011 08:47 PM, James Smith wrote:
>>> Hi,
>>>
>>> All of a sudden, a SAN pair which was running without any problems for six 
>>> months, now decides to fall over every couple of hours.
>>
>> So what did you change? ;-)
>>
>>>
>>> The logs I have to go on are below:
>>>
>>> Oct 29 19:09:23 iscsi2cl6 last message repeated 12 times Oct 29
>>> 19:09:23 iscsi2cl6 kernel: iscsi_trgt: Abort Task (01) issued on
>>> tid:1
>>> lun:6 by sid:844424967684608 (Function Complete) Oct 29 19:09:24
>>> iscsi2cl6 lrmd: [4677]: info: RA output: (ClusterIP:monitor:stderr)
>>> Converted dotted-quad netmask to CIDR as: 24 Oct 29 19:09:49
>>> iscsi2cl6 last message repeated 24 times Oct 29 19:09:49 iscsi2cl6 kernel:
>>> iscsi_trgt: Abort Task (01) issued on tid:1 lun:6 by
>>> sid:1125899927618048 (Function Complete) Oct 29 19:09:49 iscsi2cl6
>>> kernel: iscsi_trgt: Abort Task (01) issued on tid:1 lun:6 by
>>> sid:1407374904328704 (Function Complete) Oct 29 19:09:49 iscsi2cl6
>>> kernel: iscsi_trgt: Abort Task (01) issued on tid:1 lun:6 by
>>> sid:281474997486080 (Function Complete) Oct 29 19:09:50 iscsi2cl6
>>> lrmd: [4677]: info: RA output: (ClusterIP:monitor:stderr) Converted
>>> dotted-quad netmask to CIDR as: 24 Oct 29 19:09:50 iscsi2cl6 kernel:
>>> iscsi_trgt: Abort Task (01) issued on tid:1 lun:6 by
>>> sid:562949974196736 (Function Complete) Oct 29 19:09:51 iscsi2cl6
>>> lrmd: [4677]: info: RA output: (ClusterIP:monitor:stderr) Converted
>>> dotted-quad netmask to CIDR as: 24 Oct 29 19:09:53 iscsi2cl6 last
>>> message repeated 2 times Oct 29 19:09:53 iscsi2cl6 kernel: iscsi_trgt:
>>> Abort Task (01) issued on tid:1 lun:6 by sid:844424967684608
>>> (Function
>>> Complete) Oct 29 19:09:53 iscsi2cl6 kernel: iscsi_trgt: Abort Task
>>> (01) issued on tid:1 lun:6 by sid:844424967684608 (Function Complete)
>>> Oct 29 19:09:54 iscsi2cl6 lrmd: [4677]: info: RA output:
>>> (ClusterIP:monitor:stderr) Converted dotted-quad netmask to CIDR as:
>>> 24 Oct 29 19:10:05 iscsi2cl6 last message repeated 11 times Oct 29
>>> 19:10:06 iscsi2cl6 kernel: iscsi_trgt: Abort Task (01) issued on
>>> tid:1
>>> lun:6 by sid:1407374904328704 (Function Complete) Oct 29 19:10:06
>>> iscsi2cl6 last message repeated 4 times Oct 29 19:10:06 iscsi2cl6
>>> kernel: block drbd0: istiod1[4695] Concurrent local write detected!
>>> [DISCARD L] new: 2077806177s +3584; pending: 2077806177s +3584 Oct 29
>>> 19:10:06 iscsi2cl6 kernel: block drbd0: istiod1[4695] Concurrent
>>> local write detected! [DISCARD L] new: 2077806184s +512; pending:
>>> 2077806184s +512 Oct 29 19:10:06 iscsi2cl6 kernel: block drbd0:
>>> istiod1[4695] Concurrent local write detected! [DISCARD L] new:
>>> 1693425337s +3584; pending: 1693425337s +3584 Oct 29 19:10:06
>>> iscsi2cl6 kernel: block drbd0: istiod1[4695] Concurrent local write
>>> detected! [DISCARD L] new: 1693425344s +512; pending: 1693425344s
>>> +512 Oct 29 19:10:06 iscsi2cl6 kernel: block drbd0: istiod1[4695]
>>> Concurrent local write detected! [DISCARD L] new: 1693425321s +3584;
>>> pending: 1693425321s +3584 Oct 29 19:10:06 iscsi2cl6 kernel: block
>>> drbd0: istiod1[4695] Concurrent local write detected! [DISCARD L] new:
>>> 1693425328s +512; pending: 1693425328s +512 Oct 29 19:10:06 iscsi2cl6
>>> kernel: block drbd0: istiod1[4695] Concurrent local write detected!
>>> [DISCARD L] new: 1693425313s +3584; pending: 1693425313s +3584 Oct 29
>>> 19:10:06 iscsi2cl6 kernel: block drbd0: istiod1[4695] Concurrent
>>> local write detected! [DISCARD L] new: 1693425320s +512; pending:
>>> 1693425320s +512 Oct 29 19:10:06 iscsi2cl6 kernel: block drbd0:
>>> istiod1[4695] Concurrent local write detected! [DISCARD L] new:
>>> 1743088585s +3584; pending: 1743088585s +3584 Oct 29 19:10:06
>>> iscsi2cl6 kernel: block drbd0: istiod1[4695] Concurrent local write
>>> detected! [DISCARD L] new: 1743088592s +512; pending: 1743088592s
>>> +512
>>
>> Concurrent local writes .... Is there any kind of cluster software using a 
>> shared quorum disk or sthg. like that using this lun? Or this lun shared 
>> between several VMWare ESX VMs?
>>
>>> Oct 29 19:10:06 iscsi2cl6 lrmd: [4677]: info: RA output:
>>> (ClusterIP:monitor:stderr) Converted dotted-quad netmask to CIDR as:
>>> 24
>>>
>>> After this event, both members of the SAN pair reboot.  It is very 
>>> disruptive, as it's killing the VMs using this SAN, requiring fsck's after 
>>> failure.  The load on the SAN doesn't need to be very high for this happen.
>>>
>>
>> They reboot because of a kernel panic, or because of some fencing mechanism?
>>
>>> Running the following:
>>>
>>> CentOS 5 with kernel 2.6.18-274.7.1.el5 IET 1.4.20.2 Pacemaker
>>> 1.0.11-1.2.el5 DRBD 8.3.11
>>
>> Would be interesting to see Pacemaker/DRBD/IET config ....
>>
>> Regards,
>> Andreas
>> --
>> Need help with Pacemaker?
>> http://www.hastexo.com/now
>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Reply via email to