This is pretty simple:
(1) Every server have, in most cases, more then 1 physical interface. In
many cases, these interfaces are already
connected to a different switches or even uses different medias. Even if
bonded, we have still a few of them.
(2) Cluster is extremely sensitive to the lost of interconnection, because
it cause fencing or self-fencing
(reboot or panic, in simple words) on some nodes, and delays on anothers.
(3) If we use heartbeat over few independent IP interfaces, then we don't
depend of switch timeouts, bonding timeouts, bonding rules
and other things that affects all low level bondings, trunkings and so on.
These timeouts are not small - for example, STP reconfiguration in Ethernet
takes 40 seconds BY DEFAULT.
(4) WE DO HAVE THEE INTERFACES ALREADY.
(5) Implementing few heartbeats costs almost 0 for developers.
It explains why most clusters supports few heartbeats. I always configure
heartbeat over ALL available medias. If I have NFS cluster, I use 3 media -
eth0 (public), eth1 (SAN), eth2 (management), plus serial. The reason is
simple - it is very important to keep cluster interconnection in case of
medial failure - it allows nodes to switchover in a few seconds because they
can communicate and know each other's status.
Now let's image that I can not use multi interface for heartbeat (or VCS, it
don't make much difference) cluster. Switch fail so that node1 lost
ethernet1.
What shoudl node2 do:
- reboot? But what if node1 is dead.
- take control? But it is not sure if node1 is dead.
- kill node1? But what if node1 lost ethernet and node2 is the only alive
member.
No way to resolve it,. If I use bonding (aside of need in 2 more
interfaces) - node still must wait 30 - 40 seconds to eliminate switch
timeouts, bonding timeouts and so on.
Now notice - nodes can talk, they are interconnected. By eth0, eth2 and
serial. So what's the problem? heartbeat is pretty simple thing with OR
rule - if
any channel respond, node is alive. No complexity to implement it.
Now I run thru a few implementations. Every one which have not ORACLE in
name allows me to use many heartbeats. 5 examples - Cisco PIX,
Veritas VCS, Linux heartbeat1, Linux heartbeat2, NetApp cluster (2
independent inter-connections). And Oracle is the only exception, Why?
----- Original Message -----
From: "Bob Murching" <[EMAIL PROTECTED]>
I'm still trying to understand your earlier post. I understand why Veritas
wants multiple heartbeats. That said, I thought that RAC mitigates the risk
with voting disk, etc. I guess I'm not understanding why "single heartbeat,
use bonding" is necessarily a bad thing...?
-----Original Message-----
From: Alexei_Roudnev [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 04, 2007 5:20 PM
To: [EMAIL PROTECTED]
Subject: Re: [Ocfs2-users] Aditional interconnect interface
It did not changed, as I can see:
- <<IT is recommenede to configure at leat 2 huigh priority links and 1 low
priority links>>
(link == interface)
When configuring heartbeat, I configure 3 links:
- 1 over access network
- 1 over SAN network
- 1 over Serial or over management interface
(all on different switches).
----- Original Message -----
From: "Bob Murching" <[EMAIL PROTECTED]>
To: "'Alexei_Roudnev'" <[EMAIL PROTECTED]>
Sent: Thursday, October 04, 2007 1:43 PM
Subject: RE: [Ocfs2-users] Aditional interconnect interface
nevermind!
http://seer.entsupport.symantec.com/docs/283846.htm
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Alexei_Roudnev
Sent: Thursday, October 04, 2007 3:31 PM
To: Sunil Mushran; Pedro Figueira
Cc: [email protected]
Subject: Re: [Ocfs2-users] Aditional interconnect interface
It's old relogious discussion. All people who works in data centers knows,
that cluster MUST use more then 1 heartbeat (better 3 or 4, not bonded)
interface for stability, but Oracle developers (both RAC and OCFSv2) believe
in bonding.
(Bonding DO NOT solve the problem! It have sugnificant switchover time and
it is not reliable enough. Open Veritas cluster documentation and READ IT -
they have a pretty good explanation why you must configure 2 independent
heartbeats using 2 independent switches or hubs. Veritas use multi
interface. Cisco PIX uses multi interface. Heartbeat1 uses multi interface.
Heartbeat2 uses multi interfaces. And only Oracle* developers use 1
interface and trust into the bonding!)
Using additional loopback interface and routing (via OSPF which is pretty
fast) can solve problem too. (for OCFS bot not for RAC cluster).
----- Original Message -----
From: "Sunil Mushran" <[EMAIL PROTECTED]>
To: "Pedro Figueira" <[EMAIL PROTECTED]>
Cc: <[email protected]>
Sent: Wednesday, October 03, 2007 9:26 AM
Subject: Re: [Ocfs2-users] Aditional interconnect interface
Use network bonding.
Pedro Figueira wrote:
Hello
Is there a way of using more than one network interface for ocfs2
interconnect? I've searched for this in the documentation but I cannot
find any information about it.
Best regards
Pedro Figueira Serviço de Estrangeiros e Fronteiras Direcção Central de
Informática
Departamento de Produção Telefone: + 351 217 115 153
CONFIDENCIAL NOTICE:
This message, as well as any existing attached files, is confidential and
intended exclusively for the individual(s) named as addressees. If you are
not the intended recipient, you are kindly requested not to make any use
whatsoever of its contents and to proceed to the destruction of the
message, thereby notifying the sender. DISCLAIMER:
The sender of this message can NOT ensure the security of its electronic
transmission and consequently does not accept liability for any fact,
which may interfere with the integrity of its content.
_______________________________________________
Ocfs2-users mailing list
[email protected]
http://oss.oracle.com/mailman/listinfo/ocfs2-users
_______________________________________________
Ocfs2-users mailing list
[email protected]
http://oss.oracle.com/mailman/listinfo/ocfs2-users
_______________________________________________
Ocfs2-users mailing list
[email protected]
http://oss.oracle.com/mailman/listinfo/ocfs2-users
_______________________________________________
Ocfs2-users mailing list
[email protected]
http://oss.oracle.com/mailman/listinfo/ocfs2-users
