Re: [pve-devel] [PATCH docs 2/2] Update pvecm documentation for corosync 3
Added a few notes, mostly regarding style and readability.
On 7/8/19 6:26 PM, Stefan Reiter wrote:
Parts about multicast and RRP have been removed entirely. Instead, a new
section 'Corosync Redundancy' has been added explaining the concept of
links and link priorities.
Signed-off-by: Stefan Reiter
---
pvecm.adoc | 372 +
1 file changed, 147 insertions(+), 225 deletions(-)
diff --git a/pvecm.adoc b/pvecm.adoc
index 1c0b9e7..1246111 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -56,13 +56,8 @@ Grouping nodes into a cluster has the following advantages:
Requirements
-* All nodes must be in the same network as `corosync` uses IP Multicast
- to communicate between nodes (also see
- http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
- ports 5404 and 5405 for cluster communication.
-+
-NOTE: Some switches do not support IP multicast by default and must be
-manually enabled first.
+* All nodes must be able to contact each other via UDP ports 5404 and 5405 for
+ corosync to work.
Maybe "connect" instead of "contact"?
* Date and time have to be synchronized.
@@ -84,6 +79,11 @@ NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this is not supported as
production configuration and should only used temporarily during upgrading the
whole cluster from one to another major version.
+NOTE: Mixing {pve} 6.x and earlier versions is not supported, because of the
+major corosync upgrade. While possible to run corosync 3 on {pve} 5.4, this
+configuration is not supported for production environments and should only be
+used while upgrading a cluster.
+
"NOTE: Running a cluster of {pve} 6.x with earlier versions is not
possible. The cluster protocol (corosync) between {pve} 6.x and earlier
versions changed fundamentally. The corosync 3 packages for {pve} 5.4
are only intended for the upgrade procedure to {pve} 6.0."
Preparing Nodes
---
@@ -96,10 +96,12 @@ Currently the cluster creation can either be done on the
console (login via
`ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
Cluster__).
-While it's often common use to reference all other nodenames in `/etc/hosts`
-with their IP this is not strictly necessary for a cluster, which normally uses
-multicast, to work. It maybe useful as you then can connect from one node to
-the other with SSH through the easier to remember node name.
+While it's common to reference all nodenames and their IPs in `/etc/hosts` (or
+make their names resolveable through other means), this is not strictly
+necessary for a cluster to work. It may be useful however, as you can then
+connect from one node to the other with SSH via the easier to remember node
+name. (see also xref:pvecm_corosync_addresses[Link Address Types])
+
"node names" instead of one word. "resolvable" instead of "resolveable".
But maybe we should tell people to keep away from host names and rely on
IP addresses? If someone still wants to use the hosts file they will,
but let not put ideas in the heads of not so experienced users who do
not know possible pitfalls and the fragility of such an approach.
"We highly recommend to reference the nodes by their IP addresses in the
cluster configuration. This will prevent (circular) dependencies on
other means to resolve a host name to an IP address like DNS or manual
entries in the `/etc/hosts` file."
Not sure if we should have the "(circular)" in there or not.
[[pvecm_create_cluster]]
Create the Cluster
@@ -113,31 +115,12 @@ node names.
hp1# pvecm create CLUSTERNAME
-CAUTION: The cluster name is used to compute the default multicast address.
-Please use unique cluster names if you run more than one cluster inside your
-network. To avoid human confusion, it is also recommended to choose different
-names even if clusters do not share the cluster network.
-
To check the state of your cluster use:
hp1# pvecm status
-Multiple Clusters In Same Network
-~
-
-It is possible to create multiple clusters in the same physical or logical
-network. Each cluster must have a unique name, which is used to generate the
-cluster's multicast group address. As long as no duplicate cluster names are
-configured in one network segment, the different clusters won't interfere with
-each other.
-
-If multiple clusters operate in a single network it may be beneficial to setup
-an IGMP querier and enable IGMP Snooping in said network. This may reduce the
-load of the network significantly because multicast packets are only delivered
-to endpoints of the respective member nodes.
-
[[pvecm_join_node_to_cluster]]
Adding Nodes to the Cluster
@@ -150,7 +133,7 @@ Login via `ssh` to the node you want to add.
For `IP-ADDRESS-CLUSTER` use the IP or hostname of an existing cluster node.
-An IP address is recommended (see
Re: [pve-devel] [PATCH docs 2/2] Update pvecm documentation for corosync 3
Thanks for feedback!
Regarding patch 1/2: I grep'd through the sources and could not find any
references to the heading names I changed. A quick look through the GUI
also didn't reveal any obvious references.
Some of my own notes inline, I will send v2 today.
On 7/9/19 9:19 AM, Thomas Lamprecht wrote:
On 7/8/19 6:26 PM, Stefan Reiter wrote:
Parts about multicast and RRP have been removed entirely. Instead, a new
section 'Corosync Redundancy' has been added explaining the concept of
links and link priorities.
note bad at all, still some notes inline.
Signed-off-by: Stefan Reiter
---
pvecm.adoc | 372 +
1 file changed, 147 insertions(+), 225 deletions(-)
diff --git a/pvecm.adoc b/pvecm.adoc
index 1c0b9e7..1246111 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -56,13 +56,8 @@ Grouping nodes into a cluster has the following advantages:
Requirements
-* All nodes must be in the same network as `corosync` uses IP Multicast
- to communicate between nodes (also see
- http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
- ports 5404 and 5405 for cluster communication.
-+
-NOTE: Some switches do not support IP multicast by default and must be
-manually enabled first.
+* All nodes must be able to contact each other via UDP ports 5404 and 5405 for
+ corosync to work.
* Date and time have to be synchronized.
@@ -84,6 +79,11 @@ NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this is not supported as
production configuration and should only used temporarily during upgrading the
whole cluster from one to another major version.
+NOTE: Mixing {pve} 6.x and earlier versions is not supported, because of the
+major corosync upgrade. While possible to run corosync 3 on {pve} 5.4, this
+configuration is not supported for production environments and should only be
+used while upgrading a cluster.
+
Preparing Nodes
---
@@ -96,10 +96,12 @@ Currently the cluster creation can either be done on the
console (login via
`ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
Cluster__).
-While it's often common use to reference all other nodenames in `/etc/hosts`
-with their IP this is not strictly necessary for a cluster, which normally uses
-multicast, to work. It maybe useful as you then can connect from one node to
-the other with SSH through the easier to remember node name.
+While it's common to reference all nodenames and their IPs in `/etc/hosts` (or
+make their names resolveable through other means), this is not strictly
+necessary for a cluster to work. It may be useful however, as you can then
+connect from one node to the other with SSH via the easier to remember node
+name. (see also xref:pvecm_corosync_addresses[Link Address Types])
+
[[pvecm_create_cluster]]
Create the Cluster
@@ -113,31 +115,12 @@ node names.
hp1# pvecm create CLUSTERNAME
-CAUTION: The cluster name is used to compute the default multicast address.
-Please use unique cluster names if you run more than one cluster inside your
-network. To avoid human confusion, it is also recommended to choose different
-names even if clusters do not share the cluster network.
Maybe move this from a "CAUTION" to a "NOTE" and keep the hint that it still
makes sense to use unique cluster names, to avoid human confusion and as I have
a feeling that there are other assumption in corosync which depend on that.
Also, _if_ multicast gets integrated into knet we probably have a similar issue
again, so try to bring people in lane now already, even if not 100% required.
Makes sense. I just wanted to avoid mentioning multicast in the general
instructions, to avoid people reading the docs for the first time being
confused if they need it or not.
-
To check the state of your cluster use:
hp1# pvecm status
-Multiple Clusters In Same Network
-~
-
-It is possible to create multiple clusters in the same physical or logical
-network. Each cluster must have a unique name, which is used to generate the
-cluster's multicast group address. As long as no duplicate cluster names are
-configured in one network segment, the different clusters won't interfere with
-each other.
-
-If multiple clusters operate in a single network it may be beneficial to setup
-an IGMP querier and enable IGMP Snooping in said network. This may reduce the
-load of the network significantly because multicast packets are only delivered
-to endpoints of the respective member nodes.
-
It's still possible to create multiple clusters in the same network, so I'd keep
above and just adapt to non-multicast for now..
[[pvecm_join_node_to_cluster]]
Adding Nodes to the Cluster
@@ -150,7 +133,7 @@ Login via `ssh` to the node you want to add.
For `IP-ADDRESS-CLUSTER` use the IP or hostname of an existing cluster node.
-An IP address is recommended
Re: [pve-devel] [PATCH docs 2/2] Update pvecm documentation for corosync 3
On 7/8/19 6:26 PM, Stefan Reiter wrote:
> Parts about multicast and RRP have been removed entirely. Instead, a new
> section 'Corosync Redundancy' has been added explaining the concept of
> links and link priorities.
>
note bad at all, still some notes inline.
> Signed-off-by: Stefan Reiter
> ---
> pvecm.adoc | 372 +
> 1 file changed, 147 insertions(+), 225 deletions(-)
>
> diff --git a/pvecm.adoc b/pvecm.adoc
> index 1c0b9e7..1246111 100644
> --- a/pvecm.adoc
> +++ b/pvecm.adoc
> @@ -56,13 +56,8 @@ Grouping nodes into a cluster has the following advantages:
> Requirements
>
>
> -* All nodes must be in the same network as `corosync` uses IP Multicast
> - to communicate between nodes (also see
> - http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
> - ports 5404 and 5405 for cluster communication.
> -+
> -NOTE: Some switches do not support IP multicast by default and must be
> -manually enabled first.
> +* All nodes must be able to contact each other via UDP ports 5404 and 5405
> for
> + corosync to work.
>
> * Date and time have to be synchronized.
>
> @@ -84,6 +79,11 @@ NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this
> is not supported as
> production configuration and should only used temporarily during upgrading
> the
> whole cluster from one to another major version.
>
> +NOTE: Mixing {pve} 6.x and earlier versions is not supported, because of the
> +major corosync upgrade. While possible to run corosync 3 on {pve} 5.4, this
> +configuration is not supported for production environments and should only be
> +used while upgrading a cluster.
> +
>
> Preparing Nodes
> ---
> @@ -96,10 +96,12 @@ Currently the cluster creation can either be done on the
> console (login via
> `ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
> Cluster__).
>
> -While it's often common use to reference all other nodenames in `/etc/hosts`
> -with their IP this is not strictly necessary for a cluster, which normally
> uses
> -multicast, to work. It maybe useful as you then can connect from one node to
> -the other with SSH through the easier to remember node name.
> +While it's common to reference all nodenames and their IPs in `/etc/hosts`
> (or
> +make their names resolveable through other means), this is not strictly
> +necessary for a cluster to work. It may be useful however, as you can then
> +connect from one node to the other with SSH via the easier to remember node
> +name. (see also xref:pvecm_corosync_addresses[Link Address Types])
> +
>
> [[pvecm_create_cluster]]
> Create the Cluster
> @@ -113,31 +115,12 @@ node names.
> hp1# pvecm create CLUSTERNAME
>
>
> -CAUTION: The cluster name is used to compute the default multicast address.
> -Please use unique cluster names if you run more than one cluster inside your
> -network. To avoid human confusion, it is also recommended to choose different
> -names even if clusters do not share the cluster network.
Maybe move this from a "CAUTION" to a "NOTE" and keep the hint that it still
makes sense to use unique cluster names, to avoid human confusion and as I have
a feeling that there are other assumption in corosync which depend on that.
Also, _if_ multicast gets integrated into knet we probably have a similar issue
again, so try to bring people in lane now already, even if not 100% required.
> -
> To check the state of your cluster use:
>
>
> hp1# pvecm status
>
>
> -Multiple Clusters In Same Network
> -~
> -
> -It is possible to create multiple clusters in the same physical or logical
> -network. Each cluster must have a unique name, which is used to generate the
> -cluster's multicast group address. As long as no duplicate cluster names are
> -configured in one network segment, the different clusters won't interfere
> with
> -each other.
> -
> -If multiple clusters operate in a single network it may be beneficial to
> setup
> -an IGMP querier and enable IGMP Snooping in said network. This may reduce the
> -load of the network significantly because multicast packets are only
> delivered
> -to endpoints of the respective member nodes.
> -
It's still possible to create multiple clusters in the same network, so I'd keep
above and just adapt to non-multicast for now..
>
> [[pvecm_join_node_to_cluster]]
> Adding Nodes to the Cluster
> @@ -150,7 +133,7 @@ Login via `ssh` to the node you want to add.
>
>
> For `IP-ADDRESS-CLUSTER` use the IP or hostname of an existing cluster node.
> -An IP address is recommended (see xref:pvecm_corosync_addresses[Ring Address
> Types]).
> +An IP address is recommended (see xref:pvecm_corosync_addresses[Link Address
> Types]).
Maybe somewhere a note that while the new things are named "Link" the config
still refers to "ringX_addr" for backward compatibility.
>
> CAUTION: A new node cannot hold any
[pve-devel] [PATCH docs 2/2] Update pvecm documentation for corosync 3
Parts about multicast and RRP have been removed entirely. Instead, a new
section 'Corosync Redundancy' has been added explaining the concept of
links and link priorities.
Signed-off-by: Stefan Reiter
---
pvecm.adoc | 372 +
1 file changed, 147 insertions(+), 225 deletions(-)
diff --git a/pvecm.adoc b/pvecm.adoc
index 1c0b9e7..1246111 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -56,13 +56,8 @@ Grouping nodes into a cluster has the following advantages:
Requirements
-* All nodes must be in the same network as `corosync` uses IP Multicast
- to communicate between nodes (also see
- http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
- ports 5404 and 5405 for cluster communication.
-+
-NOTE: Some switches do not support IP multicast by default and must be
-manually enabled first.
+* All nodes must be able to contact each other via UDP ports 5404 and 5405 for
+ corosync to work.
* Date and time have to be synchronized.
@@ -84,6 +79,11 @@ NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this
is not supported as
production configuration and should only used temporarily during upgrading the
whole cluster from one to another major version.
+NOTE: Mixing {pve} 6.x and earlier versions is not supported, because of the
+major corosync upgrade. While possible to run corosync 3 on {pve} 5.4, this
+configuration is not supported for production environments and should only be
+used while upgrading a cluster.
+
Preparing Nodes
---
@@ -96,10 +96,12 @@ Currently the cluster creation can either be done on the
console (login via
`ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
Cluster__).
-While it's often common use to reference all other nodenames in `/etc/hosts`
-with their IP this is not strictly necessary for a cluster, which normally uses
-multicast, to work. It maybe useful as you then can connect from one node to
-the other with SSH through the easier to remember node name.
+While it's common to reference all nodenames and their IPs in `/etc/hosts` (or
+make their names resolveable through other means), this is not strictly
+necessary for a cluster to work. It may be useful however, as you can then
+connect from one node to the other with SSH via the easier to remember node
+name. (see also xref:pvecm_corosync_addresses[Link Address Types])
+
[[pvecm_create_cluster]]
Create the Cluster
@@ -113,31 +115,12 @@ node names.
hp1# pvecm create CLUSTERNAME
-CAUTION: The cluster name is used to compute the default multicast address.
-Please use unique cluster names if you run more than one cluster inside your
-network. To avoid human confusion, it is also recommended to choose different
-names even if clusters do not share the cluster network.
-
To check the state of your cluster use:
hp1# pvecm status
-Multiple Clusters In Same Network
-~
-
-It is possible to create multiple clusters in the same physical or logical
-network. Each cluster must have a unique name, which is used to generate the
-cluster's multicast group address. As long as no duplicate cluster names are
-configured in one network segment, the different clusters won't interfere with
-each other.
-
-If multiple clusters operate in a single network it may be beneficial to setup
-an IGMP querier and enable IGMP Snooping in said network. This may reduce the
-load of the network significantly because multicast packets are only delivered
-to endpoints of the respective member nodes.
-
[[pvecm_join_node_to_cluster]]
Adding Nodes to the Cluster
@@ -150,7 +133,7 @@ Login via `ssh` to the node you want to add.
For `IP-ADDRESS-CLUSTER` use the IP or hostname of an existing cluster node.
-An IP address is recommended (see xref:pvecm_corosync_addresses[Ring Address
Types]).
+An IP address is recommended (see xref:pvecm_corosync_addresses[Link Address
Types]).
CAUTION: A new node cannot hold any VMs, because you would get
conflicts about identical VM IDs. Also, all existing configuration in
@@ -173,7 +156,7 @@ Date: Mon Apr 20 12:30:13 2015
Quorum provider: corosync_votequorum
Nodes:4
Node ID: 0x0001
-Ring ID: 1928
+Ring ID: 1/8
Quorate: Yes
Votequorum information
@@ -217,15 +200,15 @@ Adding Nodes With Separated Cluster Network
~~~
When adding a node to a cluster with a separated cluster network you need to
-use the 'ringX_addr' parameters to set the nodes address on those networks:
+use the 'link0' parameter to set the nodes address on that network:
[source,bash]
-pvecm add IP-ADDRESS-CLUSTER -ring0_addr IP-ADDRESS-RING0
+pvecm add IP-ADDRESS-CLUSTER -link0 LOCAL-IP-ADDRESS-LINK0
-If you want to use the Redundant Ring Protocol you will also want to pass the
-'ring1_addr' parameter.
+If you want to use the
