[jira] [Commented] (ARTEMIS-3157) uneven number of connections in a cluster
[
https://issues.apache.org/jira/browse/ARTEMIS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684284#comment-17684284
]
Erwin Dondorp commented on ARTEMIS-3157:
> Can you reproduce this on a simpler configuration (e.g. 2 live brokers, no
> backups)?
and then I responded with an even more complicated example... :-( sorry!
but I'm still looking to have the list of connections that we see explained.
the setup is now a cluster with just 2 nodes, each having a static connector to
the other node.
there are no regular clients involved.
in this case, both nodes show roughly the same list of connections. see the
pictures below.
there is 1 cluster connection that I understand, it has the expected username
and it has a non-zero session count.
but the other 2 connections are a puzzle to me.
so the questions are: should these 2 connections be present, and if so, what
are they for?
{panel:title=connections in cluster node 1}
!screenshot-2.png!
{panel}
{panel:title=connections in cluster node 2}
!screenshot-3.png!
{panel}
> uneven number of connections in a cluster
> -
>
> Key: ARTEMIS-3157
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3157
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.17.0
>Reporter: Erwin Dondorp
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> Using a cluster of 3 master + 3 slave nodes. full interconnect, each with a
> fresh virtual disk. no other constructions
> (bridges/federation/brokerconn/etc) and also no clients.
> On each master nodes, the 2 static connections to the other master nodes are
> visible, and each is marked with the dedicated cluster username. so that part
> seems ok.
> but without any clients having connected, there are additional connections.
> the amount is not the same in each master node. Some connections show
> "127.0.0.1" as address, something that is not in my configuration. none of
> the extra connections have any sessions. Also the connections do not have any
> sessions.
> the details of an example:
> * broker1: 3 connections to own slave; 2 extra to/from broker2; 1 extra
> to/from backup of broker3
> * broker2: 3 connections to own slave; 2 extra to/from broker1; 2 extra
> to/from broker3; 1 extra to/from slave of broker3
> * broker3: 1 connection to own slave; no other extra connections
> the exact amount of connections varies a little between startups and also
> seems to depend on the exact startup sequence.
> my assumption is that these connections should not be present, and that this
> is not intended, hence this report. my wild guess is that these are remnants
> from connections that did not succeed due to the other cluster-members not
> fully available yet.
> When the cluster is started one node at a time, the effect seems to exists
> only on the first node that was started.
> Note: not related to ARTEMIS-2870 as this report is still valid in
> 2.18.0-20210322.234647-43.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-3157) uneven number of connections in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440131#comment-17440131 ] Erwin Dondorp commented on ARTEMIS-3157: The problem seems to be related to special conditions that exist in a container-based environment. The hostnames of the other nodes cannot be resolved until these nodes have actually been started. This is because the IP numbers are assigned dynamically and are only added to the internal DNS system as soon as a node is started. The first node hits that problem more often than the nodes that are started later, so that is consistent My assumption is now that Artemis keeps the connection entry when it hits a unknown-host-exception. > uneven number of connections in a cluster > - > > Key: ARTEMIS-3157 > URL: https://issues.apache.org/jira/browse/ARTEMIS-3157 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.17.0 >Reporter: Erwin Dondorp >Priority: Major > Attachments: screenshot-1.png > > > Using a cluster of 3 master + 3 slave nodes. full interconnect, each with a > fresh virtual disk. no other constructions > (bridges/federation/brokerconn/etc) and also no clients. > On each master nodes, the 2 static connections to the other master nodes are > visible, and each is marked with the dedicated cluster username. so that part > seems ok. > but without any clients having connected, there are additional connections. > the amount is not the same in each master node. Some connections show > "127.0.0.1" as address, something that is not in my configuration. none of > the extra connections have any sessions. Also the connections do not have any > sessions. > the details of an example: > * broker1: 3 connections to own slave; 2 extra to/from broker2; 1 extra > to/from backup of broker3 > * broker2: 3 connections to own slave; 2 extra to/from broker1; 2 extra > to/from broker3; 1 extra to/from slave of broker3 > * broker3: 1 connection to own slave; no other extra connections > the exact amount of connections varies a little between startups and also > seems to depend on the exact startup sequence. > my assumption is that these connections should not be present, and that this > is not intended, hence this report. my wild guess is that these are remnants > from connections that did not succeed due to the other cluster-members not > fully available yet. > When the cluster is started one node at a time, the effect seems to exists > only on the first node that was started. > Note: not related to ARTEMIS-2870 as this report is still valid in > 2.18.0-20210322.234647-43. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARTEMIS-3157) uneven number of connections in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412794#comment-17412794 ] Erwin Dondorp commented on ARTEMIS-3157: > Can you reproduce this on a simpler configuration (e.g. 2 live brokers, no > backups)? yes currently running cluster of 5 on my development machine, all static cluster connections, no backups 1st node has 4 broker connections as it should plus 5 anonymous connections without sessions (2 from the same IP) 2nd node has 4 broker connections as it should plus 1 anonymous connection without sessions 3rd node has 4 broker connections as it should plus 3 anonymous connections without sessions 4th node has 4 broker connections as it should plus 1 anonymous connection without sessions 5th node has 4 broker connections as it should and no anonymous connections without sessions summary of extra connections: 5+1+3+1+0 This is also the startup sequence though there is some overlap. So far, I noticed that the "early-starters" have a higher chance of having the extra connections. When after startup each of the brokers is restarted, one-at-a-time, some of the extra connections do not re-appear. summary of extra connections now: 4+2+0+0+0 this seems to be a minimum that I cannot lower any further. screenshot from the first node (with the 4 extra connections) the hidden value is the cluster-user, redacted as it contains company specific details. !screenshot-1.png! redone this test with 3-node cluster: 2+4+0 extra anonymous connections without sessions I have another observation in this area! when the hostname of a cluster-node is not valid (yet), the broker connects to 127.0.0.1 (aka localhost, aka itself) according to the webconsole. this is a common situation when servers are created and registered on-demand. easily reproducible by just starting one cluster node that has references to a few non-existing hosts. > Also, is there any functional impact? no; everything works fine and the expected number of cluster connections is always exactly present > uneven number of connections in a cluster > - > > Key: ARTEMIS-3157 > URL: https://issues.apache.org/jira/browse/ARTEMIS-3157 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.17.0 >Reporter: Erwin Dondorp >Priority: Major > Attachments: screenshot-1.png > > > Using a cluster of 3 master + 3 slave nodes. full interconnect, each with a > fresh virtual disk. no other constructions > (bridges/federation/brokerconn/etc) and also no clients. > On each master nodes, the 2 static connections to the other master nodes are > visible, and each is marked with the dedicated cluster username. so that part > seems ok. > but without any clients having connected, there are additional connections. > the amount is not the same in each master node. Some connections show > "127.0.0.1" as address, something that is not in my configuration. none of > the extra connections have any sessions. Also the connections do not have any > sessions. > the details of an example: > * broker1: 3 connections to own slave; 2 extra to/from broker2; 1 extra > to/from backup of broker3 > * broker2: 3 connections to own slave; 2 extra to/from broker1; 2 extra > to/from broker3; 1 extra to/from slave of broker3 > * broker3: 1 connection to own slave; no other extra connections > the exact amount of connections varies a little between startups and also > seems to depend on the exact startup sequence. > my assumption is that these connections should not be present, and that this > is not intended, hence this report. my wild guess is that these are remnants > from connections that did not succeed due to the other cluster-members not > fully available yet. > When the cluster is started one node at a time, the effect seems to exists > only on the first node that was started. > Note: not related to ARTEMIS-2870 as this report is still valid in > 2.18.0-20210322.234647-43. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARTEMIS-3157) uneven number of connections in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412697#comment-17412697 ] Justin Bertram commented on ARTEMIS-3157: - Can you reproduce this on a simpler configuration (e.g. 2 live brokers, no backups)? Also, is there any functional impact? > uneven number of connections in a cluster > - > > Key: ARTEMIS-3157 > URL: https://issues.apache.org/jira/browse/ARTEMIS-3157 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.17.0 >Reporter: Erwin Dondorp >Priority: Major > > Using a cluster of 3 master + 3 slave nodes. full interconnect, each with a > fresh virtual disk. no other constructions > (bridges/federation/brokerconn/etc) and also no clients. > On each master nodes, the 2 static connections to the other master nodes are > visible, and each is marked with the dedicated cluster username. so that part > seems ok. > but without any clients having connected, there are additional connections. > the amount is not the same in each master node. Some connections show > "127.0.0.1" as address, something that is not in my configuration. none of > the extra connections have any sessions. Also the connections do not have any > sessions. > the details of an example: > * broker1: 3 connections to own slave; 2 extra to/from broker2; 1 extra > to/from backup of broker3 > * broker2: 3 connections to own slave; 2 extra to/from broker1; 2 extra > to/from broker3; 1 extra to/from slave of broker3 > * broker3: 1 connection to own slave; no other extra connections > the exact amount of connections varies a little between startups and also > seems to depend on the exact startup sequence. > my assumption is that these connections should not be present, and that this > is not intended, hence this report. my wild guess is that these are remnants > from connections that did not succeed due to the other cluster-members not > fully available yet. > When the cluster is started one node at a time, the effect seems to exists > only on the first node that was started. > Note: not related to ARTEMIS-2870 as this report is still valid in > 2.18.0-20210322.234647-43. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARTEMIS-3157) uneven number of connections in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303775#comment-17303775 ] Erwin Dondorp commented on ARTEMIS-3157: relation with ARTEMIS-2870? > uneven number of connections in a cluster > - > > Key: ARTEMIS-3157 > URL: https://issues.apache.org/jira/browse/ARTEMIS-3157 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.17.0 >Reporter: Erwin Dondorp >Priority: Minor > > Using a cluster of 3 master + 3 slave nodes. full interconnect, each with a > fresh virtual disk. > On each master nodes, the 2 static connections to the other master nodes are > visible, and each is marked with the dedicated cluster username. so that part > seems ok. > but without any clients having connected, there are additional connections. > the amount is not the same in each master node. Some connections show > "127.0.0.1" as address, something that is not in my configuration. none of > the extra connections have any sessions. > the details of an example: > * broker1: 3 connections to own slave; 2 extra to/from broker2; 1 extra > to/from backup of broker3 > * broker2: 3 connections to own slave; 2 extra to/from broker1; 2 extra > to/from broker3; 1 extra to/from slave of broker3 > * broker3: 1 connection to own slave; no other extra connections > the exact amount of connections varies a little between startups and also > seems to depend on the exact startup sequence. > my assumption is that these connections should not be present, and that this > is not intended, hence this report. my wild guess is that these are remnants > from connections that did not success due to the cluster not fully available > yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
