Bugs item #804491, was opened at 2003-09-11 14:56
Message generated for change (Comment added) made by bcotton969
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=804491&group_id=22866
Category: None
Group: None
>Status: Closed
>Resolution: Works For Me
Priority: 5
Submitted By: Bob Cotton (bcotton969)
Assigned to: Nobody/Anonymous (nobody)
Summary: Unable to add new nodes to cluster
Initial Comment:
OS: Solaris 8
JDK: 1.3.1
Logs as attachment
We run a cluster of EJB vms with JNDI and EJBs as
seperate cluster services.
Lately, in production, the cluster gets into a state
where each node thinks there are no other nodes in the
cluster.
The clients are ok with this because the client-side
stubs remeber the last cluster configuration and
continue to RR over each vm.
However, when starting new VMs that want to join the
cluster, they do not join. If multiple VMs are started,
none of the new VMs cluster with each other.
To correct this problem we have to bring down the whole
system.
Cluster logs with full TRACE debugging are attached.
----------------------------------------------------------------------
>Comment By: Bob Cotton (bcotton969)
Date: 2003-09-23 21:07
Message:
Logged In: YES
user_id=424902
Solaris multipathing was the problem here. We were using the
FAILOVER flags, which by default also tries to load balance
outgoing traffic over all NICs in the group.
By adding 'standby' to the config of the interface, the
problem went away.
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-15 22:06
Message:
Logged In: YES
user_id=424902
> I see. So we need to really turn autodiscovery off for
> HANamingService, because it is clearly (as seen in the logs)
> running.
ok, how do we do this?
Bob
----------------------------------------------------------------------
Comment By: Bela Ban (belaban)
Date: 2003-09-15 21:37
Message:
Logged In: YES
user_id=34890
> Reading the docs on jnp.disableDiscovery it seems that
> autoDiscovery is used if there is no Context.PROVIDER_URL
> specified or if there are no naming services found.
That's true, but you can separately turn autodiscovery off
altogether (I forgot the flag).
> From the EJB clients we do set java.naming.provider.url to
> the list of ejb servers that are ALWAYS running. So the URL
> is set and the naming servers are always there.
I see. So we need to really turn autodiscovery off for
HANamingService, because it is clearly (as seen in the logs)
running.
> Also, the "AutoDiscovery" in the logs are just :
> clusterprod-ejb1.log:2003-09-11 00:38:30,445 TRACE
> [org.jboss.ha.jndi.HANamingService$AutomaticDiscovery]
> HA-JNDI AutomaticDiscovery waiting for queries...
Lots of theselogging statements...
> The CPU usage on the system is only 50% (this is an 8-way
> machine)
It must have been higher; but it looks as if at some time,
the HANaming Service is restarted, so the spinnig goes away.
However, if during the spinning, some new member tries to
join, this may as well prevent it.
> So if the machine gets busy does this mean that the clusters
> will break?
Depends on how busy they get...
Bela
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-15 20:38
Message:
Logged In: YES
user_id=424902
Reading the docs on jnp.disableDiscovery it seems that
autoDiscovery is used if there is no Context.PROVIDER_URL
specified or if there are no naming services found.
>From the EJB clients we do set java.naming.provider.url to
the list of ejb servers that are ALWAYS running. So the URL
is set and the naming servers are always there.
The EJB servers (which are the only vms that clustered) do
not have java.naming.provider.url set, but all EJB->EJB
calls should be in-vm.
Also, the "AutoDiscovery" in the logs are just :
clusterprod-ejb1.log:2003-09-11 00:38:30,445 TRACE
[org.jboss.ha.jndi.HANamingService$AutomaticDiscovery]
HA-JNDI AutomaticDiscovery waiting for queries...
How is this caused by the client, which shold not be doing
autoDiscovery?
The CPU usage on the system is only 50% (this is an 8-way
machine)
So if the machine gets busy does this mean that the clusters
will break?
----------------------------------------------------------------------
Comment By: Sacha Labourey (slaboure)
Date: 2003-09-12 18:36
Message:
Logged In: YES
user_id=95900
It is not possible to disable it on the server (only on the
client).
Cheers,
sacha
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-12 15:57
Message:
Logged In: YES
user_id=424902
How do you disable automatic discovery?
----------------------------------------------------------------------
Comment By: Sacha Labourey (slaboure)
Date: 2003-09-12 06:43
Message:
Logged In: YES
user_id=95900
3.2.1 is old, it is from May, I am pretty sure this has been
fixed after, in some of the 3.2.2RCx
----------------------------------------------------------------------
Comment By: Bela Ban (belaban)
Date: 2003-09-12 01:14
Message:
Logged In: YES
user_id=34890
Looks like the AutomaticDiscovery endless loop on
HANamingService.stop() is still there. Grep for
'AutomaticDiscovery' and you'll see that 35% of *all* logs
statements are these.
This sucks a lot of CPU out of the system, maybe the JOINs
run into timeouts all the time, that's why we never see each
other.
Sacha: I thought this was fixed in 3.2.1 ?
Bob: can you disable automaticDiscovery for now and test again ?
Bela
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-11 18:26
Message:
Logged In: YES
user_id=424902
Here is jndi-cluster-service.xml
<?xml version="1.0" encoding="UTF-8"?>
<server>
<classpath codebase="lib" archives="jbossha.jar"/>
<!--
====================================================================
-->
<!-- Cluster Partition: defines cluster
-->
<!-- Based on cluser-services.xml from
jboss/server/all/deploy/ -->
<!--
====================================================================
-->
<mbean code="org.jboss.ha.framework.server.ClusterPartition"
name="jboss:service=JndiPartition">
<!-- Name of the partition being built -->
<attribute name="PartitionName">DefaultPartition</attribute>
<!-- Determine if deadlock detection is enabled -->
<attribute name="DeadlockDetection">False</attribute>
<!-- The JavaGroups protocol configuration -->
<attribute name="PartitionConfig">
<Config>
<!-- UDP: if you have a multihomed machine,
set the bind_addr attribute to the appropriate
NIC IP address -->
<!-- UDP: On Windows machines, because of the media
sense feature
being broken with multicast (even after
disabling media sense)
set the loopback attribute to true -->
<UDP mcast_addr="228.1.2.3" mcast_port="45566"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
loopback="false" />
<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true" />
<MERGE2 min_interval="5000" max_interval="10000" />
<FD up_thread="true" down_thread="true"
timeout="2500" max_tries="5" shun="true"/>
<VERIFY_SUSPECT timeout="3000" num_msgs="3"
up_thread="true" down_thread="true" />
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true" />
<pbcast.NAKACK gc_lag="50"
retransmit_timeout="300,600,1200,2400,4800"
up_thread="true" down_thread="true" />
<UNICAST timeout="5000" window_size="100"
min_threshold="10"
down_thread="true" />
<FRAG frag_size="8192"
down_thread="true" up_thread="true" />
<pbcast.GMS join_timeout="5000"
join_retry_timeout="2000"
shun="true" print_local_addr="true" />
<pbcast.STATE_TRANSFER up_thread="true"
down_thread="true" />
</Config>
</attribute>
<attribute name="PartitionName">Jndi</attribute>
</mbean>
<!--
====================================================================
-->
<!-- HA JNDI
-->
<!--
====================================================================
-->
<mbean code="org.jboss.ha.jndi.HANamingService"
name="jboss:service=HAJNDI">
<depends>jboss:service=JndiPartition</depends>
<!-- Name of the partition to which the service is
linked -->
<attribute name="PartitionName">Jndi</attribute>
<!-- RmiPort to be used by the HA-JNDI service
once bound. 0 => auto. -->
<attribute name="RmiPort">0</attribute>
<!-- Port on which the HA-JNDI stub is made available -->
<attribute name="Port">1098</attribute>
<!-- Backlog to be used for client-server RMI
invocations during JNDI queries -->
<attribute name="Backlog">50</attribute>
<!-- Client socket factory to be used for client-server
RMI invocations during JNDI queries -->
<!--attribute
name="ClientSocketFactory">custom</attribute-->
<!-- Server socket factory to be used for client-server
RMI invocations during JNDI queries -->
<!--attribute
name="ServerSocketFactory">custom</attribute-->
</mbean>
<mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
name="jboss:service=invoker,type=jrmpha">
<!--
<attribute name="RMIObjectPort">0</attribute>
<attribute name="RMIClientSocketFactory">custom</attribute>
<attribute name="RMIServerSocketFactory">custom</attribute>
<attribute name="RMIServerSocketAddr">custom</attribute>
-->
</mbean>
</server>
>From the service-bindings.xml
<service-config name="jboss:service=JndiPartition"
delegateClass="org.jboss.services.binding.AttributeMappingDelegate">
<delegate-config>
<attribute name="PartitionName"
>ProdJndi</attribute>
</delegate-config>
<binding/>
</service-config>
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-11 18:17
Message:
Logged In: YES
user_id=424902
This is jboss 3.2.1 with a new javagroups.jar that I
received from you.
Decompile of org.javagroups.Version yields:
public static String version = "2.0.8";
public static byte[] version_id = { 48, 50, 48, 56 };
public static String cvs
= "$Id: Version.java,v 1.19 2003/04/04 22:54:12
belaban Exp $";
----------------------------------------------------------------------
Comment By: Sacha Labourey (slaboure)
Date: 2003-09-11 15:20
Message:
Logged In: YES
user_id=95900
Which specific JBoss release? JavaGroups bugs have been
fixed in recent jboss release that fix that kind of merge
issues, you may want to try it.
----------------------------------------------------------------------
Comment By: Bob Cotton (bcotton969)
Date: 2003-09-11 15:02
Message:
Logged In: YES
user_id=424902
Logs too big to attach. Logs can be accessed at:
http://clubcotton.homeip.net/jboss/synxis-cluster-logs.zip
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=804491&group_id=22866
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
JBoss-Development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development