[
https://issues.apache.org/jira/browse/CASSANDRA-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-5345:
-----------------------------------------
Assignee: Joshua McKenzie (was: Ryan McGuire)
> Potential problem with GarbageCollectorMXBean
> ---------------------------------------------
>
> Key: CASSANDRA-5345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5345
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.7
> Environment: JVM:JVM vendor/version: Java HotSpot(TM) 64-Bit Server
> VM/1.6.0_30 typical 6 node 2 availability zone Mutli DC cluster on linux vms
> with
> and mx4j-tools.jar and jna.jar both on path. Default configuration bar token
> setup(equispaced), sensible cassandra-topology.properties file and use of
> said snitch.
> Reporter: Matt Byrd
> Assignee: Joshua McKenzie
>
> I am not certain this is definitely a bug, but I thought it might be worth
> posting to see if someone with more JVM//JMX knowledge could disprove my
> reasoning. Apologies if I've failed to understand something.
> We've seen an intermittent problem where there is an uncaught exception in
> the scheduled task of logging gc results in GcInspector.java:
> {code}
> ...
> ERROR [ScheduledTasks:1] 2013-03-08 01:09:06,335
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[ScheduledTasks:1,5,main]
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy0.getName(Unknown Source)
> at
> org.apache.cassandra.service.GCInspector.logGCResults(GCInspector.java:95)
> at
> org.apache.cassandra.service.GCInspector.access$000(GCInspector.java:41)
> at org.apache.cassandra.service.GCInspector$1.run(GCInspector.java:85)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.management.InstanceNotFoundException:
> java.lang:name=ParNew,type=GarbageCollector
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at
> com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:106)
> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:148)
> at
> javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:248)
> ... 13 more
> ...
> {code}
> I think the problem, may be caused by the following reasoning:
> In GcInspector we populate a list of mxbeans when the GcInspector instance is
> instantiated:
> {code}
> ...
> List<GarbageCollectorMXBean> beans = new ArrayList<GarbageCollectorMXBean>();
> MBeanServer server = ManagementFactory.getPlatformMBeanServer();
> try
> {
> ObjectName gcName = new
> ObjectName(ManagementFactory.GARBAGE_COLLECTOR_MXBEAN_DOMAIN_TYPE + ",*");
> for (ObjectName name : server.queryNames(gcName, null))
> {
> GarbageCollectorMXBean gc =
> ManagementFactory.newPlatformMXBeanProxy(server, name.getCanonicalName(),
> GarbageCollectorMXBean.class);
> beans.add(gc);
> }
> }
> catch (Exception e)
> {
> throw new RuntimeException(e);
> }
> ...
> {code}
> Cassandra then periodically calls:
> {code}
> ...
> private void logGCResults()
> {
> for (GarbageCollectorMXBean gc : beans)
> {
> Long previousTotal = gctimes.get(gc.getName());
> ...
> {code}
> In the oracle javadocs, they seem to suggest that these beans could disappear
> at any time.(I'm not sure why when or how this might happen)
> http://docs.oracle.com/javase/6/docs/api/
> See: getGarbageCollectorMXBeans
> {code}
> ...
> public static List<GarbageCollectorMXBean> getGarbageCollectorMXBeans()
> Returns a list of GarbageCollectorMXBean objects in the Java virtual machine.
> The Java virtual machine may have one or more GarbageCollectorMXBean objects.
> It may add or remove GarbageCollectorMXBean during execution.
> Returns:
> a list of GarbageCollectorMXBean objects.
> ...
> {code}
> Correct me if I'm wrong, but do you think this might be causing the problem?
> That somehow the JVM decides to remove the GarbageCollectorMXBean temporarily
> or permanently (causing said exception) and if this is expected behaviour,
> should it be handled in some way?
> Also I'd like to point out that this may be an issue on other versions as
> well as I don't believe this code has changed in quite a long time.
> Unfortunately I haven't been able to reproduce this outside of the production
> environment, if you have any tips, questions or are able to explain//disprove
> my concerns, I'd be very grateful.
> Thanks,
> Matt
--
This message was sent by Atlassian JIRA
(v6.2#6252)
