I experimented a bit more. In the waitForZooKeeperClientThreads
method, I added a "thread.interrupt" call inside the "while
(thread.isAlive())" loop. It seems like the thread is interrupted
after a few seconds, but then is reconnected. Darn that resiliency!
I've watched it reconnect four times before I killed the process. Can
I reset the retry number somehow? Am I too far down the rabbit hole?
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1045)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1101)
2014-11-30 09:29:32,596 INFO
[com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:20000
2014-11-30 09:29:32,597 INFO
[com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(947)) - Socket
connection established to localhost/127.0.0.1:20000, initiating
session
2014-11-30 09:29:32,599 INFO
[com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
zookeeper.ClientCnxn (ClientCnxn.java:readConnectResult(736)) -
Session establishment complete on server localhost/127.0.0.1:20000,
sessionid = 0x1
On Sun, Nov 30, 2014 at 9:10 AM, David Medinets
<david.medin...@gmail.com> wrote:
> Right again. I changed the package name to "jaredwinick" and the
> "sealing violation" issue was resolved. But the JIRA ticket comments
> were correct about the code's being brittle. It has a String literal
> that refers to "org.apache.accumulo.core.zookeeper.ZooSession"which
> does not exist in Accumulo v1.6.1.
>
> On Sun, Nov 30, 2014 at 12:05 AM, Josh Elser <josh.el...@gmail.com> wrote:
>> I believe that means you re-used the package org.apache.accumulo.core.util
>> which was already defined in accumulo-core.jar.
>>
>> Don't do that.
>>
>>
>> David Medinets wrote:
>>>
>>> Thanks Josh. That explanation makes sense. I added the referenced
>>> ClientThreads.shutdownNow() method to my code and ran into the
>>> following:
>>>
>>> java.lang.SecurityException: sealing violation: package
>>> org.apache.accumulo.core.util is sealed
>>>
>>>
>>> I can live with this issue until v2.0 since it happens when the JVM is
>>> exiting. And I'm just writing demonstration code for no particular
>>> reason. At the least, we'll have a small project to test the fix
>>> against.
>>>
>>>
>>> On Sat, Nov 29, 2014 at 10:35 PM, Josh Elser<josh.el...@gmail.com> wrote:
>>>>
>>>> From http://mojo.codehaus.org/exec-maven-plugin/usage.html,
>>>>
>>>> "By default daemon threads are joined and interrupted once all known non
>>>> daemon threads have quitted"
>>>>
>>>> So, your main thread finished, but before the exec plugin stopped the
>>>> application, it tried to interrupt the other non-daemon threads that were
>>>> still running (which should be a sign for them quit on their own).
>>>>
>>>> However, it's a known issue[1] that we have ZooKeeper and Thrift
>>>> resources
>>>> that are not cleaned up (a problem which is slated to be fixed by
>>>> Christopher's 2.0 API refactoring).
>>>>
>>>> Hope that makes sense.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/ACCUMULO-2113
>>>>
>>>>
>>>> David Medinets wrote:
>>>>>
>>>>> This might be a stupid David issue but it's at least marginally
>>>>> related to Accumulo so I'm hoping someone will recognize the problem
>>>>> and help.
>>>>>
>>>>> I am seeing this error when my program is run via maven:
>>>>>
>>>>> java.lang.InterruptedException: sleep interrupted
>>>>>
>>>>> The program connects to a MiniAccumuloCluster and creates a table if
>>>>> it does not exist. The program works fine inside Netbeans.
>>>>>
>>>>> In both NetBeans and Maven, the program connects to the
>>>>> MiniAccumuloCluster just fine. The shutdown is the problem.
>>>>>
>>>>> I have a github project which should demonstrate the issue.
>>>>>
>>>>> # First clone the project
>>>>> git clone https://github.com/medined/jodoc.git
>>>>> cd jodoc
>>>>>
>>>>> # Then build the java image
>>>>> cd java_zulu7
>>>>> ./build_image.sh
>>>>>
>>>>> # Then compile and create a jar file
>>>>> mvn package
>>>>>
>>>>> # Then build the jodoc image
>>>>> ./build_image.sh
>>>>>
>>>>> # And run the jodoc image. This image runs a MiniAccumuloCluster
>>>>> # using port 20000 for Zookeeper.
>>>>> ./run_image.sh
>>>>>
>>>>> # Finally run the client program via Maven
>>>>> mvn exec:java -Dexec.mainClass="com.codebits.jodoc.WriteAndReadDriver"
>>>>> -DZOOKEEPER_PORT=20000
>>>>>
>>>>> Here is the full error stack:
>>>>>
>>>>> java.lang.InterruptedException: sleep interrupted
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.closeConnections(ThriftTransportPool.java:137)
>>>>> at
>>>>>
>>>>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:147)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 2014-11-29 21:05:18,011 ERROR
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main()-EventThread]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(513)) - Event thread exiting
>>>>> due to interruption
>>>>> java.lang.InterruptedException
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>>>>> 2014-11-29 21:05:18,012 INFO
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main()-EventThread]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(516)) - EventThread shut
>>>>> down
>>>>> 2014-11-29 21:05:28,010 WARN
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>>> 0x149fe6a2c5a0007 for server localhost/127.0.0.1:20000, unexpected
>>>>> error, closing socket connection and attempting reconnect
>>>>> java.nio.channels.ClosedByInterruptException
>>>>> at
>>>>>
>>>>> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:890)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1154)
>>>>> 2014-11-29 21:05:29,596 INFO
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>>> socket connection to server localhost/127.0.0.1:20000
>>>>> 2014-11-29 21:05:29,596 INFO
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(947)) - Socket
>>>>> connection established to localhost/127.0.0.1:20000, initiating
>>>>> session
>>>>> 2014-11-29 21:05:29,598 INFO
>>>>> [com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000)]
>>>>> zookeeper.ClientCnxn (ClientCnxn.java:readConnectResult(736)) -
>>>>> Session establishment complete on server localhost/127.0.0.1:20000,
>>>>> sessionid = 0x149fe6a2c5a0007, negotiated timeout = 30000
>>>>> [WARNING] thread
>>>>>
>>>>>
>>>>> Thread[com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000),5,com.codebits.jodoc.WriteAndReadDriver]
>>>>> was interrupted but is still alive after waiting at least 15000msecs
>>>>> [WARNING] thread
>>>>>
>>>>>
>>>>> Thread[com.codebits.jodoc.WriteAndReadDriver.main(localhost:20000),5,com.codebits.jodoc.WriteAndReadDriver]
>>>>> will linger despite being asked to die via interruption
>>>>> [WARNING] thread Thread[Thrift Connection Pool
>>>>> Checker,5,com.codebits.jodoc.WriteAndReadDriver] will linger despite
>>>>> being asked to die via interruption
>>>>> [WARNING] NOTE: 2 thread(s) did not finish despite being asked to via
>>>>> interruption. This is not a problem with exec:java, it is a problem
>>>>> with the running code. Although not serious, it should be remedied.
>>>>> [WARNING] Couldn't destroy threadgroup
>>>>>
>>>>>
>>>>> org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.codebits.jodoc.WriteAndReadDriver,maxpri=10]
>>>>> java.lang.IllegalThreadStateException
>>>>> at java.lang.ThreadGroup.destroy(ThreadGroup.java:778)
>>>>> at
>>>>> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
>>>>> at
>>>>>
>>>>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>>>>> at
>>>>>
>>>>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
>>>>> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
>>>>> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>>>>> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
>>>>> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
>>>>> at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:483)
>>>>> at
>>>>>
>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>>>>> at
>>>>>
>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>>>>> at
>>>>>
>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>>>>> at
>>>>>
>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>>>>>
>>>>> ----
>>>>>
>>>>> Any ideas?
