Not sure if it is related or helpful, but while debugging tests for SOLR-7468
yesterday, I encountered this
java.lang.NoSuchFieldError:
totalTermCount
few times, I had to forcefully clean at root of the project and it worked. I
remember Anshum had to do that clean thing more than once to make it work and
he remarked "don't ask why".
Sent from my Windows Phone
-----Original Message-----
From: "Erick Erickson" <erickerick...@gmail.com>
Sent: 5/23/2015 6:15 AM
To: "dev@lucene.apache.org" <dev@lucene.apache.org>
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
OK, this is somewhat weird. I still have the original tree that I
checked in from which was up to date before I committed the code and
the tests run from there fine. But a current trunk fails every time.
Now, the machine it works on is my Mac Pro, and the failures are on my
MacBook so there may be something going on there.
I've got to leave for a while, I'll copy the tree that works on the
Pro, update the copy and see if this test fails when I get back. If
they fail, I can diff the trees to see what changed and see if I can
make any sense out of this.
I can always @Ignore this test to cut down on the noise, probably do
that tonight if I don't have any revelations.
I see this stack trace which makes no sense to me whatsoever (see the
lines with lots of ***** in front). I looked at where the code
originates (BufferedUpdatesStream[277]) and it looks like this:
if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
And it's telling me there's no such field? Whaaaaatttt????
Which is freaking me out since I don't see how this would trigger the
exception. Is this a red herring? And, of course, this doesn't fail in
IntelliJ but it does fail every time from the shell. Siiiigggghhh.
Of course if this were something fundamental to Lucene, it seems like
this would be failing all over the place so I assume it's something to
do with CDCR... But what do I know?
1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
status=0 QTime=8
************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
c:source_collection s:shard1 r:core_node3
x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
totalTermCount
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
************* [junit4] 2> at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
[junit4] 2> at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
[junit4] 2> at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
[junit4] 2> at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
[junit4] 2> at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
[junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
[junit4] 2> ... 23 more
On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <erickerick...@gmail.com> wrote:
> Looking......
>
>
> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> <jenk...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 4 tests failed.
>> FAILED:
>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> Some resources were not closed, shutdown, or released.
>>
>> Stack Trace:
>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>> released.
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at org.junit.Assert.assertTrue(Assert.java:43)
>> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED:
>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> 5 threads leaked from SUITE scope at
>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>> Thread[id=4675,
>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4)
>> Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> 1) Thread[id=4675,
>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4676,
>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED:
>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1)
>> Thread[id=4675,
>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2)
>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>> group=TGRP-CdcrReplicationDistributedZkTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.ut
[The entire original message is not included.]
