Hi community, I wanted to know if the solution I supplied can fix the IllegalMonitorStateException issue. Our work is pending on this and we'd like to proceed ASAP. Sorry for bothering.
On Mon, Mar 23, 2015 at 4:32 PM, tao xiao <xiaotao...@gmail.com> wrote: > I think I worked out the answer to question 1. > java.lang.IllegalMonitorStateException > was thrown due to no ownership of ReentrantLock when trying to call await() > on the lock condition. > > Here is the code snippet from the AbstractFetcherThread.scala in trunk > > partitionMapLock synchronized { > partitionsWithError ++= partitionMap.keys > // there is an error occurred while fetching partitions, sleep > a while > partitionMapCond.await(fetchBackOffMs, TimeUnit.MILLISECONDS) > } > > as shown above partitionMapLock is not acquired before calling > partitionMapCond.await > > we can fix this by explicitly calling partitionMapLock.lock(). below code > block should work > > inLock(partitionMapLock) { > partitionsWithError ++= partitionMap.keys > // there is an error occurred while fetching partitions, sleep > a while > partitionMapCond.await(fetchBackOffMs, TimeUnit.MILLISECONDS) > } > > On Mon, Mar 23, 2015 at 1:50 PM, tao xiao <xiaotao...@gmail.com> wrote: > >> Hi, >> >> I was running a mirror maker and got >> java.lang.IllegalMonitorStateException that caused the underlying fetcher >> thread completely stopped. Here is the log from mirror maker. >> >> [2015-03-21 02:11:53,069] INFO Reconnect due to socket error: >> java.io.EOFException: Received -1 when reading from channel, socket has >> likely been closed. (kafka.consumer.SimpleConsumer) >> [2015-03-21 02:11:53,081] WARN >> [ConsumerFetcherThread-localhost-1426853968478-4ebaa354-0-6], Error in >> fetch Name: FetchRequest; Version: 0; CorrelationId: 2398588; ClientId: >> phx-slc-mm-user-3; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; >> RequestInfo: [test.topic,0] -> PartitionFetchInfo(3766065,1048576). >> Possible cause: java.nio.channels.ClosedChannelException >> (kafka.consumer.ConsumerFetcherThread) >> [2015-03-21 02:11:53,083] ERROR >> [ConsumerFetcherThread-localhost-1426853968478-4ebaa354-0-6], Error due to >> (kafka.consumer.ConsumerFetcherThread) >> java.lang.IllegalMonitorStateException >> at >> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.fullyRelease(AbstractQueuedSynchronizer.java:1723) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2166) >> at >> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:106) >> at >> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:90) >> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) >> [2015-03-21 02:11:53,083] INFO >> [ConsumerFetcherThread-localhost-1426853968478-4ebaa354-0-6], Stopped >> (kafka.consumer.ConsumerFetcherThread) >> >> I am still investigating what caused the connection error on server side >> but I have a couple of questions related to mirror maker itself >> >> 1. What is root cause of java.lang.IllegalMonitorStateException? As shown >> in the AbstractFetcherThread source the fetcher thread should catch the >> java.io.EOFException thrown from underlying simplyConsumer and sleep a >> while before next run. >> 2. Mirror maker is unaware of the termination of fetcher thread. That >> makes it unable to detect the failure and trigger rebalancing. I have 3 >> mirror maker instances running in 3 different machines listening to the >> same topic. I would expect the mirror maker will release the partition >> ownership when underlying fetcher thread terminates so that rebalancing can >> be triggered.but in fact this is not the case. is this expected behavior or >> do I miss configure anything? >> >> I am running the trunk version as of commit >> 82789e75199fdc1cae115c5c2eadfd0f1ece4d0d >> >> -- >> Regards, >> Tao >> > > > > -- > Regards, > Tao > -- Regards, Tao