Hi Raghu, thanks for your help.
Just answering your previous question, the following logs were the same as
before the error, as if the pipeline were still getting the messages, for
example:
(...)
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10.
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 15.
ERROR
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 22.
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 30.
(...)
But when checking the Kafka Consumer Group, the current offset stays at 15,
the commited offset from the last processed message, before the error.
We'll file a bug, but we could now reproduce the issue in a Dev scenario.
We started the same pipeline using the direct runner, without Google
Dataflow. We blocked the Kafka Broker network and the same error was
thrown. Then we unblocked the network and the pipeline was able to
successfully process the subsequent messages.
When we started the same pipeline in the Dataflow runner and did the same
test, the same problem from our production scenario happened, Dataflow
couldn't process the new messages. Unfortunately, we've stopped the
dataflow job in production, but the problematic dev job is still running
and the log file of the VM is attached. Thank you very much.
Best regards
Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <rang...@google.com>
escreveu:
> Specifically, I am interested if you have any thread running
> 'consumerPollLoop()' [1]. There should always be one (if a worker is
> assigned one of the partitions). It is possible that KafkaClient itself is
> hasn't recovered from the group coordinator error (though unlikely).
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>
> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <rang...@google.com> wrote:
>
>> Hi Eduardo,
>>
>> In case of any error, the pipeline should keep on trying to fetch. I
>> don't know about this particular error. Do you see any others afterwards in
>> the log?
>> Couple of things you could try if the logs are not useful :
>> - login to one of the VMs and get stacktrace of java worker (look for a
>> container called java-streaming)
>> - file a support bug or stackoverflow question with jobid so that
>> Dataflow oncall can take a look.
>>
>> Raghu.
>>
>>
>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>> eduardo.sold...@arquivei.com.br> wrote:
>>
>>> Hi,
>>> We have a Apache Beam pipeline running in Google Dataflow using KafkaIO.
>>> Suddenly the pipeline stop fetching Kafka messages at all, as our other
>>> workers from other pipelines continued to get Kafka messages.
>>>
>>> At the moment it stopped we got these messages:
>>>
>>> I [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch
>>> request (sessionId=1396189203, epoch=2431598) to node 3:
>>> org.apache.kafka.common.errors.DisconnectException.
>>> I [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator
>>> 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will
>>> attempt rediscovery
>>> I [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group
>>> coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>
>>> And then the pipeline stopped reading the messages.
>>>
>>> This is the KafkaIO setup we have:
>>>
>>> KafkaIO.read[String,String]()
>>> .withBootstrapServers(server)
>>> .withTopic(topic)
>>> .withKeyDeserializer(classOf[StringDeserializer])
>>> .withValueDeserializer(classOf[StringDeserializer])
>>> .updateConsumerProperties(properties)
>>> .commitOffsetsInFinalize()
>>> .withoutMetadata()
>>>
>>> Any help will be much appreciated.
>>>
>>> Best regards,
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
--
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>
--- Threads (106): [Thread[Thread-1508,1,main], Thread[Thread-1550,1,main],
Thread[Thread-1575,1,main], Thread[Thread-1556,1,main],
Thread[Thread-1505,1,main], Thread[Thread-1514,1,main],
Thread[Thread-1527,1,main], Thread[Thread-1513,1,main],
Thread[Thread-1483,1,main], Thread[Thread-1472,1,main],
Thread[Thread-1524,1,main], Thread[Thread-1494,1,main],
Thread[Thread-1529,1,main], Thread[Thread-1545,1,main],
Thread[Thread-1475,1,main], Thread[Thread-1549,1,main],
Thread[Thread-1482,1,main], Thread[Thread-1511,1,main],
Thread[Thread-1489,1,main], Thread[Thread-1559,1,main],
Thread[Thread-1506,1,main], Thread[Thread-1555,1,main],
Thread[Thread-1486,1,main], Thread[Thread-1492,1,main],
Thread[Thread-1487,1,main], Thread[Thread-1536,1,main],
Thread[Thread-1560,1,main], Thread[Thread-1573,1,main],
Thread[Thread-1491,1,main], Thread[Thread-1481,1,main],
Thread[Thread-1478,1,main], Thread[Thread-1558,1,main],
Thread[Thread-1572,1,main], Thread[Thread-1565,1,main],
Thread[Thread-1476,1,main], Thread[Thread-1518,1,main],
Thread[Thread-1567,1,main], Thread[Thread-1562,1,main],
Thread[Thread-1569,1,main], Thread[Thread-1499,1,main],
Thread[Thread-1577,1,main], Thread[Thread-1496,1,main],
Thread[Thread-1485,1,main], Thread[Thread-1574,1,main],
Thread[Thread-1546,1,main], Thread[Thread-1477,1,main],
Thread[Thread-1523,1,main], Thread[Thread-1543,1,main],
Thread[Thread-1547,1,main], Thread[Thread-1528,1,main],
Thread[Thread-1548,1,main], Thread[Thread-1509,1,main],
Thread[Thread-1484,1,main], Thread[Thread-1479,1,main],
Thread[Thread-1534,1,main], Thread[Thread-1561,1,main],
Thread[Thread-1498,1,main], Thread[Thread-1570,1,main],
Thread[Thread-1538,1,main], Thread[Thread-1535,1,main],
Thread[Thread-1474,1,main], Thread[Thread-1564,1,main],
Thread[Thread-1557,1,main], Thread[Thread-1544,1,main],
Thread[Thread-1515,1,main], Thread[Thread-1516,1,main],
Thread[Thread-1500,1,main], Thread[Thread-1537,1,main],
Thread[Thread-1568,1,main], Thread[Thread-1532,1,main],
Thread[Thread-1502,1,main], Thread[Thread-1553,1,main],
Thread[Thread-1566,1,main], Thread[Thread-1563,1,main],
Thread[Thread-1542,1,main], Thread[Thread-1525,1,main],
Thread[Thread-1501,1,main], Thread[Thread-1541,1,main],
Thread[Thread-1493,1,main], Thread[Thread-1533,1,main],
Thread[Thread-1554,1,main], Thread[Thread-1571,1,main],
Thread[Thread-1488,1,main], Thread[Thread-1470,1,main],
Thread[Thread-1512,1,main], Thread[Thread-1490,1,main],
Thread[Thread-1521,1,main], Thread[Thread-1473,1,main],
Thread[Thread-1522,1,main], Thread[Thread-1540,1,main],
Thread[Thread-1551,1,main], Thread[Thread-1504,1,main],
Thread[Thread-1526,1,main], Thread[Thread-1552,1,main],
Thread[Thread-1495,1,main], Thread[Thread-1480,1,main],
Thread[Thread-1517,1,main], Thread[Thread-1503,1,main],
Thread[Thread-1539,1,main], Thread[Thread-1497,1,main],
Thread[Thread-1530,1,main], Thread[Thread-1507,1,main],
Thread[Thread-1531,1,main], Thread[Thread-1510,1,main],
Thread[Thread-1520,1,main], Thread[Thread-1519,1,main]] State: TIMED_WAITING
stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
--- Threads (10): [Thread[GetDataThread3,5,main],
Thread[GetDataThread7,5,main], Thread[GetDataThread4,5,main],
Thread[GetDataThread1,5,main], Thread[GetDataThread9,5,main],
Thread[GetDataThread0,5,main], Thread[GetDataThread2,5,main],
Thread[GetDataThread5,5,main], Thread[GetDataThread8,5,main],
Thread[GetDataThread6,5,main]] State: WAITING stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
com.google.cloud.dataflow.worker.MetricTrackingWindmillServerStub.getDataLoop(MetricTrackingWindmillServerStub.java:138)
com.google.cloud.dataflow.worker.MetricTrackingWindmillServerStub.access$000(MetricTrackingWindmillServerStub.java:45)
com.google.cloud.dataflow.worker.MetricTrackingWindmillServerStub$1.run(MetricTrackingWindmillServerStub.java:123)
--- Threads (5): [Thread[qtp1603696865-33,5,main],
Thread[qtp1603696865-27,5,main], Thread[qtp1603696865-31,5,main],
Thread[qtp1603696865-32,5,main], Thread[qtp1603696865-30,5,main]] State:
TIMED_WAITING stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
java.lang.Thread.run(Thread.java:745)
--- Threads (2): [Thread[RefreshActiveWork,5,main],
Thread[GlobalWorkerUpdatesTimer,5,main]] State: TIMED_WAITING stack: ---
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
--- Threads (2): [Thread[pool-4-thread-1,5,main],
Thread[Scheduler-603658030,5,main]] State: TIMED_WAITING stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
--- Threads (2): [Thread[DestroyJavaVM,5,main], Thread[Signal
Dispatcher,9,system]] State: RUNNABLE stack: ---
--- Threads (1): [Thread[state-sampler-0,5,main]] State: TIMED_WAITING stack:
---
java.lang.Thread.sleep(Native Method)
com.google.cloud.dataflow.worker.util.common.worker.ExecutionStateSampler.lambda$start$0(ExecutionStateSampler.java:87)
com.google.cloud.dataflow.worker.util.common.worker.ExecutionStateSampler$$Lambda$45/203149502.call(Unknown
Source)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[MemoryMonitor,1,main]] State: TIMED_WAITING stack: ---
java.lang.Object.wait(Native Method)
com.google.cloud.dataflow.worker.util.MemoryMonitor.run(MemoryMonitor.java:464)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[Thread-1576,1,main]] State: TIMED_WAITING stack: ---
java.lang.Thread.sleep(Native Method)
java.lang.Thread.sleep(Thread.java:340)
java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
com.google.cloud.dataflow.worker.repackaged.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:285)
com.google.cloud.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.advance(WorkerCustomSources.java:804)
com.google.cloud.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:770)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:360)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:193)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1211)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:137)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:959)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[RMI TCP Accept-0,5,system]] State: RUNNABLE stack: ---
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:545)
java.net.ServerSocket.accept(ServerSocket.java:513)
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[RMI TCP Accept-5555,5,system]] State: RUNNABLE stack:
---
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:545)
java.net.ServerSocket.accept(ServerSocket.java:513)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
java.lang.Thread.run(Thread.java:745)
--- Threads (1):
[Thread[qtp1603696865-26-selector-ServerConnectorManager@25c6a7b1/0,5,main]]
State: RUNNABLE stack: ---
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[Finalizer,8,system]] State: WAITING stack: ---
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
--- Threads (1): [Thread[pool-3-thread-1,5,main]] State: WAITING stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
--- Threads (1):
[Thread[qtp1603696865-28-acceptor-0@6cc3ddd7-ServerConnector@62042e3f{HTTP/1.1}{0.0.0.0:8081},5,main]]
State: RUNNABLE stack: ---
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
com.google.cloud.dataflow.worker.repackaged.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[CommitThread,10,main]] State: WAITING stack: ---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$WeightedBoundedQueue.take(StreamingDataflowWorker.java:280)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.commitLoop(StreamingDataflowWorker.java:1329)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$400(StreamingDataflowWorker.java:137)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$2.run(StreamingDataflowWorker.java:595)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[DispatchThread,1,main]] State: TIMED_WAITING stack: ---
java.lang.Thread.sleep(Native Method)
java.lang.Thread.sleep(Thread.java:340)
java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
com.google.cloud.dataflow.worker.repackaged.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:285)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.sleep(StreamingDataflowWorker.java:850)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.dispatchLoop(StreamingDataflowWorker.java:881)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$200(StreamingDataflowWorker.java:137)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$1.run(StreamingDataflowWorker.java:579)
java.lang.Thread.run(Thread.java:745)
--- Threads (1): [Thread[Reference Handler,10,system]] State: WAITING stack: ---
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
--- Threads (1): [Thread[pool-6-thread-1,5,main]] State: TIMED_WAITING stack:
---
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
