Re: REST API requests hang with no response

Devin Anderson Thu, 20 Feb 2020 14:56:14 -0800

Hi Ilya,

I'm attaching the `jstack` dump to this message. When I took the dump, therewere ten requests that had been made to the Apache Ignite REST API using POSTdata of the form:


    cmd=add&key=[key]&value=[value]

... where [key] and [value] were proper URI encoded data.

Thanks in advance for any help. I'll take a look as well and try to figure outwhat's going on.


Devin

On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:

Hello!

Please collect thread dump (jstack) from affected node, share it with us.

Regards,
--
Ilya Kasnacheev

чт, 20 февр. 2020 г. в 16:17, Devin Anderson <dande...@akamai.com<mailto:dande...@akamai.com>>:


    Hi all,

    I'm seeing issues wherein the Apache Ignite REST API appears to accept
    requests, but doesn't ever reply.  This doesn't always happen; for
    example, if
    I make a request that I expect the API to reject, I get back a response:

    ----------

    # curl -v -X GET 'http://127.0.0.1:8080/ignite
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
     > GET /ignite HTTP/1.1
     > User-Agent: curl/7.35.0
     > Host: 127.0.0.1:8080
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
     > Accept: */*
     >
    < HTTP/1.1 400 Bad Request
    < Date: Thu, 20 Feb 2020 11:21:22 GMT
    < Content-Type: application/json;charset=utf-8
    < Content-Length: 0
    * Server Jetty(9.4.11.v20180605) is not blacklisted
    < Server: Jetty(9.4.11.v20180605)
    <
    * Connection #0 to host 127.0.0.1 left intact

    ----------

    I expect the (above) request above to fail because I don't supply the
    required
    `cmd` parameter.

    However, if I make a request that I believe should succeed, I never
    receive a
    response:

    ----------

    # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
     > GET /ignite?cmd=version HTTP/1.1
     > User-Agent: curl/7.35.0
     > Host: 127.0.0.1:8080
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
     > Accept: */*
     >

    ----------

    `curl` will wait forever for a response that never arrives.

    I tried looking for logs output by the REST API so that I could debug the
    issue, but failed to find any log entries emitted from the REST API.

    Given that I'm new to Apache Ignite, I suspect that I'm almost certainly
    doing
    something wrong, but I have absolutely no idea what that could be.

    I have a few questions:

    1.) Has anyone encountered an issue similar to this with the REST API? 
    If so,
    how did you solve it?
    2.) Can anyone tell me where I might find REST API log entries?
    3.) I have persistence enabled in the XML configuration. Would that conflict
    with the REST API?
    4.) I have additional nodes (3 nodes in total) in the XML configuration.
    AFAICT from log entries and some liberal use of `ss`, the nodes are
    communicating with each other.  Would having multiple nodes conflict with
    the
    REST API?
    5.) Is there anything else I'm missing that might not be obvious to me
    because
    I'm new to Apache Ignite?

    Other than (3) and (4), I think the configuration file I'm using is
    rather bland:

    ----------

    <beans xmlns="http://www.springframework.org/schema/beans
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
            xsi:schemaLocation="
    http://www.springframework.org/schema/beans
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
    http://www.springframework.org/schema/beans/spring-beans.xsd
    
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
       <bean class="org.apache.ignite.configuration.IgniteConfiguration">

         <property name="dataStorageConfiguration">
           <bean
    class="org.apache.ignite.configuration.DataStorageConfiguration">
             <!--
               Default memory region that grows endlessly. A cache is bound
    to this
               memory region unless it sets another one in its
    CacheConfiguration/
             -->
             <property name="defaultDataRegionConfiguration">
               <bean
    class="org.apache.ignite.configuration.DataRegionConfiguration">
                 <property name="persistenceEnabled" value="true"/>
                 <property name="name" value="Default_Region"/>
                 <!-- 100 MB memory region with disabled eviction -->
                 <property name="initialSize" value="#{100 * 1024 * 1024}"/>
               </bean>
             </property>
           </bean>
         </property>

         <property name="discoverySpi">
           <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
             <property name="ipFinder">
               <bean
    
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                 <property name="addresses">
                   <list>
                     <value>[other node address]</value><value>[other node
    address]</value>
                   </list>
                 </property>
               </bean>
             </property>
           </bean>
         </property>

       </bean>
    </beans>

    ----------

    (Obviously, each instance of '[other node address]' is substituted with a
    real
    IP address in the actual files I'm using.)

    Thanks in advance for any help.

--Devin

2020-02-20 22:32:21
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"Attach Listener" #133 daemon prio=9 os_prio=0 tid=0x00007f7e20001000 
nid=0x608d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"rest-#44" #96 prio=5 os_prio=0 tid=0x00007f7da4005800 nid=0x6053 waiting on 
condition [0x00007f7e5049a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000007240a0fb0> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#45" #91 prio=5 os_prio=0 tid=0x00007f7d60005000 nid=0x6052 waiting on 
condition [0x00007f7e5059b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x000000072383fd70> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#46" #93 prio=5 os_prio=0 tid=0x00007f7da0004800 nid=0x6051 waiting on 
condition [0x00007f7e5069c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000007240d6998> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#43" #95 prio=5 os_prio=0 tid=0x000055fc8551b800 nid=0x6050 waiting on 
condition [0x00007f7e5079d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000007238991f0> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#48" #94 prio=5 os_prio=0 tid=0x00007f7d74006000 nid=0x604f waiting on 
condition [0x00007f7e5089e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000723cc2478> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#47" #92 prio=5 os_prio=0 tid=0x00007f7d98005000 nid=0x604e waiting on 
condition [0x00007f7e5099f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000724017400> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#49" #90 prio=5 os_prio=0 tid=0x00007f7d8800a000 nid=0x604d waiting on 
condition [0x00007f7e50aa0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000723a3e440> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"rest-#42" #89 prio=5 os_prio=0 tid=0x00007f7d78005000 nid=0x604c waiting on 
condition [0x00007f7e50ba1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9688ee8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:229)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000723bd5f38> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

"qtp1928312257-88" #88 prio=5 os_prio=0 tid=0x00007f7d7c002800 nid=0x604b 
waiting on condition [0x00007f7e50ca2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9517a88> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"Scheduler-512256603" #87 prio=5 os_prio=0 tid=0x00007f7d84007000 nid=0x604a 
waiting on condition [0x00007f7e50da3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e963e550> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"tcp-disco-srvr-#3" #85 prio=10 os_prio=0 tid=0x00007f7e811e4800 nid=0x6048 
runnable [0x00007f7e50ea4000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:560)
        at java.net.ServerSocket.accept(ServerSocket.java:528)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-disco-msg-worker-#2" #84 prio=10 os_prio=0 tid=0x00007f7e812cf800 
nid=0x6047 runnable [0x00007f7e50fa5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e9301990> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at 
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"disco-notifier-worker-#41" #83 prio=5 os_prio=0 tid=0x00007f7e8125c000 
nid=0x6046 waiting on condition [0x00007f7e510a6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8d6a500> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"Thread-22" #82 daemon prio=5 os_prio=0 tid=0x00007f7e811d4800 nid=0x6045 
waiting on condition [0x00007f7e511a7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e94f1db8> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-data-loader-flusher-#40" #81 prio=5 os_prio=0 tid=0x00007f7e81106000 
nid=0x6044 waiting on condition [0x00007f7e519e4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e94a8680> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at 
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-80" #80 prio=5 os_prio=0 tid=0x00007f7e81041800 nid=0x6043 
waiting on condition [0x00007f7e514a8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-79" #79 prio=5 os_prio=0 tid=0x00007f7e8103f800 nid=0x6042 
waiting on condition [0x00007f7e515a9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-78" #78 prio=5 os_prio=0 tid=0x00007f7e8103d800 nid=0x6041 
waiting on condition [0x00007f7e516aa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-77" #77 prio=5 os_prio=0 tid=0x00007f7e8103b800 nid=0x6040 
runnable [0x00007f7e517ab000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9594088> (a sun.nio.ch.Util$3)
        - locked <0x00000005e9594078> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e9593f60> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-76" #76 prio=5 os_prio=0 tid=0x00007f7e81039800 nid=0x603f 
runnable [0x00007f7e518ac000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9594b18> (a sun.nio.ch.Util$3)
        - locked <0x00000005e9594b08> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e95949f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-75" #75 prio=5 os_prio=0 tid=0x00007f7e81037800 nid=0x603e 
waiting on condition [0x00007f7e51ae5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-74" #74 prio=5 os_prio=0 tid=0x00007f7e81036000 nid=0x603d 
waiting on condition [0x00007f7e51be6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-73" #73 prio=5 os_prio=0 tid=0x00007f7e81034000 nid=0x603c 
waiting on condition [0x00007f7e51ce7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000723762b88> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-72" #72 prio=5 os_prio=0 tid=0x00007f7e81032800 nid=0x603b 
waiting on condition [0x00007f7e51de8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007237bf160> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-71" #71 prio=5 os_prio=0 tid=0x00007f7e81030800 nid=0x603a 
waiting on condition [0x00007f7e51ee9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-70" #70 prio=5 os_prio=0 tid=0x00007f7e8102f000 nid=0x6039 
waiting on condition [0x00007f7e51fea000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-69" #69 prio=5 os_prio=0 tid=0x00007f7e8102d000 nid=0x6038 
waiting on condition [0x00007f7e520eb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000723ff9d50> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-68" #68 prio=5 os_prio=0 tid=0x00007f7e8102b800 nid=0x6037 
waiting on condition [0x00007f7e521ec000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000723762c18> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-67" #67 prio=5 os_prio=0 tid=0x00007f7e81022000 nid=0x6036 
waiting on condition [0x00007f7e522ed000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-66" #66 prio=5 os_prio=0 tid=0x00007f7e81018000 nid=0x6035 
waiting on condition [0x00007f7e523ee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007237bf0d0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-65-acceptor-0@7eca76e7-ServerConnector@700d974e{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}"
 #65 prio=3 os_prio=0 tid=0x00007f7e8100e800 nid=0x6034 runnable 
[0x00007f7e524ef000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
        - locked <0x00000005e9592100> (a java.lang.Object)
        at 
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-64" #64 prio=5 os_prio=0 tid=0x00007f7e81004800 nid=0x6033 
waiting on condition [0x00007f7e525f0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$37/1015799131.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-63" #63 prio=5 os_prio=0 tid=0x00007f7e80ff3000 nid=0x6032 
runnable [0x00007f7e526f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e963ce70> (a sun.nio.ch.Util$3)
        - locked <0x00000005e963ce60> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e963cd48> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$37/1015799131.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-62" #62 prio=5 os_prio=0 tid=0x00007f7e80ff1800 nid=0x6031 
runnable [0x00007f7e527f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e95931f0> (a sun.nio.ch.Util$3)
        - locked <0x00000005e95931e0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e95930c8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$37/1015799131.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"qtp1928312257-61" #61 prio=5 os_prio=0 tid=0x00007f7e80ff0000 nid=0x6030 
waiting on condition [0x00007f7e528f3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
        at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
        at 
org.apache.ignite.internal.processors.rest.GridRestProcessor$1.handle(GridRestProcessor.java:143)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.processRequest(GridJettyRestHandler.java:395)
        at 
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestHandler.handle(GridJettyRestHandler.java:318)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$37/1015799131.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-3-#38" #58 prio=5 os_prio=0 tid=0x00007f7e80d0c800 
nid=0x602f runnable [0x00007f7e52df4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e94a9220> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e94ab2b0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e94a9178> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-2-#37" #57 prio=5 os_prio=0 tid=0x00007f7e80d0b000 
nid=0x602e runnable [0x00007f7e52ef5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e94ac2f8> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e951e298> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e94ac250> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-1-#36" #56 prio=5 os_prio=0 tid=0x00007f7e80c8b000 
nid=0x602d runnable [0x00007f7e52ff6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9520948> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e95229d8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e95208a0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-rest-0-#35" #55 prio=5 os_prio=0 tid=0x00007f7e80c89000 
nid=0x602c runnable [0x00007f7e530f7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9525088> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e958f070> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e9524fe0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-tcp-rest-#39" #59 prio=5 os_prio=0 tid=0x00007f7e80c87800 
nid=0x602b runnable [0x00007f7e531f8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e94abc88> (a sun.nio.ch.Util$3)
        - locked <0x00000005e94abc78> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e94ab870> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-3-#32" #52 prio=5 os_prio=0 
tid=0x00007f7e80c67000 nid=0x602a runnable [0x00007f7e532f9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9368bd0> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e936ac60> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e9368b28> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-2-#31" #51 prio=5 os_prio=0 
tid=0x00007f7e80c65000 nid=0x6029 runnable [0x00007f7e533fa000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9302648> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e9367b58> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e93025a0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-1-#30" #50 prio=5 os_prio=0 
tid=0x00007f7e80c63800 nid=0x6028 runnable [0x00007f7e534fb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e936d330> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e9426d80> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e936d288> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-0-#29" #49 prio=5 os_prio=0 
tid=0x00007f7e80c61800 nid=0x6027 runnable [0x00007f7e535fc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9429450> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e942b4e0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e94293a8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-client-listener-#33" #53 prio=5 os_prio=0 tid=0x00007f7e80c60800 
nid=0x6026 runnable [0x00007f7e536fd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e9368540> (a sun.nio.ch.Util$3)
        - locked <0x00000005e9368530> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e9368128> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"tcp-comm-worker-#1" #48 prio=5 os_prio=0 tid=0x00007f7e80991000 nid=0x6025 
waiting on condition [0x00007f7e53ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c378d0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-3-#27" #42 prio=5 os_prio=0 tid=0x00007f7e8098f000 
nid=0x6024 runnable [0x00007f7e5c1be000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e8c474b8> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e8c474d8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e8c47470> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-2-#26" #41 prio=5 os_prio=0 tid=0x00007f7e8098e000 
nid=0x6023 runnable [0x00007f7e5c7e8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e8c3fad0> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e8c3faf0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e8c3fa88> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-1-#25" #40 prio=5 os_prio=0 tid=0x00007f7e8098a000 
nid=0x6022 runnable [0x00007f7e5c6e7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e8c4f478> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e8c4f498> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e8c4f430> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-0-#24" #39 prio=5 os_prio=0 tid=0x00007f7e80987000 
nid=0x6021 runnable [0x00007f7e5c9ea000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e8c57438> (a 
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000005e8c57458> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e8c573f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-tcp-comm-#28" #47 prio=5 os_prio=0 tid=0x00007f7e80988800 
nid=0x6020 runnable [0x00007f7e5c8e9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000005e8c37bc0> (a sun.nio.ch.Util$3)
        - locked <0x00000005e8c37bd0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000005e8c37b78> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
        at 
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-timeout-worker-#23" #38 prio=5 os_prio=0 tid=0x00007f7e808e9800 
nid=0x601b in Object.wait() [0x00007f7e5caeb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000005e8cc9238> (a java.lang.Object)
        at 
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
        - locked <0x00000005e8cc9238> (a java.lang.Object)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-clock" #37 daemon prio=10 os_prio=0 tid=0x00007f7e80816000 nid=0x601a 
sleeping[0x00007f7e5cbec000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3383)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#22" #36 prio=5 os_prio=0 tid=0x00007f7e80841800 nid=0x6019 waiting on 
condition [0x00007f7e5ceed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#21" #35 prio=5 os_prio=0 tid=0x00007f7e80835000 nid=0x6018 waiting on 
condition [0x00007f7e5cfee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#20" #34 prio=5 os_prio=0 tid=0x00007f7e80833800 nid=0x6017 waiting on 
condition [0x00007f7e5d0ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#19" #33 prio=5 os_prio=0 tid=0x00007f7e80832000 nid=0x6016 waiting on 
condition [0x00007f7e5d1f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#18" #32 prio=5 os_prio=0 tid=0x00007f7e8082f800 nid=0x6015 waiting on 
condition [0x00007f7e5d2f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pub-#17" #31 prio=5 os_prio=0 tid=0x00007f7e8082e800 nid=0x6014 waiting on 
condition [0x00007f7e5d3f2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005e8c3fe80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"jvm-pause-detector-worker" #30 daemon prio=5 os_prio=0 tid=0x00007f7e807de000 
nid=0x6013 waiting on condition [0x00007f7e5d4f3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-7-#16" #29 prio=5 os_prio=0 tid=0x00007f7e8076b800 
nid=0x6012 waiting on condition [0x00007f7e5d5f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-6-#15" #28 prio=5 os_prio=0 tid=0x00007f7e80769800 
nid=0x6011 waiting on condition [0x00007f7e5d6f5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-5-#14" #27 prio=5 os_prio=0 tid=0x00007f7e80768000 
nid=0x6010 waiting on condition [0x00007f7e5d7f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-4-#13" #26 prio=5 os_prio=0 tid=0x00007f7e80766800 
nid=0x600f waiting on condition [0x00007f7e5d8f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-3-#12" #25 prio=5 os_prio=0 tid=0x00007f7e80764800 
nid=0x600e waiting on condition [0x00007f7e5d9f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-2-#11" #24 prio=5 os_prio=0 tid=0x00007f7e80763000 
nid=0x600d waiting on condition [0x00007f7e5daf9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-1-#10" #23 prio=5 os_prio=0 tid=0x00007f7e80761000 
nid=0x600c waiting on condition [0x00007f7e5dbfa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-0-#9" #22 prio=5 os_prio=0 tid=0x00007f7e80760000 
nid=0x600b waiting on condition [0x00007f7e5dcfb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-7-#8" #21 prio=5 os_prio=0 tid=0x00007f7e8075d800 nid=0x600a 
waiting on condition [0x00007f7e5ddfc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-6-#7" #20 prio=5 os_prio=0 tid=0x00007f7e8075c000 nid=0x6009 
waiting on condition [0x00007f7e5defd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-5-#6" #19 prio=5 os_prio=0 tid=0x00007f7e8075a800 nid=0x6008 
waiting on condition [0x00007f7e5dffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-4-#5" #18 prio=5 os_prio=0 tid=0x00007f7e80759000 nid=0x6007 
waiting on condition [0x00007f7e5e0ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-3-#4" #17 prio=5 os_prio=0 tid=0x00007f7e8073f000 nid=0x6006 
waiting on condition [0x00007f7e5e200000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-2-#3" #16 prio=5 os_prio=0 tid=0x00007f7e8073d800 nid=0x6005 
waiting on condition [0x00007f7e5e301000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-1-#2" #15 prio=5 os_prio=0 tid=0x00007f7e8073b800 nid=0x6004 
waiting on condition [0x00007f7e5e402000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-0-#1" #14 prio=5 os_prio=0 tid=0x00007f7e80747000 nid=0x6003 
waiting on condition [0x00007f7e5e503000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
        at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-0" #13 daemon prio=5 os_prio=0 tid=0x00007f7e80414800 
nid=0x6001 runnable [0x00007f7e5ef18000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:560)
        at java.net.ServerSocket.accept(ServerSocket.java:528)
        at 
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-49146" #12 daemon prio=5 os_prio=0 tid=0x00007f7e80400000 
nid=0x6000 runnable [0x00007f7e5f019000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:560)
        at java.net.ServerSocket.accept(ServerSocket.java:528)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-0" #11 daemon prio=5 os_prio=0 tid=0x00007f7e803ee800 
nid=0x5fff runnable [0x00007f7e5f11a000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:560)
        at java.net.ServerSocket.accept(ServerSocket.java:528)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f7e800dd800 nid=0x5ffe 
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f7e800c8800 
nid=0x5ffd waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f7e800c6800 
nid=0x5ffc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f7e800c4800 
nid=0x5ffb waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f7e800c1800 
nid=0x5ffa waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f7e800c0800 
nid=0x5ff9 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7e80091000 nid=0x5ff8 in 
Object.wait() [0x00007f7e6c36f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000005e8c27460> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00000005e8c27460> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7e8008c800 
nid=0x5ff7 in Object.wait() [0x00007f7e6c470000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000005e8c51270> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000005e8c51270> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on 
condition [0x00007f7e8875f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1131)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
        at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
        at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
        at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
        - locked <0x00000005e8cca5b8> (a 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
        at 
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
        at org.apache.ignite.Ignition.start(Ignition.java:348)
        at 
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)

   Locked ownable synchronizers:
        - <0x00000005e8c5a2a8> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a360> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a418> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a4d0> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a588> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a640> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a6f8> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000005e8c5a7b0> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

"VM Thread" os_prio=0 tid=0x00007f7e80083000 nid=0x5ff6 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f7e80022800 nid=0x5fee 
runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f7e80024000 nid=0x5fef 
runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f7e80026000 nid=0x5ff0 
runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f7e80027800 nid=0x5ff1 
runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f7e80029800 nid=0x5ff2 
runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f7e8002b000 nid=0x5ff3 
runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f7e8002d000 nid=0x5ff4 
runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f7e8002e800 nid=0x5ff5 
runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f7e80416800 nid=0x6002 waiting 
on condition 

JNI global references: 375

Re: REST API requests hang with no response

Reply via email to