Hello! Your node has never finished joining to cluster nor was able to self-discover to form a cluster of its own, as evident by:
"main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on condition [0x00007f7e8875f000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778) at org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage* (ServerImpl.java:1131) at org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology* (ServerImpl.java:910) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730) - locked <0x00000005e8cca5b8> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700) at org.apache.ignite.Ignition.start(Ignition.java:348) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) Until node is in topology, obviously it can't serve any requests. Regards, -- Ilya Kasnacheev пт, 21 февр. 2020 г. в 01:55, Devin Anderson <dande...@akamai.com>: > Hi Ilya, > > I'm attaching the `jstack` dump to this message. When I took the dump, > there > were ten requests that had been made to the Apache Ignite REST API using > POST > data of the form: > > cmd=add&key=[key]&value=[value] > > ... where [key] and [value] were proper URI encoded data. > > Thanks in advance for any help. I'll take a look as well and try to > figure out > what's going on. > > Devin > > On 2/20/20 5:43 AM, Ilya Kasnacheev wrote: > > Hello! > > > > Please collect thread dump (jstack) from affected node, share it with us. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <dande...@akamai.com > > <mailto:dande...@akamai.com>>: > > > > Hi all, > > > > I'm seeing issues wherein the Apache Ignite REST API appears to > accept > > requests, but doesn't ever reply. This doesn't always happen; for > > example, if > > I make a request that I expect the API to reject, I get back a > response: > > > > ---------- > > > > # curl -v -X GET 'http://127.0.0.1:8080/ignite > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e= > >' > > * Hostname was NOT found in DNS cache > > * Trying 127.0.0.1... > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > > > GET /ignite HTTP/1.1 > > > User-Agent: curl/7.35.0 > > > Host: 127.0.0.1:8080 > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e= > > > > > Accept: */* > > > > > < HTTP/1.1 400 Bad Request > > < Date: Thu, 20 Feb 2020 11:21:22 GMT > > < Content-Type: application/json;charset=utf-8 > > < Content-Length: 0 > > * Server Jetty(9.4.11.v20180605) is not blacklisted > > < Server: Jetty(9.4.11.v20180605) > > < > > * Connection #0 to host 127.0.0.1 left intact > > > > ---------- > > > > I expect the (above) request above to fail because I don't supply the > > required > > `cmd` parameter. > > > > However, if I make a request that I believe should succeed, I never > > receive a > > response: > > > > ---------- > > > > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e= > >' > > * Hostname was NOT found in DNS cache > > * Trying 127.0.0.1... > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > > > GET /ignite?cmd=version HTTP/1.1 > > > User-Agent: curl/7.35.0 > > > Host: 127.0.0.1:8080 > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e= > > > > > Accept: */* > > > > > > > ---------- > > > > `curl` will wait forever for a response that never arrives. > > > > I tried looking for logs output by the REST API so that I could > debug the > > issue, but failed to find any log entries emitted from the REST API. > > > > Given that I'm new to Apache Ignite, I suspect that I'm almost > certainly > > doing > > something wrong, but I have absolutely no idea what that could be. > > > > I have a few questions: > > > > 1.) Has anyone encountered an issue similar to this with the REST > API? > > If so, > > how did you solve it? > > 2.) Can anyone tell me where I might find REST API log entries? > > 3.) I have persistence enabled in the XML configuration. Would that > conflict > > with the REST API? > > 4.) I have additional nodes (3 nodes in total) in the XML > configuration. > > AFAICT from log entries and some liberal use of `ss`, the nodes are > > communicating with each other. Would having multiple nodes conflict > with > > the > > REST API? > > 5.) Is there anything else I'm missing that might not be obvious to > me > > because > > I'm new to Apache Ignite? > > > > Other than (3) and (4), I think the configuration file I'm using is > > rather bland: > > > > ---------- > > > > <beans xmlns="http://www.springframework.org/schema/beans > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e= > >" > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e= > >" > > xsi:schemaLocation=" > > http://www.springframework.org/schema/beans > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e= > > > > http://www.springframework.org/schema/beans/spring-beans.xsd > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e= > >"> > > <bean class="org.apache.ignite.configuration.IgniteConfiguration"> > > > > <property name="dataStorageConfiguration"> > > <bean > > class="org.apache.ignite.configuration.DataStorageConfiguration"> > > <!-- > > Default memory region that grows endlessly. A cache is > bound > > to this > > memory region unless it sets another one in its > > CacheConfiguration/ > > --> > > <property name="defaultDataRegionConfiguration"> > > <bean > > class="org.apache.ignite.configuration.DataRegionConfiguration"> > > <property name="persistenceEnabled" value="true"/> > > <property name="name" value="Default_Region"/> > > <!-- 100 MB memory region with disabled eviction --> > > <property name="initialSize" value="#{100 * 1024 * > 1024}"/> > > </bean> > > </property> > > </bean> > > </property> > > > > <property name="discoverySpi"> > > <bean > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> > > <property name="ipFinder"> > > <bean > > > > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> > > <property name="addresses"> > > <list> > > <value>[other node address]</value><value>[other > node > > address]</value> > > </list> > > </property> > > </bean> > > </property> > > </bean> > > </property> > > > > </bean> > > </beans> > > > > ---------- > > > > (Obviously, each instance of '[other node address]' is substituted > with a > > real > > IP address in the actual files I'm using.) > > > > Thanks in advance for any help. > > > > -- > > Devin > > > >
