Hi Ilya,
That certainly makes sense, but I'm not totally sure how to act on that
information yet. A couple questions:
1.) Given the configuration file I posted (which is basically the same on each
node in the cluster, save that the IP addresses of the nodes to discover are
different), should each of the nodes eventually join the cluster? Am I missing
anything?
2.) Is there a message that I should see in the logs that identifies that the
node has successfully joined the cluster?
3.) How long should I expect the process to take with three nodes (just a loose
estimate)?
Thanks for your ongoing help. It's much appreciated. :)
Devin
On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
Hello!
Your node has never finished joining to cluster nor was able to self-discover
to form a cluster of its own, as evident by:
"main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
condition [0x00007f7e8875f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
at
org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
at
org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
- locked <0x00000005e8cca5b8> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Until node is in topology, obviously it can't serve any requests.
Regards,
--
Ilya Kasnacheev
пт, 21 февр. 2020 г. в 01:55, Devin Anderson <dande...@akamai.com
<mailto:dande...@akamai.com>>:
Hi Ilya,
I'm attaching the `jstack` dump to this message. When I took the dump,
there
were ten requests that had been made to the Apache Ignite REST API using
POST
data of the form:
cmd=add&key=[key]&value=[value]
... where [key] and [value] were proper URI encoded data.
Thanks in advance for any help. I'll take a look as well and try to
figure out
what's going on.
Devin
On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> Hello!
>
> Please collect thread dump (jstack) from affected node, share it with us.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 20 февр. 2020 г. в 16:17, Devin Anderson <dande...@akamai.com
<mailto:dande...@akamai.com>
> <mailto:dande...@akamai.com <mailto:dande...@akamai.com>>>:
>
> Hi all,
>
> I'm seeing issues wherein the Apache Ignite REST API appears to accept
> requests, but doesn't ever reply. This doesn't always happen; for
> example, if
> I make a request that I expect the API to reject, I get back a
response:
>
> ----------
>
> # curl -v -X GET 'http://127.0.0.1:8080/ignite
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > Accept: */*
> >
> < HTTP/1.1 400 Bad Request
> < Date: Thu, 20 Feb 2020 11:21:22 GMT
> < Content-Type: application/json;charset=utf-8
> < Content-Length: 0
> * Server Jetty(9.4.11.v20180605) is not blacklisted
> < Server: Jetty(9.4.11.v20180605)
> <
> * Connection #0 to host 127.0.0.1 left intact
>
> ----------
>
> I expect the (above) request above to fail because I don't supply the
> required
> `cmd` parameter.
>
> However, if I make a request that I believe should succeed, I never
> receive a
> response:
>
> ----------
>
> # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite?cmd=version HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > Accept: */*
> >
>
> ----------
>
> `curl` will wait forever for a response that never arrives.
>
> I tried looking for logs output by the REST API so that I could
debug the
> issue, but failed to find any log entries emitted from the REST API.
>
> Given that I'm new to Apache Ignite, I suspect that I'm almost
certainly
> doing
> something wrong, but I have absolutely no idea what that could be.
>
> I have a few questions:
>
> 1.) Has anyone encountered an issue similar to this with the REST API?
> If so,
> how did you solve it?
> 2.) Can anyone tell me where I might find REST API log entries?
> 3.) I have persistence enabled in the XML configuration. Would that
conflict
> with the REST API?
> 4.) I have additional nodes (3 nodes in total) in the XML
configuration.
> AFAICT from log entries and some liberal use of `ss`, the nodes are
> communicating with each other. Would having multiple nodes
conflict with
> the
> REST API?
> 5.) Is there anything else I'm missing that might not be obvious to me
> because
> I'm new to Apache Ignite?
>
> Other than (3) and (4), I think the configuration file I'm using is
> rather bland:
>
> ----------
>
> <beans xmlns="http://www.springframework.org/schema/beans
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
> http://www.springframework.org/schema/beans/spring-beans.xsd
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <!--
> Default memory region that grows endlessly. A cache is
bound
> to this
> memory region unless it sets another one in its
> CacheConfiguration/
> -->
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled" value="true"/>
> <property name="name" value="Default_Region"/>
> <!-- 100 MB memory region with disabled eviction -->
> <property name="initialSize" value="#{100 * 1024 *
1024}"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="discoverySpi">
> <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
>
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>[other node address]</value><value>[other node
> address]</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
> </bean>
> </beans>
>
> ----------
>
> (Obviously, each instance of '[other node address]' is substituted
with a
> real
> IP address in the actual files I'm using.)
>
> Thanks in advance for any help.
>
> --
> Devin
>
