Hello!
I'll start in the reverse order:
3. If everything is OK, the process should take around a second (for
non-persistent cluster).
2. As soon you see "Topology snapshot" in the log, the process is complete.
1. From your configuration I see suspicious that it only lists two other
nodes and not itself. It is recommended that any node can discover itself
and start a cluster as 1st node. Try adding <value>[own ip address]</value>
Regards,
--
Ilya Kasnacheev
пт, 21 февр. 2020 г. в 14:52, Devin Anderson <dande...@akamai.com
<mailto:dande...@akamai.com>>:
Hi Ilya,
That certainly makes sense, but I'm not totally sure how to act on that
information yet. A couple questions:
1.) Given the configuration file I posted (which is basically the same on
each
node in the cluster, save that the IP addresses of the nodes to discover
are
different), should each of the nodes eventually join the cluster? Am I
missing
anything?
2.) Is there a message that I should see in the logs that identifies that
the
node has successfully joined the cluster?
3.) How long should I expect the process to take with three nodes (just a
loose
estimate)?
Thanks for your ongoing help. It's much appreciated. :)
Devin
On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
> Hello!
>
> Your node has never finished joining to cluster nor was able to
self-discover
> to form a cluster of its own, as evident by:
>
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
> condition [0x00007f7e8875f000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
> at
>
org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
> at
>
org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
> at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
> at
>
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
> at
>
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> at
>
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
> at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
> at
>
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
>
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> - locked <0x00000005e8cca5b8> (a
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at
>
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> at org.apache.ignite.Ignition.start(Ignition.java:348)
> at
>
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
>
> Until node is in topology, obviously it can't serve any requests.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 21 февр. 2020 г. в 01:55, Devin Anderson <dande...@akamai.com
<mailto:dande...@akamai.com>
> <mailto:dande...@akamai.com <mailto:dande...@akamai.com>>>:
>
> Hi Ilya,
>
> I'm attaching the `jstack` dump to this message. When I took the
dump,
> there
> were ten requests that had been made to the Apache Ignite REST API
using
> POST
> data of the form:
>
> cmd=add&key=[key]&value=[value]
>
> ... where [key] and [value] were proper URI encoded data.
>
> Thanks in advance for any help. I'll take a look as well and try to
> figure out
> what's going on.
>
> Devin
>
> On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> > Hello!
> >
> > Please collect thread dump (jstack) from affected node, share it
with us.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <dande...@akamai.com
<mailto:dande...@akamai.com>
> <mailto:dande...@akamai.com <mailto:dande...@akamai.com>>
> > <mailto:dande...@akamai.com <mailto:dande...@akamai.com>
<mailto:dande...@akamai.com <mailto:dande...@akamai.com>>>>:
> >
> > Hi all,
> >
> > I'm seeing issues wherein the Apache Ignite REST API appears
to accept
> > requests, but doesn't ever reply. This doesn't always
happen; for
> > example, if
> > I make a request that I expect the API to reject, I get back a
> response:
> >
> > ----------
> >
> > # curl -v -X GET 'http://127.0.0.1:8080/ignite
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=bT604T6RDd_DwnVyBCW-NHSDUMXvUZq9XsvGzHDbueI&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > Accept: */*
> > >
> > < HTTP/1.1 400 Bad Request
> > < Date: Thu, 20 Feb 2020 11:21:22 GMT
> > < Content-Type: application/json;charset=utf-8
> > < Content-Length: 0
> > * Server Jetty(9.4.11.v20180605) is not blacklisted
> > < Server: Jetty(9.4.11.v20180605)
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> >
> > ----------
> >
> > I expect the (above) request above to fail because I don't
supply the
> > required
> > `cmd` parameter.
> >
> > However, if I make a request that I believe should succeed, I
never
> > receive a
> > response:
> >
> > ----------
> >
> > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=ZJdkLEsuKL4bxIqpkGT9pyoxIYEgK40dRUHMJHGytTs&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite?cmd=version HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > Accept: */*
> > >
> >
> > ----------
> >
> > `curl` will wait forever for a response that never arrives.
> >
> > I tried looking for logs output by the REST API so that I could
> debug the
> > issue, but failed to find any log entries emitted from the
REST API.
> >
> > Given that I'm new to Apache Ignite, I suspect that I'm almost
> certainly
> > doing
> > something wrong, but I have absolutely no idea what that
could be.
> >
> > I have a few questions:
> >
> > 1.) Has anyone encountered an issue similar to this with the
REST API?
> > If so,
> > how did you solve it?
> > 2.) Can anyone tell me where I might find REST API log entries?
> > 3.) I have persistence enabled in the XML configuration.
Would that
> conflict
> > with the REST API?
> > 4.) I have additional nodes (3 nodes in total) in the XML
> configuration.
> > AFAICT from log entries and some liberal use of `ss`, the
nodes are
> > communicating with each other. Would having multiple nodes
> conflict with
> > the
> > REST API?
> > 5.) Is there anything else I'm missing that might not be
obvious to me
> > because
> > I'm new to Apache Ignite?
> >
> > Other than (3) and (4), I think the configuration file I'm
using is
> > rather bland:
> >
> > ----------
> >
> > <beans xmlns="http://www.springframework.org/schema/beans
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=7PzeJ7X6RCdABzpGj5jdaon_Eks7xHDjJoBXZ7gyu7o&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
> > xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
> > http://www.springframework.org/schema/beans/spring-beans.xsd
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=DVCBX8SLQnDmVGJ5Zs9Wb3q4TH7tGwWBK3mxXRiXmOQ&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
> > <bean
class="org.apache.ignite.configuration.IgniteConfiguration">
> >
> > <property name="dataStorageConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > <!--
> > Default memory region that grows endlessly. A
cache is bound
> > to this
> > memory region unless it sets another one in its
> > CacheConfiguration/
> > -->
> > <property name="defaultDataRegionConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > <property name="persistenceEnabled" value="true"/>
> > <property name="name" value="Default_Region"/>
> > <!-- 100 MB memory region with disabled
eviction -->
> > <property name="initialSize" value="#{100 * 1024 *
> 1024}"/>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > <property name="discoverySpi">
> > <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > <property name="ipFinder">
> > <bean
> >
>
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > <property name="addresses">
> > <list>
> > <value>[other node
address]</value><value>[other node
> > address]</value>
> > </list>
> > </property>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > </bean>
> > </beans>
> >
> > ----------
> >
> > (Obviously, each instance of '[other node address]' is
substituted
> with a
> > real
> > IP address in the actual files I'm using.)
> >
> > Thanks in advance for any help.
> >
> > --
> > Devin
> >
>
