Thanks for the info Denis.
Removing the failureDetectionTimeout and using the networkTimeout
seems to allow the nodes to join the topology in about the same
amount of time. I'm still only having occasional success running
anything (even just the pi estimator)
I seem to always see a bunch of warnings...a summary is dumped below
along with my config at the end, any guidance you can provide is
appreciated.
Thanks,
Joe
Every node seems to see a bunch of "Retrying preload partition",
with the lowest locNodeOrder having fewer nodes in the remaining
[14:52:38,979][WARN
][ignite-#104%sys-null%][GridDhtPartitionsExchangeFuture] Retrying
preload partition exchange due to timeout [done=false, dummy=false,
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=fd9620f5, evt=NODE_JOINED],
rcvdIds=[], rmtIds=[0ab29a08, 5216f6ba, f882885f, 0d232f1a,
b74f5ebb, 5790761a, 55d2082e, b1bf93b3, 2fd79f9f, a899ccce,
3dd74aba, 320d05fd, 0d44a4b3, 9a00f235, 4426467e, 7837fdfc,
e8778da0, 4a988e3e, f8cabdbb, 494ad6fd, 7c05abfb, 5902c851,
c406028e, a0b57685, e213b903, c85a0b46, df981c08, 187cd54f,
f0b7b298, 94ec7576, 041975f5, aecba5d0, 5549256d, f9b5a77a,
596d0df7, 26266d8c, 0e664e25, 97d112b2, aac08043, 6b81a2b1,
5a2a1012, 534ac94b, b34cb942, 837785eb, 966d70b2, 3aab732e,
4e34ad89, 6df0ffff, 4c7c3c47, 85eea5fe, 1c5e2f6b, 3f426f4e,
27a9bef9, cd874e96, dc3256a7, 4da50521, 1d370c9e, 19c334eb,
24be15dd, 6c922af3, 01ea2812], remaining=[0ab29a08, 5216f6ba,
f882885f, 0d232f1a, b74f5ebb, 5790761a, 55d2082e, b1bf93b3,
2fd79f9f, a899ccce, 3dd74aba, 320d05fd, 0d44a4b3, 9a00f235,
4426467e, 7837fdfc, e8778da0, 4a988e3e, f8cabdbb, 494ad6fd,
7c05abfb, 5902c851, c406028e, a0b57685, e213b903, c85a0b46,
df981c08, 187cd54f, f0b7b298, 94ec7576, 041975f5, aecba5d0,
5549256d, f9b5a77a, 596d0df7, 26266d8c, 0e664e25, 97d112b2,
aac08043, 6b81a2b1, 5a2a1012, 534ac94b, b34cb942, 837785eb,
966d70b2, 3aab732e, 4e34ad89, 6df0ffff, 4c7c3c47, 85eea5fe,
1c5e2f6b, 3f426f4e, 27a9bef9, cd874e96, dc3256a7, 4da50521,
1d370c9e, 19c334eb, 24be15dd, 6c922af3, 01ea2812], init=true,
initFut=true, ready=true, replied=false, added=true,
oldest=0d44a4b3, oldestOrder=1, evtLatch=0, locNodeOrder=62,
locNodeId=fd9620f5-3ebb-4a71-a482-73d6a81b1688]
[14:38:41,893][WARN
][ignite-#95%sys-null%][GridDhtPartitionsExchangeFuture] Retrying
preload partition exchange due to timeout [done=false, dummy=false,
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=25, minorTopVer=0], nodeId=df981c08, evt=NODE_JOINED],
rcvdIds=[7c05abfb, b34cb942, e213b903, 320d05fd, 5902c851, f0b7b298,
1d370c9e, 0d232f1a, 494ad6fd, 5a2a1012, b1bf93b3, 55d2082e,
7837fdfc, 85eea5fe, 4e34ad89, 5790761a, 3f426f4e, aac08043,
187cd54f, 01ea2812, c406028e, 24be15dd, 966d70b2], rmtIds=[0d232f1a,
5790761a, 55d2082e, b1bf93b3, aac08043, 5a2a1012, b34cb942,
320d05fd, 966d70b2, 4e34ad89, 85eea5fe, 7837fdfc, 3f426f4e,
1d370c9e, 494ad6fd, 7c05abfb, 5902c851, c406028e, 24be15dd,
e213b903, df981c08, 187cd54f, f0b7b298, 01ea2812],
remaining=[df981c08], init=true, initFut=true, ready=true,
replied=false, added=true, oldest=0d44a4b3, oldestOrder=1,
evtLatch=0, locNodeOrder=1,
locNodeId=0d44a4b3-4d10-4f67-b8bd-005be226b1df]
I also see a little over half the nodes getting "Still waiting for
initial partition map exchange" warnings like this
[14:39:37,848][WARN ][main][GridCachePartitionExchangeManager] Still
waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=27a9bef9-de04-486d-aac0-bfa749e9007d,
addrs=[0:0:0:0:0:0:0:1%1, 10.148.0.87, 10.159.1.182, 127.0.0.1],
sockAddrs=[r1i4n10.redacted.com/10.148.0.87:47500,
/0:0:0:0:0:0:0:1%1:47500, /10.159.1.182:47500, /10.148.0.87:47500,
/10.159.1.182:47500, /127.0.0.1:47500], discPort=47500, order=48,
intOrder=48, lastExchangeTime=1445974777828, loc=true,
ver=1.4.0#19691231-sha1:00000000, isClient=false], topVer=48,
nodeId8=27a9bef9, msg=null, type=NODE_JOINED, tstamp=1445974647187],
rcvdIds=GridConcurrentHashSet [elements=[]],
rmtIds=[0ab29a08-9c95-4054-8035-225f5828b3d4,
0d232f1a-0f46-4798-a39a-63a17dc4dc7f,
f9b5a77a-a4c1-46aa-872e-aeaca9b76ee3,
596d0df7-3edf-4078-8f4a-ffa3d96296c6,
5790761a-aeeb-44d1-9fce-3fee31ef39b7,
55d2082e-3517-4828-8d47-57b4ed5a41bc,
26266d8c-cc87-4472-9fa4-c526d6da2233,
0e664e25-8dde-4df8-966b-53b60f9a1087,
b1bf93b3-24bb-4520-ade0-31d05a93558d,
aac08043-875a-485a-ab2c-cd7e66d68f8f,
2fd79f9f-9590-41d2-962e-004a3d7690b5,
5a2a1012-0766-448c-9583-25873c305de9,
534ac94b-8dd1-4fa8-a481-539fa4f4ce55,
b34cb942-e960-4a00-b4fb-10add6466a93,
320d05fd-e021-40ac-83bc-62f54756771b,
0d44a4b3-4d10-4f67-b8bd-005be226b1df,
837785eb-24e0-496a-a0cc-f795b64b5929,
9a00f235-3b6a-4be5-b0e3-93cd1beacaf4,
966d70b2-e1dc-4e20-9876-b63736545abd,
3aab732e-a075-4b19-9525-e97a1260a4fe,
4e34ad89-fa46-4503-a599-b8c937ca1f47,
4c7c3c47-6e5c-4c15-80a9-408192596bc2,
85eea5fe-9aff-4821-970c-4ce006ee853a,
7837fdfc-6255-4784-8088-09d4e6e37bb9,
3f426f4e-2d0c-402a-a4af-9d7656f46484,
e8778da0-a764-4ad9-afba-8a748564e12a,
4a988e3e-3434-4271-acd6-af2a1e30524c,
cd874e96-63cf-41c9-8e8a-75f3223bfe9d,
f8cabdbb-875a-480b-8b5e-4b5313c5fcbd,
dc3256a7-ae23-4c2e-b375-55e2884e045d,
4da50521-aad0-48a4-9f79-858bbc2e6b89,
1d370c9e-250f-4733-8b8a-7b6f5c6e1b2b,
494ad6fd-1637-44b8-8d3a-1fa19681ba64,
7c05abfb-dba1-43c3-a8b1-af504762ec60,
5902c851-5275-41fd-89c4-cd6390c88670,
19c334eb-5661-4697-879d-1082571dfef8,
c406028e-768e-404e-8417-40d2960c4ba3,
a0b57685-e5dc-498c-99a4-33b1aef32632,
24be15dd-45f7-4980-b4f8-3176ab67e8f6,
e213b903-107b-4465-8fe1-78b7b393d631,
df981c08-148d-4266-9ea7-163168012968,
187cd54f-396b-4c3c-9bfc-9883ac37f556,
f0b7b298-6432-477a-85a0-83e29e8c5380,
94ec7576-7a02-4c08-8739-4e0fc52a3d3a,
041975f5-990a-4792-b384-eded32966783,
01ea2812-5005-4152-af2e-2586bf65b4c6,
aecba5d0-9d9b-4ab6-9018-62f5abb7b809],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=48, minorTopVer=0], nodeId=27a9bef9, evt=NODE_JOINED],
init=true, ready=true, replied=false, added=true,
initFut=GridFutureAdapter [resFlag=2, res=true,
startTime=1445974657836, endTime=1445974658400,
ignoreInterrupts=false, lsnr=null, state=DONE], topSnapshot=null,
lastVer=null, partReleaseFut=GridCompoundFuture [lsnrCalls=3,
finished=true, rdc=null, init=true,
res=java.util.concurrent.atomic.AtomicMarkableReference@6b58be0e,
err=null, done=true, cancelled=false, err=null, futs=[true, true,
true]], skipPreload=false, clientOnlyExchange=false,
oldest=0d44a4b3-4d10-4f67-b8bd-005be226b1df, oldestOrder=1,
evtLatch=0, remaining=[0ab29a08-9c95-4054-8035-225f5828b3d4,
0d232f1a-0f46-4798-a39a-63a17dc4dc7f,
f9b5a77a-a4c1-46aa-872e-aeaca9b76ee3,
596d0df7-3edf-4078-8f4a-ffa3d96296c6,
5790761a-aeeb-44d1-9fce-3fee31ef39b7,
55d2082e-3517-4828-8d47-57b4ed5a41bc,
26266d8c-cc87-4472-9fa4-c526d6da2233,
0e664e25-8dde-4df8-966b-53b60f9a1087,
b1bf93b3-24bb-4520-ade0-31d05a93558d,
aac08043-875a-485a-ab2c-cd7e66d68f8f,
2fd79f9f-9590-41d2-962e-004a3d7690b5,
5a2a1012-0766-448c-9583-25873c305de9,
534ac94b-8dd1-4fa8-a481-539fa4f4ce55,
b34cb942-e960-4a00-b4fb-10add6466a93,
320d05fd-e021-40ac-83bc-62f54756771b,
0d44a4b3-4d10-4f67-b8bd-005be226b1df,
837785eb-24e0-496a-a0cc-f795b64b5929,
9a00f235-3b6a-4be5-b0e3-93cd1beacaf4,
966d70b2-e1dc-4e20-9876-b63736545abd,
3aab732e-a075-4b19-9525-e97a1260a4fe,
4e34ad89-fa46-4503-a599-b8c937ca1f47,
4c7c3c47-6e5c-4c15-80a9-408192596bc2,
85eea5fe-9aff-4821-970c-4ce006ee853a,
7837fdfc-6255-4784-8088-09d4e6e37bb9,
3f426f4e-2d0c-402a-a4af-9d7656f46484,
e8778da0-a764-4ad9-afba-8a748564e12a,
4a988e3e-3434-4271-acd6-af2a1e30524c,
cd874e96-63cf-41c9-8e8a-75f3223bfe9d,
f8cabdbb-875a-480b-8b5e-4b5313c5fcbd,
dc3256a7-ae23-4c2e-b375-55e2884e045d,
4da50521-aad0-48a4-9f79-858bbc2e6b89,
1d370c9e-250f-4733-8b8a-7b6f5c6e1b2b,
494ad6fd-1637-44b8-8d3a-1fa19681ba64,
7c05abfb-dba1-43c3-a8b1-af504762ec60,
5902c851-5275-41fd-89c4-cd6390c88670,
19c334eb-5661-4697-879d-1082571dfef8,
c406028e-768e-404e-8417-40d2960c4ba3,
a0b57685-e5dc-498c-99a4-33b1aef32632,
24be15dd-45f7-4980-b4f8-3176ab67e8f6,
e213b903-107b-4465-8fe1-78b7b393d631,
df981c08-148d-4266-9ea7-163168012968,
187cd54f-396b-4c3c-9bfc-9883ac37f556,
f0b7b298-6432-477a-85a0-83e29e8c5380,
94ec7576-7a02-4c08-8739-4e0fc52a3d3a,
041975f5-990a-4792-b384-eded32966783,
01ea2812-5005-4152-af2e-2586bf65b4c6,
aecba5d0-9d9b-4ab6-9018-62f5abb7b809], super=GridFutureAdapter
[resFlag=0, res=null, startTime=1445974657836, endTime=0,
ignoreInterrupts=false, lsnr=null, state=INIT]]]
Then on the occasions when mapreduce jobs fail I will see one node
with (it isn't always the same node)
[14:52:57,080][WARN
][grid-nio-worker-3-#135%null%][TcpCommunicationSpi] Closing NIO
session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection timed out]
[14:52:59,123][WARN
][grid-nio-worker-3-#135%null%][TcpCommunicationSpi] Failed to
process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=3, queueSize=0,
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
recovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=0,
reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=837785eb-24e0-496a-a0cc-f795b64b5929, addrs=[0:0:0:0:0:0:0:1%1,
10.148.0.81, 10.159.1.176, 127.0.0.1],
sockAddrs=[/10.159.1.176:47500, /0:0:0:0:0:0:0:1%1:47500,
r1i4n4.redacted.com/10.148.0.81:47500, /10.148.0.81:47500,
/10.159.1.176:47500, /127.0.0.1:47500], discPort=47500, order=45,
intOrder=45, lastExchangeTime=1445974625750, loc=false,
ver=1.4.0#19691231-sha1:00000000, isClient=false], connected=true,
connectCnt=1, queueLimit=5120], super=GridNioSessionImpl
[locAddr=/10.159.1.112:46222, rmtAddr=/10.159.1.176:47100,
createTime=1445974646591, closeTime=0, bytesSent=30217, bytesRcvd=9,
sndSchedTime=1445975577912, lastSndTime=1445975577912,
lastRcvTime=1445974655114, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@44de55ba,
directMode=true], GridConnectionBytesVerifyFilter], accepted=false]]
[14:52:59,124][WARN
][grid-nio-worker-3-#135%null%][TcpCommunicationSpi] Closing NIO
session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection timed out]
[14:53:00,105][WARN
][grid-nio-worker-3-#135%null%][TcpCommunicationSpi] Failed to
process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=3, queueSize=0,
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
recovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=0,
reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=4426467e-b4b4-4912-baa1-d7cc839d9188, addrs=[0:0:0:0:0:0:0:1%1,
10.148.0.106, 10.159.1.201, 127.0.0.1],
sockAddrs=[/10.159.1.201:47500, /0:0:0:0:0:0:0:1%1:47500,
r1i5n11.redacted.com/10.148.0.106:47500, /10.148.0.106:47500,
/10.159.1.201:47500, /127.0.0.1:47500], discPort=47500, order=57,
intOrder=57, lastExchangeTime=1445974625790, loc=false,
ver=1.4.0#19691231-sha1:00000000, isClient=false], connected=true,
connectCnt=1, queueLimit=5120], super=GridNioSessionImpl
[locAddr=/10.159.1.112:60869, rmtAddr=/10.159.1.201:47100,
createTime=1445974654478, closeTime=0, bytesSent=22979, bytesRcvd=0,
sndSchedTime=1445975577912, lastSndTime=1445975577912,
lastRcvTime=1445974654478, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@44de55ba,
directMode=true], GridConnectionBytesVerifyFilter], accepted=false]]
[14:53:00,105][WARN
][grid-nio-worker-3-#135%null%][TcpCommunicationSpi] Closing NIO
session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection timed out]
I've tried adjusting the timeout settings further but haven't had
much success.
Here is what my config looks like, it is obviously heavily based off
the hadoop example config.
<?xml version="1.0" encoding="UTF-8"?>
<beans
ns1:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd";
xmlns="http://www.springframework.org/schema/beans";
xmlns:ns1="http://www.w3.org/2001/XMLSchema-instance";>
<description>
Spring file for Ignite node configuration with IGFS and
Apache Hadoop map-reduce support enabled.
Ignite node will start with this configuration by default.
</description>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
id="propertyConfigurer">
<property name="systemPropertiesModeName"
value="SYSTEM_PROPERTIES_MODE_FALLBACK" />
<property name="searchSystemEnvironment" value="true" />
</bean>
<bean abstract="true"
class="org.apache.ignite.configuration.FileSystemConfiguration"
id="igfsCfgBase">
<property name="blockSize" value="#{128 * 1024}" />
<property name="perNodeBatchSize" value="512" />
<property name="perNodeParallelBatchCount" value="16" />
<property name="prefetchBlocks" value="32" />
</bean>
<bean abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration"
id="dataCacheCfgBase">
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="writeSynchronizationMode" value="FULL_SYNC" />
<property name="backups" value="0" />
<property name="affinityMapper">
<bean class="org.apache.ignite.igfs.IgfsGroupDataBlocksKeyMapper">
<constructor-arg value="512" />
</bean>
</property>
<property name="startSize" value="#{100*1024*1024}" />
<property name="offHeapMaxMemory" value="0" />
</bean>
<bean abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration"
id="metaCacheCfgBase">
<property name="cacheMode" value="REPLICATED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="writeSynchronizationMode" value="FULL_SYNC" />
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration"
id="grid.cfg">
<property name="failureDetectionTimeout" value="3000" />
<property name="hadoopConfiguration">
<bean class="org.apache.ignite.configuration.HadoopConfiguration">
<property name="finishedJobInfoTtl" value="30000" />
</bean>
</property>
<property name="connectorConfiguration">
<bean
class="org.apache.ignite.configuration.ConnectorConfiguration">
<property name="port" value="11211" />
</bean>
</property>
<property name="fileSystemConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.FileSystemConfiguration"
parent="igfsCfgBase">
<property name="name" value="igfs" />
<property name="metaCacheName" value="igfs-meta" />
<property name="dataCacheName" value="igfs-data" />
<property name="ipcEndpointConfiguration">
<bean
class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
<property name="type" value="TCP" />
<property name="host" value="r1i0n12" />
<property name="port" value="10500" />
</bean>
</property>
</bean>
</list>
</property>
<property name="cacheConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.CacheConfiguration"
parent="metaCacheCfgBase">
<property name="name" value="igfs-meta" />
</bean>
<bean
class="org.apache.ignite.configuration.CacheConfiguration"
parent="dataCacheCfgBase">
<property name="name" value="igfs-data" />
</bean>
</list>
</property>
<property name="includeEventTypes">
<list>
<ns2:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"
xmlns:ns2="http://www.springframework.org/schema/util"; />
<ns2:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"
xmlns:ns2="http://www.springframework.org/schema/util"; />
<ns2:constant
static-field="org.apache.ignite.events.EventType.EVT_JOB_MAPPED"
xmlns:ns2="http://www.springframework.org/schema/util"; />
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>r1i0n12:47500</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Quoting Denis Magda <dma...@gridgain.com>:
Hi Joe,
Great!
Please see below
On 10/27/2015 9:37 AM, d...@eiler.net wrote:
Reducing the port range (to a single port) and lowering the
IgniteConfiguration.setFailureDetectionTimeout to 1000 helped
speed up everybody joining the topology and I was able to get a pi
estimator run on 64 nodes.
I suspect that the reason was in the number of ports specified in
the range. By some reason it takes significant time to get a
response from TCP/IP stack that a connection can't be established
on a particular port number.
Please try to reduce the port range, lower
TcpDiscoverySpi.setNetworkTimeout, keep
IgniteConfiguration.setFailureDetectionTimeout's default value and
share results with us.
Thanks again for the help, I'm over the current hurdle.
Joe
Quoting d...@eiler.net:
Thanks for the quick response Denis.
I did a port range of 10 ports. I'll take a look at the
failureDetectionTimeout and networkTimeout.
Side question: Is there an easy way to map between the
programmatic API and the spring XML properties? For instance I
was trying to find the correct xml incantation for
TcpDiscoverySpi.setMaxMissedHeartbeats(int) and I might have a
similar issue finding
IgniteConfiguration.setFailureDetectionTimeout(long). It seems
like I can usually drop the set and adjust capitalization
(setFooBar() == <property name="fooBar")
Yes, your understanding is correct.
Please pardon my ignorance on terminology:
Are the nodes I run ignite.sh on considered server nodes or
cluster nodes (I would have thought they are the same)
Actually we have a notion of server and client nodes. This page
contains extensive information on the type of nodes:
https://apacheignite.readme.io/docs/clients-vs-servers
A cluster node is just a server or client node.
Regards,
Denis
Thanks,
Joe
Quoting Denis Magda <dma...@gridgain.com>:
Hi Joe,
How big is a port range, that you specified in your discovery
configuration, for a every single node?
Please take into account that the discovery may iterate over
every port from the range before one node connects to the other
and depending on the TCP related settings of your network it may
take significant time before the cluster is assembled.
Here I would recommend you to reduce the port range as much as
possible and to play with the following network related parameters:
- Try to use the failure detection timeout instead of setting
socket, ack and many other timeouts explicitly
(https://apacheignite.readme.io/docs/cluster-config#failure-detection-timeout);
- Try to play with TcpDiscoverySpi.networkTimeout because this
timeout is considered during the time when a cluster node tries
to join a cluster.
In order to help you with the hanging compute tasks and to give
you more specific recommendations regarding the slow join
process please provide us with the following:
- config files for server and cluster nodes;
- log files from all the nodes. Please start the nodes with
-DIGNITE_QUIET=false virtual machine property. If you start the
nodes using ignite.sh/bat then just pass '-v' as an argument to
the script.
- thread dumps for the nodes that are hanging waiting for the
compute tasks to be completed.
Regards,
Denis
On 10/26/2015 6:56 AM, d...@eiler.net wrote:
Hi all,
I have been experimenting with ignite and have run into a
problem scaling up to larger clusters.
I am playing with only two different use cases, 1) a Hadoop
MapReduce accelerator 2) an in memory data grid (no secondary
file system) being accessed by frameworks using the HDFS
Everything works fine with a smaller cluster (8 nodes) but with
a larger cluster (64 nodes) it takes a couple of minutes for
all the nodes to register with the cluster(which would be ok)
and mapreduce jobs just hang and never return.
I've compiled the latest Ignite 1.4 (with
ignite.edition=hadoop) from source, and am using it with Hadoop
2.7.1 just trying to run things like the pi estimator and
wordcount examples.
I started with the config/hadoop/default-config.xml
I can't use multicast so I've configured it to use static IP
based discovery with just a single node/port range.
I've increased the heartbeat frequency to 10000 and that seemed
to help make things more stable once all the nodes do join the
cluster. I've also played with increasing both the socket
timeout and the ack timeout but that seemed to just make it
take longer for nodes to attempt to join the cluster after a
failed attempt.
I have access to a couple of different clusters, we allocate
resources with slurm so I get a piece of a cluster to play with
(hence the no-multicast restriction). The nodes all have fast
networks (FDR InfiniBand) and a decent amount of memory
(64GB-128GB) but no local storage (or swap space).
As mentioned earlier, I disable the secondaryFilesystem.
Any advice/hints/example xml configs would be extremely welcome.
I also haven't been seeing the expected performance using the
hdfs api to access ignite. I've tried both using the hdfs cli
to do some simple timings of put/get and a little java program
that writes then reads a file. Even with small files (500MB)
that should be kept completely in a single node, I only see
about 250MB/s for writes and reads are much slower than that
(4x to 10x). The writes are better than hdfs (our hdfs is
backed with pretty poor storage) but reads are much slower. Now
I haven't tried scaling this at all but with an 8 node ignite
cluster and a single "client" access a single file I would hope
for something closer to memory speeds. (if you would like me to
split this into another message to the list just let me know,
I'm assuming the cause it the same---I missed a required config
setting ;-) )
Thanks in advance for any help,
Joe
