Re: Re:Re: Ignite for Spark on YARN Deployment

Hi, Regarding the ports, I'm copy-pasting the response provided by Nikolai on dev list: Ignite by default using the following ports:11211 - REST, 47100..47200 - TCP communication, 47500...47600 - Discovery, 49112 - JMX. You can change the values (see TcpCommunicationSpi#setLocalPort and

Re: Re:Re: Ignite for Spark on YARN Deployment

Can you pleas tell me what ports should be opened on driver or yarn cluster machine? this is our production env, I suspect that something need be configured on firewall ports. 1) I verified all these steps on my local cluster(3 spark machine[1 master and 2 workers], one yarn cluster machine),

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, YES, I can ping yarn cluster from driver machine. I try to telnet localhost 50075 on driver machine, but failed, is this root cause for this issues? thanks!!! -- View this message in context:

Re: Re:Re: Ignite for Spark on YARN Deployment

Hi, Could you make sure that you have access from "driver" machine to machines on which running YARN? Could you ping them? On Fri, Aug 12, 2016 at 11:26 AM, percent620 wrote: > scala> [14:52:04] New version is available at ignite.apache.org: 1.7.0 > 16/08/12 14:52:46 ERROR

Re: Re:Re: Ignite for Spark on YARN Deployment

scala> [14:52:04] New version is available at ignite.apache.org: 1.7.0 16/08/12 14:52:46 ERROR TcpDiscoverySpi: Failed to reconnect to cluster (consider increasing 'networkTimeout' configuration property) [networkTimeout=5000] 16/08/12 15:37:54 ERROR GridClockSyncProcessor: Failed to send time

Re: Re:Re: Ignite for Spark on YARN Deployment

Get the following error message after a few minutes scala> val ic = new IgniteContext[Integer, Integer](sc,"/u01/yueyi/apache-ignite-fabric-1.6.0-bin/config/default-config.xml") ic: org.apache.ignite.spark.IgniteContext[Integer,Integer] = org.apache.ignite.spark.IgniteContext@26917c50 scala>

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, I'm a new ignite, can you please provide more detail steps for this? thanks again!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6986.html Sent from the Apache Ignite Users mailing list archive at

Re: Re:Re: Ignite for Spark on YARN Deployment

> > 3) I'm running spark-shell on driver machine NOT yarn cluster? > ignite-spark starts Ignite client node which should have direct access to YARN ignite cluster (have network access, open ports and etc).

Re: Re:Re: Ignite for Spark on YARN Deployment

1) containers logs as below 1、[20:07:14] Ignite node started OK (id=a3a87e37) [20:07:14] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=2.0GB] [20:07:14] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=4.0GB] [20:07:15] Topology snapshot [ver=3, servers=3, clients=0,

Re: Re:Re: Ignite for Spark on YARN Deployment

Could you show logs from containers? spark-shell just hangs or print some logs? Also are you sure that you have direct access to machine on which running yarn cluster? On Thu, Aug 11, 2016 at 3:43 PM, percent620 wrote: > Hello, Nikolai, > 1、 > Just updated configuration

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, 1、 Just updated configuration default-config.xm on /u01/XXX/apache-ignite-fabric-1.6.0-bin/config/default-config.xm 2）./hdfs dfs -put /u01/XXX/apache-ignite-fabric-1.6.0-bin/config/default-config.xm /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/ 3) scala> import

Re: Re:Re: Ignite for Spark on YARN Deployment

Hi, You can use hdfs path as hdfs://your_host:9000/ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml. Or you can just copy this config file on local disk. ;) You should use the same configuration, but it can be different files with the same content. On Thu, Aug 11, 2016

Re: Re:Re: Ignite for Spark on YARN Deployment

Thanks Nikolai very much. As your request, and changed configuration /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml but this file is hfs file 1) $./hdfs dfs -text /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml SLF4J: Class path contains

Re: Re:Re: Ignite for Spark on YARN Deployment

Great! Ignite YARN cluster successfully started, but ignite spark shell doesn't see server nodes. You need to configure IP finder, by default YARN cluster using VmIpFinder, but ignite-spark MulticastIpFinder. Could you change configuration for YARN cluster (IGNITE_XML_CONFIG=/ignite/

Re: Re:Re: Ignite for Spark on YARN Deployment

1、Adjusted cluster.properites as below $cat cluster16.properties # The number of nodes in the cluster. IGNITE_NODE_COUNT=4 # The number of CPU Cores for each Apache Ignite node. IGNITE_RUN_CPU_PER_NODE=1 # The number of Megabytes of RAM for each Apache Ignite node. IGNITE_MEMORY_PER_NODE=2048 #

Re:Re: Ignite for Spark on YARN Deployment

Thanks very much for your quick response.!!! I will attache contain logs for this issues. Can you please also help me on this issues as below? http://apache-ignite-users.70518.x6.nabble.com/Embedded-mode-ignite-on-spark-tt6942.html At 2016-08-11 17:43:40, "Nikolai Tikhonov-2 [via Apache