Hi Nikolay,
Thank you very much for your solution. I think the ignite started successfully with your updated version of configuration file!!! :) I think so far ignite running ok on YARN, right? Next I will try to Test ignite from Spark-shell. Should I follow the link below: https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell In Starting up the cluster, step 1-4, Since our spark clusters are alway running on YARN, and I do not think I need to start spark, right? In Starting up the cluster, step 5 , do I need to start Ignite node using the following command? Since I have started a ignite YARN application and currently running. On each of the worker nodes cd to $IGNITE_HOME and start an Ignite node by running the following command:bin/ignite.sh In Working with Spark shell section, I run the following command and Spark shell launched successfully. /usr/bin/spark-shell --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar Then I ran these three command: The Warning message as below: I think I need to put the ip addresses of ignite server node into the ignite-default-config.xml, am I right? import org.apache.ignite.spark._ import org.apache.ignite.configuration._ val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration()) 16/06/10 09:18:13 WARN TcpDiscoverySpi: IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network. Will retry every 2 secs. Thank you very much for all your help!!!! Hongmei Starting up the cluster Here is the log from my console: Logs for container_e24_1464374946035_32224_01_000001 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/disk/09/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32224/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/06/10 08:43:48 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 16/06/10 08:43:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run INFO: Application master registered. Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run INFO: Making request. Memory: 2,432, cpu 1. Jun 10, 2016 8:43:48 AM org.apache.ignite.yarn.ApplicationMaster run INFO: Making request. Memory: 2,432, cpu 1. 16/06/10 08:43:49 INFO impl.AMRMClientImpl: Received new token for : c5hdp105.c5.runwaynine.com:45454 16/06/10 08:43:49 INFO impl.AMRMClientImpl: Received new token for : c5hdp112.c5.runwaynine.com:45454 Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster run INFO: Making request. Memory: 2,432, cpu 1. Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster run INFO: Making request. Memory: 2,432, cpu 1. Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated INFO: Launching container: container_e24_1464374946035_32224_01_000002. 16/06/10 08:43:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp105.c5.runwaynine.com:45454 Jun 10, 2016 8:43:49 AM org.apache.ignite.yarn.ApplicationMaster onContainersAllocated INFO: Launching container: container_e24_1464374946035_32224_01_000003. 16/06/10 08:43:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : c5hdp112.c5.runwaynine.com:45454 16/06/10 08:43:50 INFO impl.AMRMClientImpl: Received new token for : c5hdp114.c5.runwaynine.com:45454 16/06/10 08:43:50 INFO impl.AMRMClientImpl: Received new token for : c5hdp115.c5.runwaynine.com:45454 Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000006. State: COMPLETE. Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000007. State: COMPLETE. Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000004. State: COMPLETE. Jun 10, 2016 8:43:50 AM org.apache.ignite.yarn.ApplicationMaster onContainersCompleted INFO: Container completed. Container id: container_e24_1464374946035_32224_01_000005. State: COMPLETE. Logs for container_e24_1464374946035_32224_01_000002 [08:43:59] __________ ________________ [08:43:59] / _/ ___/ |/ / _/_ __/ __/ [08:43:59] _/ // (7 7 // / / / / _/ [08:43:59] /___/\___/_/|_/___/ /_/ /___/ [08:43:59] [08:43:59] ver. 1.5.22#20160519-sha1:f242f351 [08:43:59] 2016 Copyright(C) Apache Software Foundation [08:43:59] [08:43:59] Ignite documentation: http://ignite.apache.org [08:43:59] [08:43:59] Quiet mode. [08:43:59] ^-- Logging to file '/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32224/container_e24_1464374946035_32224_01_000002/ignite/gridgain-professional-fabric-1.5.22/work/log/ignite-8bf39a16.0.log' [08:43:59] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [08:43:59] [08:43:59] OS: Linux 2.6.32-573.18.1.el6.x86_64 amd64 [08:43:59] VM information: Java(TM) SE Runtime Environment 1.7.0_45-b18 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.45-b08 [08:44:00] Configured plugins: [08:44:00] ^-- None [08:44:00] [08:44:00] Security status [authentication=off, tls/ssl=off] [08:44:01] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [08:44:01] [08:44:01] Ignite node started OK (id=8bf39a16) [08:44:01] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=2.0GB] [08:44:03] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=4.0GB] > On Jun 10, 2016, at 8:20 AM, Nikolay Tikhonov <[email protected]> wrote: > > Could you download a configuration file > (https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959 > <https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959>) and put > it to > /user/hongmei/ignite/config/ignite-default-config.xml? Also change > IGNITE_XML_CONFIG from /user/hongmei/ignite/config/ to > /user/hongmei/ignite/config/ignite-default-config.xml? > > IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ignite-default-config.xml > > On Fri, Jun 10, 2016 at 2:58 PM, Hongmei Zong <[email protected] > <mailto:[email protected]>> wrote: > Hi Nikolai, > > Here is the log of one of the container: > > Logs for container_e24_1464374946035_32114_01_000014 > > > Showing 4096 bytes. Click here > <http://c5hdp100.c5.runwaynine.com:8042/node/containerlogs/container_e24_1464374946035_32114_01_000014/hongmei/stderr/?start=0> > for full log > SpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391) > at > org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104) > at > org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98) > at > org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556) > at org.apache.ignite.Ignition.start(Ignition.java:347) > ... 1 more > Caused by: > org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 > in XML document from URL > [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] > is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: > 1; columnNumber: 1; Content is not allowed in prolog. > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398) > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335) > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303) > at > org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379) > ... 9 more > Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; > Content is not allowed in prolog. > at > com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198) > at > com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177) > at > com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441) > at > com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368) > at > com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) > at > com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) > at > com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) > at > com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) > at > org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76) > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428) > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390) > ... 12 more > Failed to start grid: Failed to instantiate Spring XML application context > [springUrl=file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/, > err=Line 1 in XML document from URL > [file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_32114/container_e24_1464374946035_32114_01_000014/./ignite-config.xml/] > is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: > 1; columnNumber: 1; Content is not allowed in prolog.] > > > Thank you very much! > Hongmei > >> On Jun 10, 2016, at 5:56 AM, Nikolai Tikhonov <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Hongmei Zong! >> >> Could you show logs from other containers >> (container_e24_1464374946035_29722_01_000015) which was completed? >> >> On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Nikolay, >> >> After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ >> (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to >> check the log of the application, the attached is the stderr log information >> about containers: >> >> It looks like that the containers are allocated and then completed! The >> stderr log is very long and the container ID from XXXXX01_0000001 to >> XXXXXX01_013582. Finally all these containers are completed. >> >> I have no idea, is there anything not right? >> >> There is no information in stdout log. >> >> Thank you! >> >> Hongmei >> Logs for container_e24_1464374946035_29722_01_000001 >> >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/disk2/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_29722/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings >> <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> 16/06/09 10:48:16 INFO impl.ContainerManagementProtocolProxy: >> yarn.client.max-cached-nodemanagers-proxies : 0 >> 16/06/09 10:48:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing >> over to rm2 >> Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Application master registered. >> Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:16 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> 16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp108.c5.runwaynine.com:45454 <http://c5hdp108.c5.runwaynine.com:45454/> >> 16/06/09 10:48:17 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp111.c5.runwaynine.com:45454 <http://c5hdp111.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000002. >> 16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp108.c5.runwaynine.com:45454 <http://c5hdp108.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:17 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000003. >> 16/06/09 10:48:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp111.c5.runwaynine.com:45454 <http://c5hdp111.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000002. State: COMPLETE. >> Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000003. State: COMPLETE. >> Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:24 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> 16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp109.c5.runwaynine.com:45454 <http://c5hdp109.c5.runwaynine.com:45454/> >> 16/06/09 10:48:25 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp105.c5.runwaynine.com:45454 <http://c5hdp105.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000004. >> 16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp108.c5.runwaynine.com:45454 <http://c5hdp108.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000005. >> 16/06/09 10:48:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp111.c5.runwaynine.com:45454 <http://c5hdp111.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000006. State: COMPLETE. >> Jun 09, 2016 10:48:25 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000007. State: COMPLETE. >> Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000004. State: COMPLETE. >> Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000005. State: COMPLETE. >> Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:27 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000008. >> 16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp108.c5.runwaynine.com:45454 <http://c5hdp108.c5.runwaynine.com:45454/> >> 16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp112.c5.runwaynine.com:45454 <http://c5hdp112.c5.runwaynine.com:45454/> >> 16/06/09 10:48:28 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp114.c5.runwaynine.com:45454 <http://c5hdp114.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000009. >> 16/06/09 10:48:28 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp111.c5.runwaynine.com:45454 <http://c5hdp111.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000012. State: COMPLETE. >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000010. State: COMPLETE. >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000011. State: COMPLETE. >> Jun 09, 2016 10:48:28 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000013. State: COMPLETE. >> Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000008. State: COMPLETE. >> Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000009. State: COMPLETE. >> Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:30 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000014. >> 16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp108.c5.runwaynine.com:45454 <http://c5hdp108.c5.runwaynine.com:45454/> >> 16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp107.c5.runwaynine.com:45454 <http://c5hdp107.c5.runwaynine.com:45454/> >> 16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp110.c5.runwaynine.com:45454 <http://c5hdp110.c5.runwaynine.com:45454/> >> 16/06/09 10:48:31 INFO impl.AMRMClientImpl: Received new token for : >> c5hdp101.c5.runwaynine.com:45454 <http://c5hdp101.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> INFO: Launching container: container_e24_1464374946035_29722_01_000015. >> 16/06/09 10:48:31 INFO impl.ContainerManagementProtocolProxy: Opening proxy >> : c5hdp111.c5.runwaynine.com:45454 <http://c5hdp111.c5.runwaynine.com:45454/> >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000020. State: COMPLETE. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000019. State: COMPLETE. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000018. State: COMPLETE. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000017. State: COMPLETE. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000016. State: COMPLETE. >> Jun 09, 2016 10:48:31 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000021. State: COMPLETE. >> Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000014. State: COMPLETE. >> Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersCompleted >> INFO: Container completed. Container id: >> container_e24_1464374946035_29722_01_000015. State: COMPLETE. >> Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:33 AM org.apache.ignite.yarn.ApplicationMaster run >> INFO: Making request. Memory: 2,432, cpu 1. >> Jun 09, 2016 10:48:34 AM org.apache.ignite.yarn.ApplicationMaster >> onContainersAllocated >> >> On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov <[email protected] >> <mailto:[email protected]>> wrote: >> You set wrong value to IGNITE_XML_CONFIG property. The property should >> contains path to ignite configuration file. For example >> >> IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml >> >> I think you can comment this line in property file and ignite will start >> with default configuration. >> >> On Thu, Jun 9, 2016 at 5:08 PM, Hongmei Zong <[email protected] >> <mailto:[email protected]>> wrote: >> Hi nikolai, >> >> Thank you very much for prompt reply! >> >> I did not find the ignite-config.xml file under my ignite home directory( >> /u/hongmei/apache-ignite/ ). >> >> I find a "default-config.xml" at the path: >> /u/hongmei/apache-ignite/config/default-config.xml >> >> <?xml version="1.0" encoding="UTF-8"?> >> >> <!-- >> Licensed to the Apache Software Foundation (ASF) under one or more >> contributor license agreements. See the NOTICE file distributed with >> this work for additional information regarding copyright ownership. >> The ASF licenses this file to You under the Apache License, Version 2.0 >> (the "License"); you may not use this file except in compliance with >> the License. You may obtain a copy of the License at >> >> http://www.apache.org/licenses/LICENSE-2.0 >> <http://www.apache.org/licenses/LICENSE-2.0> >> >> Unless required by applicable law or agreed to in writing, software >> distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> See the License for the specific language governing permissions and >> limitations under the License. >> --> >> >> <beans xmlns="http://www.springframework.org/schema/beans >> <http://www.springframework.org/schema/beans>" >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance >> <http://www.w3.org/2001/XMLSchema-instance>" >> xsi:schemaLocation=" >> http://www.springframework.org/schema/beans >> <http://www.springframework.org/schema/beans> >> http://www.springframework.org/schema/beans/spring-beans.xsd >> <http://www.springframework.org/schema/beans/spring-beans.xsd>"> >> <!-- >> Alter configuration below as needed. >> --> >> <bean id="grid.cfg" >> class="org.apache.ignite.configuration.IgniteConfiguration"> >> <property name="discoverySpi"> >> <bean >> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> >> <property name="ipFinder"> >> <bean >> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> >> <property name="addresses"> >> <list> >> <!-- >> Explicitly >> specifying address of a local node to let it start >> and operate >> normally even if there is no more nodes in the cluster. >> You can also >> optionally specify an individual port or port range. >> --> >> <!-- >> >> <value>1.2.3.4</value> >> --> >> <!-- >> IP >> Address and optional port range of a remote node. >> You >> can also optionally specify an individual port and don't set >> the >> port range at all. >> --> >> >> <value>c5hdpe001.c5.runwaynine.com >> <http://c5hdpe001.c5.runwaynine.com/>:47500..47509</value> >> >> <value>c5hdpe002.c5.runwaynine.com >> <http://c5hdpe002.c5.runwaynine.com/>:47500..47509</value> >> >> <value>c5hdpe003.c5.runwaynine.com >> <http://c5hdpe003.c5.runwaynine.com/>:47500..47509</value> >> >> </list> >> </property> >> </bean> >> </property> >> </bean> >> </property> >> </bean> >> </beans> >> >> >> The "cluster.properties" at the path: >> /u/hongmei/apache-ignite/config/cluster.properties >> >> # The number of nodes in the cluster. >> IGNITE_NODE_COUNT=2 >> >> # The number of CPU Cores for each Apache Ignite node. >> IGNITE_RUN_CPU_PER_NODE=1 >> >> # The number of Megabytes of RAM for each Apache Ignite node. >> IGNITE_MEMORY_PER_NODE=2048 >> >> # The version ignite which will be run on nodes. >> IGNITE_VERSION=1.6.0 >> >> # The hdfs directory which will be used for saving Apache Ignite >> disbributives. >> IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn >> >> #The directory which will be used for saving Apache Ignite >> distributives(copy .jar file to it). >> IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir >> >> #The hdfs path to Apache Ignite config file. >> #IGNITE_XML_CONFIG=/user/yarn/ignite/ >> IGNITE_XML_CONFIG=/user/hongmei/ignite/ >> >> #The hdfs path to libs which will be added to classpath. >> IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/ >> >> #The constraint on slave hosts. >> #IGNITE_HOSTENAME_CONSTRAINT= >> >> IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite >> >> >> Thank you very much!!!! >> Hongmei >> >> >> >> >> On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <[email protected] >> <mailto:[email protected]>> wrote: >> It seems that your ignite configuration invalid. Could you share >> ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties? >> >> On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <[email protected] >> <mailto:[email protected]>> wrote: >> Thank you very much Nikolai ! >> >> I found another issue regarding my Ignite YARN Integration: >> >> I run Ignite YARN application on one client server machine 'c5hdpe001', >> screenshot as the following: >> Then I log onto one of the container node and the log is as the following: >> >> [hongmei@c5hdpe001 apache-ignite]$ hadoop jar >> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar >> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar >> /u/hongmei/apache-ignite/config/cluster.properties >> WARNING: Use "yarn jar" to launch YARN applications. >> 16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address: >> http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/ >> <http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/> >> 16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing >> over to rm2 >> 16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application >> application_1464374946035_29511 >> Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main >> INFO: Submitted application. Application id: application_1464374946035_29511 >> Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main >> INFO: Application application_1464374946035_29511 is RUNNING. >> >> >> Then I log onto one of the container node and the log is as the following: >> "It failed to start grid". >> Any good suggestion? Thank you very much!!! >> >> >> class org.apache.ignite.IgniteException: Failed to instantiate Spring XML >> application context >> [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, >> err=Line 1 in XML document from URL >> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] >> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: >> 1; columnNumber: 1; Content is not allowed in prolog.] >> at >> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904) >> at org.apache.ignite.Ignition.start(Ignition.java:350) >> at >> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302) >> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to >> instantiate Spring XML application context >> [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, >> err=Line 1 in XML document from URL >> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] >> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: >> 1; columnNumber: 1; Content is not allowed in prolog.] >> at >> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391) >> at >> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104) >> at >> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98) >> at >> org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606) >> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807) >> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716) >> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586) >> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556) >> at org.apache.ignite.Ignition.start(Ignition.java:347) >> ... 1 more >> Caused by: >> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line >> 1 in XML document from URL >> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] >> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: >> 1; columnNumber: 1; Content is not allowed in prolog. >> at >> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398) >> at >> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335) >> at >> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303) >> at >> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379) >> ... 9 more >> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; >> Content is not allowed in prolog. >> at >> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198) >> at >> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177) >> at >> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441) >> at >> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368) >> at >> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436) >> at >> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999) >> at >> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) >> at >> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) >> at >> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) >> at >> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) >> at >> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) >> at >> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) >> at >> org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76) >> at >> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428) >> at >> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390) >> ... 12 more >> Failed to start grid: Failed to instantiate Spring XML application context >> [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/, >> err=Line 1 in XML document from URL >> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/] >> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: >> 1; columnNumber: 1; Content is not allowed in prolog.] >> >> >> Hongmei >> >> >> On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <[email protected] >> <mailto:[email protected]>> wrote: >> your_address1:47500..47510,your_address2:47500..47510 and >> your_address3:47500..47510 are the YARN master_host address, right? >> >> No, this addresses hosts on which deploy YARN cluster. For example, you have >> YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case >> you will have the following configuration: >> >> ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510", >> "10.0.0.2:47500..47510")); >> >> >> >> >> >> >> > >
