the spark history server and the yarn history server are totally independent. Spark knows nothing about yarn logs, and vice versa, so unfortunately there isn't any way to get all the info in one place.
On Tue, Feb 24, 2015 at 12:36 PM, Colin Kincaid Williams <disc...@uw.edu> wrote: > Looks like in my tired state, I didn't mention spark the whole time. > However, it might be implied by the application log above. Spark log > aggregation appears to be working, since I can run the yarn command above. > I do have yarn logging setup for the yarn history server. I was trying to > use the spark history-server, but maybe I should try setting > > spark.yarn.historyServer.address > > to the yarn history-server, instead of the spark history-server? I tried > this configuration when I started, but didn't have much luck. > > Are you getting your spark apps run in yarn client or cluster mode in your > yarn history server? If so can you share any spark settings? > > On Tue, Feb 24, 2015 at 8:48 AM, Christophe Préaud < > christophe.pre...@kelkoo.com> wrote: > >> Hi Colin, >> >> Here is how I have configured my hadoop cluster to have yarn logs >> available through both the yarn CLI and the _yarn_ history server (with >> gzip compression and 10 days retention): >> >> 1. Add the following properties in the yarn-site.xml on each node >> managers and on the resource manager: >> <property> >> <name>yarn.log-aggregation-enable</name> >> <value>true</value> >> </property> >> <property> >> <name>yarn.log-aggregation.retain-seconds</name> >> <value>864000</value> >> </property> >> <property> >> <name>yarn.log.server.url</name> >> <value> >> http://dc1-kdp-dev-hadoop-03.dev.dc1.kelkoo.net:19888/jobhistory/logs >> </value> >> </property> >> <property> >> <name>yarn.nodemanager.log-aggregation.compression-type</name> >> <value>gz</value> >> </property> >> >> 2. Restart yarn and then start the yarn history server on the server >> defined in the yarn.log.server.url property above: >> >> /opt/hadoop/sbin/mr-jobhistory-daemon.sh stop historyserver # should fail >> if historyserver is not yet started >> /opt/hadoop/sbin/stop-yarn.sh >> /opt/hadoop/sbin/start-yarn.sh >> /opt/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver >> >> >> It may be slightly different for you if the resource manager and the >> history server are not on the same machine. >> >> Hope it will work for you as well! >> Christophe. >> >> On 24/02/2015 06:31, Colin Kincaid Williams wrote: >> > Hi, >> > >> > I have been trying to get my yarn logs to display in the spark >> history-server or yarn history-server. I can see the log information >> > >> > >> > yarn logs -applicationId application_1424740955620_0009 >> > 15/02/23 22:15:14 INFO client.ConfiguredRMFailoverProxyProvider: >> Failing over to us3sm2hbqa04r07-comp-prod-local >> > >> > >> > Container: container_1424740955620_0009_01_000002 on >> us3sm2hbqa07r07.comp.prod.local_8041 >> > >> =========================================================================================== >> > LogType: stderr >> > LogLength: 0 >> > Log Contents: >> > >> > LogType: stdout >> > LogLength: 897 >> > Log Contents: >> > [GC [PSYoungGen: 262656K->23808K(306176K)] 262656K->23880K(1005568K), >> 0.0283450 secs] [Times: user=0.14 sys=0.03, real=0.03 secs] >> > Heap >> > PSYoungGen total 306176K, used 111279K [0x00000000eaa80000, >> 0x0000000100000000, 0x0000000100000000) >> > eden space 262656K, 33% used >> [0x00000000eaa80000,0x00000000effebbe0,0x00000000fab00000) >> > from space 43520K, 54% used >> [0x00000000fab00000,0x00000000fc240320,0x00000000fd580000) >> > to space 43520K, 0% used >> [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) >> > ParOldGen total 699392K, used 72K [0x00000000bff80000, >> 0x00000000eaa80000, 0x00000000eaa80000) >> > object space 699392K, 0% used >> [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) >> > PSPermGen total 35328K, used 34892K [0x00000000bad80000, >> 0x00000000bd000000, 0x00000000bff80000) >> > object space 35328K, 98% used >> [0x00000000bad80000,0x00000000bcf93088,0x00000000bd000000) >> > >> > >> > >> > Container: container_1424740955620_0009_01_000003 on >> us3sm2hbqa09r09.comp.prod.local_8041 >> > >> =========================================================================================== >> > LogType: stderr >> > LogLength: 0 >> > Log Contents: >> > >> > LogType: stdout >> > LogLength: 896 >> > Log Contents: >> > [GC [PSYoungGen: 262656K->23725K(306176K)] 262656K->23797K(1005568K), >> 0.0358650 secs] [Times: user=0.28 sys=0.04, real=0.04 secs] >> > Heap >> > PSYoungGen total 306176K, used 65712K [0x00000000eaa80000, >> 0x0000000100000000, 0x0000000100000000) >> > eden space 262656K, 15% used >> [0x00000000eaa80000,0x00000000ed380bf8,0x00000000fab00000) >> > from space 43520K, 54% used >> [0x00000000fab00000,0x00000000fc22b4f8,0x00000000fd580000) >> > to space 43520K, 0% used >> [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) >> > ParOldGen total 699392K, used 72K [0x00000000bff80000, >> 0x00000000eaa80000, 0x00000000eaa80000) >> > object space 699392K, 0% used >> [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) >> > PSPermGen total 29696K, used 29486K [0x00000000bad80000, >> 0x00000000bca80000, 0x00000000bff80000) >> > object space 29696K, 99% used >> [0x00000000bad80000,0x00000000bca4b838,0x00000000bca80000) >> > >> > >> > >> > Container: container_1424740955620_0009_01_000001 on >> us3sm2hbqa09r09.comp.prod.local_8041 >> > >> =========================================================================================== >> > LogType: stderr >> > LogLength: 0 >> > Log Contents: >> > >> > LogType: stdout >> > LogLength: 21 >> > Log Contents: >> > Pi is roughly 3.1416 >> > >> > I can see some details for the application in the spark history-server >> at this url >> http://us3sm2hbqa04r07.comp.prod.local:18080/history/application_1424740955620_0009/jobs/ >> . When running in spark-master mode, I can see the stdout and stderror >> somewhere in the spark history-server. Then how do I get the information >> which I see above into the Spark history-server ? >> >> >> Kelkoo SAS >> Société par Actions Simplifiée >> Au capital de € 4.168.964,30 >> Siège social : 158 Ter Rue du Temple 75003 Paris >> 425 093 069 RCS Paris >> >> Ce message et les pièces jointes sont confidentiels et établis à >> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le >> destinataire de ce message, merci de le détruire et d'en avertir >> l'expéditeur. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
