You can see this information in the yarn web UI using the configuration I provided in my former mail (click on the application id, then on logs; you will then be automatically redirected to the yarn history server UI).
On 24/02/2015 19:49, Colin Kincaid Williams wrote: So back to my original question. I can see the spark logs using the example above: yarn logs -applicationId application_1424740955620_0009 This shows yarn log aggregation working. I can see the std out and std error in that container information above. Then how can I get this information in a web-ui ? Is this not currently supported? On Tue, Feb 24, 2015 at 10:44 AM, Imran Rashid <iras...@cloudera.com<mailto:iras...@cloudera.com>> wrote: the spark history server and the yarn history server are totally independent. Spark knows nothing about yarn logs, and vice versa, so unfortunately there isn't any way to get all the info in one place. On Tue, Feb 24, 2015 at 12:36 PM, Colin Kincaid Williams <disc...@uw.edu<mailto:disc...@uw.edu>> wrote: Looks like in my tired state, I didn't mention spark the whole time. However, it might be implied by the application log above. Spark log aggregation appears to be working, since I can run the yarn command above. I do have yarn logging setup for the yarn history server. I was trying to use the spark history-server, but maybe I should try setting spark.yarn.historyServer.address to the yarn history-server, instead of the spark history-server? I tried this configuration when I started, but didn't have much luck. Are you getting your spark apps run in yarn client or cluster mode in your yarn history server? If so can you share any spark settings? On Tue, Feb 24, 2015 at 8:48 AM, Christophe Préaud <christophe.pre...@kelkoo.com<mailto:christophe.pre...@kelkoo.com>> wrote: Hi Colin, Here is how I have configured my hadoop cluster to have yarn logs available through both the yarn CLI and the _yarn_ history server (with gzip compression and 10 days retention): 1. Add the following properties in the yarn-site.xml on each node managers and on the resource manager: <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>864000</value> </property> <property> <name>yarn.log.server.url</name> <value>http://dc1-kdp-dev-hadoop-03.dev.dc1.kelkoo.net:19888/jobhistory/logs</value> </property> <property> <name>yarn.nodemanager.log-aggregation.compression-type</name> <value>gz</value> </property> 2. Restart yarn and then start the yarn history server on the server defined in the yarn.log.server.url property above: /opt/hadoop/sbin/mr-jobhistory-daemon.sh stop historyserver # should fail if historyserver is not yet started /opt/hadoop/sbin/stop-yarn.sh /opt/hadoop/sbin/start-yarn.sh /opt/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver It may be slightly different for you if the resource manager and the history server are not on the same machine. Hope it will work for you as well! Christophe. On 24/02/2015 06:31, Colin Kincaid Williams wrote: > Hi, > > I have been trying to get my yarn logs to display in the spark history-server > or yarn history-server. I can see the log information > > > yarn logs -applicationId application_1424740955620_0009 > 15/02/23 22:15:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing over > to us3sm2hbqa04r07-comp-prod-local > > > Container: container_1424740955620_0009_01_000002 on > us3sm2hbqa07r07.comp.prod.local_8041 > =========================================================================================== > LogType: stderr > LogLength: 0 > Log Contents: > > LogType: stdout > LogLength: 897 > Log Contents: > [GC [PSYoungGen: 262656K->23808K(306176K)] 262656K->23880K(1005568K), > 0.0283450 secs] [Times: user=0.14 sys=0.03, real=0.03 secs] > Heap > PSYoungGen total 306176K, used 111279K [0x00000000eaa80000, > 0x0000000100000000, 0x0000000100000000) > eden space 262656K, 33% used > [0x00000000eaa80000,0x00000000effebbe0,0x00000000fab00000) > from space 43520K, 54% used > [0x00000000fab00000,0x00000000fc240320,0x00000000fd580000) > to space 43520K, 0% used > [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) > ParOldGen total 699392K, used 72K [0x00000000bff80000, > 0x00000000eaa80000, 0x00000000eaa80000) > object space 699392K, 0% used > [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) > PSPermGen total 35328K, used 34892K [0x00000000bad80000, > 0x00000000bd000000, 0x00000000bff80000) > object space 35328K, 98% used > [0x00000000bad80000,0x00000000bcf93088,0x00000000bd000000) > > > > Container: container_1424740955620_0009_01_000003 on > us3sm2hbqa09r09.comp.prod.local_8041 > =========================================================================================== > LogType: stderr > LogLength: 0 > Log Contents: > > LogType: stdout > LogLength: 896 > Log Contents: > [GC [PSYoungGen: 262656K->23725K(306176K)] 262656K->23797K(1005568K), > 0.0358650 secs] [Times: user=0.28 sys=0.04, real=0.04 secs] > Heap > PSYoungGen total 306176K, used 65712K [0x00000000eaa80000, > 0x0000000100000000, 0x0000000100000000) > eden space 262656K, 15% used > [0x00000000eaa80000,0x00000000ed380bf8,0x00000000fab00000) > from space 43520K, 54% used > [0x00000000fab00000,0x00000000fc22b4f8,0x00000000fd580000) > to space 43520K, 0% used > [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) > ParOldGen total 699392K, used 72K [0x00000000bff80000, > 0x00000000eaa80000, 0x00000000eaa80000) > object space 699392K, 0% used > [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) > PSPermGen total 29696K, used 29486K [0x00000000bad80000, > 0x00000000bca80000, 0x00000000bff80000) > object space 29696K, 99% used > [0x00000000bad80000,0x00000000bca4b838,0x00000000bca80000) > > > > Container: container_1424740955620_0009_01_000001 on > us3sm2hbqa09r09.comp.prod.local_8041 > =========================================================================================== > LogType: stderr > LogLength: 0 > Log Contents: > > LogType: stdout > LogLength: 21 > Log Contents: > Pi is roughly 3.1416 > > I can see some details for the application in the spark history-server at > this url > http://us3sm2hbqa04r07.comp.prod.local:18080/history/application_1424740955620_0009/jobs/ > . When running in spark-master mode, I can see the stdout and stderror > somewhere in the spark history-server. Then how do I get the information > which I see above into the Spark history-server ? Kelkoo SAS Société par Actions Simplifiée Au capital de € 4.168.964,30 Siège social : 158 Ter Rue du Temple 75003 Paris 425 093 069 RCS Paris Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> ________________________________ Kelkoo SAS Société par Actions Simplifiée Au capital de € 4.168.964,30 Siège social : 158 Ter Rue du Temple 75003 Paris 425 093 069 RCS Paris Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
