Hi, I have been trying to get my yarn logs to display in the spark history-server or yarn history-server. I can see the log information
yarn logs -applicationId application_1424740955620_0009 15/02/23 22:15:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to us3sm2hbqa04r07-comp-prod-local Container: container_1424740955620_0009_01_000002 on us3sm2hbqa07r07.comp.prod.local_8041 =========================================================================================== LogType: stderr LogLength: 0 Log Contents: LogType: stdout LogLength: 897 Log Contents: [GC [PSYoungGen: 262656K->23808K(306176K)] 262656K->23880K(1005568K), 0.0283450 secs] [Times: user=0.14 sys=0.03, real=0.03 secs] Heap PSYoungGen total 306176K, used 111279K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000) eden space 262656K, 33% used [0x00000000eaa80000,0x00000000effebbe0,0x00000000fab00000) from space 43520K, 54% used [0x00000000fab00000,0x00000000fc240320,0x00000000fd580000) to space 43520K, 0% used [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) ParOldGen total 699392K, used 72K [0x00000000bff80000, 0x00000000eaa80000, 0x00000000eaa80000) object space 699392K, 0% used [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) PSPermGen total 35328K, used 34892K [0x00000000bad80000, 0x00000000bd000000, 0x00000000bff80000) object space 35328K, 98% used [0x00000000bad80000,0x00000000bcf93088,0x00000000bd000000) Container: container_1424740955620_0009_01_000003 on us3sm2hbqa09r09.comp.prod.local_8041 =========================================================================================== LogType: stderr LogLength: 0 Log Contents: LogType: stdout LogLength: 896 Log Contents: [GC [PSYoungGen: 262656K->23725K(306176K)] 262656K->23797K(1005568K), 0.0358650 secs] [Times: user=0.28 sys=0.04, real=0.04 secs] Heap PSYoungGen total 306176K, used 65712K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000) eden space 262656K, 15% used [0x00000000eaa80000,0x00000000ed380bf8,0x00000000fab00000) from space 43520K, 54% used [0x00000000fab00000,0x00000000fc22b4f8,0x00000000fd580000) to space 43520K, 0% used [0x00000000fd580000,0x00000000fd580000,0x0000000100000000) ParOldGen total 699392K, used 72K [0x00000000bff80000, 0x00000000eaa80000, 0x00000000eaa80000) object space 699392K, 0% used [0x00000000bff80000,0x00000000bff92010,0x00000000eaa80000) PSPermGen total 29696K, used 29486K [0x00000000bad80000, 0x00000000bca80000, 0x00000000bff80000) object space 29696K, 99% used [0x00000000bad80000,0x00000000bca4b838,0x00000000bca80000) Container: container_1424740955620_0009_01_000001 on us3sm2hbqa09r09.comp.prod.local_8041 =========================================================================================== LogType: stderr LogLength: 0 Log Contents: LogType: stdout LogLength: 21 Log Contents: Pi is roughly 3.1416 I can see some details for the application in the spark history-server at this url http://us3sm2hbqa04r07.comp.prod.local:18080/history/application_1424740955620_0009/jobs/ . When running in spark-master mode, I can see the stdout and stderror somewhere in the spark history-server. Then how do I get the information which I see above into the Spark history-server ?
