I am running spark-1.6.1 and the stand alone cluster manager. I am running into performance problems with spark streaming and added some extra metrics to my log files. I submit my app in cluster mode. (I.e. The driver runs on a slave not master)
I am not able to get the driver log files while the app is running using the documented rest api <http://spark.apache.org/docs/latest/monitoring.html#rest-api> http://spark.apache.org/docs/latest/monitoring.html#rest-api I think the issue is the rest-api give you access to the app log files. I need the driver log file? $ curl http://$host/api/v1/applications/ [ { "id" : "app-20160711185337-0049", "name" : "gnip1", "attempts" : [ { "startTime" : "2016-07-11T18:53:35.318GMT", "endTime" : "1969-12-31T23:59:59.999GMT", "sparkUser" : "", "completed" : false } ] } ][ec2-user@ip-172-31-22-140 tmp]$ $ curl -o$outputFile http://$host/api/v1/applications/$appID/logs $outputFile will always be an empty zip file If I use executors/. I get info about the drivers and executors how ever no way to get' the log files. The driver output does not have any executorLogs and the workers executorLogs are version of the log files rendered in HTML not the actual log file. $ curl http://$host/api/v1/applications/$appID/executors [ { "id" : "driver", "hostPort" : "172.31.23.203:33303", "rddBlocks" : 0, "memoryUsed" : 0, "diskUsed" : 0, "activeTasks" : 0, "failedTasks" : 0, "completedTasks" : 0, "totalTasks" : 0, "totalDuration" : 0, "totalInputBytes" : 0, "totalShuffleRead" : 0, "totalShuffleWrite" : 0, "maxMemory" : 535953408, "executorLogs" : { } }, { "id" : "1", "hostPort" : "ip-172-31-23-200.us-west-1.compute.internal:51560", "rddBlocks" : 218, "memoryUsed" : 452224280, "diskUsed" : 0, "activeTasks" : 1, "failedTasks" : 0, "completedTasks" : 27756, "totalTasks" : 27757, "totalDuration" : 1650935, "totalInputBytes" : 9619224986, "totalShuffleRead" : 0, "totalShuffleWrite" : 507615, "maxMemory" : 535953408, "executorLogs" : { "stdout" : "http://ec2-xxx.compute.amazonaws.com:8081/logPage/?appId=app-20160711185337 -0049&executorId=1&logType=stdout", "stderr" : "http://ec2-xxx.us-west-1.compute.amazonaws.com:8081/logPage/?appId=app-2016 0711185337-0049&executorId=1&logType=stderr" } Any suggestions would be greatly appreciated Andy
