[
https://issues.apache.org/jira/browse/MESOS-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336124#comment-15336124
]
Mallik Singaraju commented on MESOS-4087:
-----------------------------------------
Hi, we are using spark 1.6.1 deployed and running on mesos and I need some info
on how to capture the log of spark executors running on mesos in spark 1.6.1.
We are not using the container based approach to deploy spark on mesos. Instead
we are currently just deploying spark job (<myapp>.jar) though spark-submit.
I am not currently able to override the default behavior of spark executors
always picking up the http://log4j.properties in the <SPARK_HOME>/conf .
I tried setting the log4j.configuration to http://log4j.properties in the
classpath of <my-application>.jar and did supply that as argument to
spark-submit. That does not seem to capture any logs of the spark executor
tasks in mesos. I did figure out that you worked on the logging piece through
JIRA. Do you have any recommendation on how to approach this?
> Introduce a module for logging executor/task output
> ---------------------------------------------------
>
> Key: MESOS-4087
> URL: https://issues.apache.org/jira/browse/MESOS-4087
> Project: Mesos
> Issue Type: Task
> Components: containerization, modules
> Reporter: Joseph Wu
> Assignee: Joseph Wu
> Labels: logging, mesosphere
> Fix For: 0.27.0
>
>
> Existing executor/task logs are logged to files in their sandbox directory,
> with some nuances based on which containerizer is used (see background
> section in linked document).
> A logger for executor/task logs has the following requirements:
> * The logger is given a command to run and must handle the stdout/stderr of
> the command.
> * The handling of stdout/stderr must be resilient across agent failover.
> Logging should not stop if the agent fails.
> * Logs should be readable, presumably via the web UI, or via some other
> module-specific UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
