[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128381#comment-17128381 ] Apache Spark commented on SPARK-8981: - User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/28756 > Set applicationId and appName in log4j MDC > -- > > Key: SPARK-8981 > URL: https://issues.apache.org/jira/browse/SPARK-8981 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Reporter: Paweł Kopiczko >Assignee: Izek Greenfield >Priority: Minor > Fix For: 3.1.0 > > > It would be nice to have, because it's good to have logs in one file when > using log agents (like logentires) in standalone mode. Also allows > configuring rolling file appender without a mess when multiple applications > are running. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128382#comment-17128382 ] Apache Spark commented on SPARK-8981: - User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/28756 > Set applicationId and appName in log4j MDC > -- > > Key: SPARK-8981 > URL: https://issues.apache.org/jira/browse/SPARK-8981 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Reporter: Paweł Kopiczko >Assignee: Izek Greenfield >Priority: Minor > Fix For: 3.1.0 > > > It would be nice to have, because it's good to have logs in one file when > using log agents (like logentires) in standalone mode. Also allows > configuring rolling file appender without a mess when multiple applications > are running. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979236#comment-16979236 ] Izek Greenfield commented on SPARK-8981: [~srowen] I created pull request for that. > Set applicationId and appName in log4j MDC > -- > > Key: SPARK-8981 > URL: https://issues.apache.org/jira/browse/SPARK-8981 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Reporter: Paweł Kopiczko >Priority: Minor > > It would be nice to have, because it's good to have logs in one file when > using log agents (like logentires) in standalone mode. Also allows > configuring rolling file appender without a mess when multiple applications > are running. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623703#comment-14623703 ] Sean Owen commented on SPARK-8981: -- I think it's OK if you can do this via the slf4j API and it doesn't add overhead. I am not sure Logging is actually going to be removed; it's not to be used by apps though. Logging can't use a SparkContext; it's not used where a SparkContext is used, always. I don't think that's important. MDC has static methods. Are you proposing to change the default log message or just make these values available? it might be less intrusive to not change the log output Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623726#comment-14623726 ] Sean Owen commented on SPARK-8981: -- The constructors? Have a look through org.apache.spark.executor. The app ID should be in env.conf Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623712#comment-14623712 ] Sean Owen commented on SPARK-8981: -- Can MDC methods be invoked during executor initialization? where the app name is available? Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623720#comment-14623720 ] Paweł Kopiczko commented on SPARK-8981: --- I think so. Would you mind pointing me to executor initialization code? Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[
https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623700#comment-14623700
]
Paweł Kopiczko commented on SPARK-8981:
---
slf4j supports MDC as well: http://www.slf4j.org/api/org/slf4j/MDC.html
I've analysed how {{Logging}} trait is implemented. If I'm correct every
executor process calls {{initializeLogging}} method because of transient
{{log_}} field. It looks to me that right now it's impossible to pass there
{{SparkContext}} instance (or any other value) without breaking the API. Do you
agree? Have you any idea how to bypass that?
Int terms of this comment: ??This will likely be changed or removed in future
releases.??, are you considering any change right now?
Set applicationId and appName in log4j MDC
--
Key: SPARK-8981
URL: https://issues.apache.org/jira/browse/SPARK-8981
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Paweł Kopiczko
Priority: Minor
It would be nice to have, because it's good to have logs in one file when
using log agents (like logentires) in standalone mode. Also allows
configuring rolling file appender without a mess when multiple applications
are running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623707#comment-14623707 ] Paweł Kopiczko commented on SPARK-8981: --- ??MDC has static methods?? Yes, but I'm not sure how to invoke these in executor thread. Any idea? ??Are you proposing to change the default log message or just make these values available??? Available only. I think it may be needed especially by standalone mode users. YARN users don't need that functionality, because CM stores logs in HDFS by applicationId. I'm not familiar with Mesos, but probably it has ability to store separated logs for each container. I believe the overhead is minimal since it's only two String values in a static map. Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623440#comment-14623440 ] Sean Owen commented on SPARK-8981: -- I think this is specific to log4j though right? it won't help if log4j isn't used. Spark tries to minimize use of log4j in favor of slf4j. Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[
https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622666#comment-14622666
]
Paweł Kopiczko commented on SPARK-8981:
---
Sure. I believe that when executor is spawned it has access to `appName` and
`applicationId` properties of `SparkContext` instance. I'd like it to put these
values in MDC
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html#put(java.lang.String,
java.lang.Object). Having those values in MDC and setting `%X{appName}` and
`%X{applicationId}` in log4j's PatternLayout would allow filtering out specific
application logs from a single file. Does that make sense?
Set applicationId and appName in log4j MDC
--
Key: SPARK-8981
URL: https://issues.apache.org/jira/browse/SPARK-8981
Project: Spark
Issue Type: New Feature
Reporter: Paweł Kopiczko
Priority: Minor
It would be nice to have, because it's good to have logs in one file when
using log agents (like logentires) in standalone mode. Also allows
configuring rolling file appender without a mess when multiple applications
are running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8981) Set applicationId and appName in log4j MDC
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622573#comment-14622573 ] Sean Owen commented on SPARK-8981: -- Can you clarify what this means or provide a PR? Set applicationId and appName in log4j MDC -- Key: SPARK-8981 URL: https://issues.apache.org/jira/browse/SPARK-8981 Project: Spark Issue Type: New Feature Reporter: Paweł Kopiczko Priority: Minor It would be nice to have, because it's good to have logs in one file when using log agents (like logentires) in standalone mode. Also allows configuring rolling file appender without a mess when multiple applications are running. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
