[jira] [Closed] (SPARK-12282) Document spark.jars
[ https://issues.apache.org/jira/browse/SPARK-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Bailey closed SPARK-12282. - > Document spark.jars > --- > > Key: SPARK-12282 > URL: https://issues.apache.org/jira/browse/SPARK-12282 > Project: Spark > Issue Type: Documentation > Components: Documentation >Reporter: Justin Bailey >Priority: Trivial > > The spark.jars property (as implemented in SparkSubmit.scala, > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L516) > is not documented anywhere, and should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-12282) Document spark.jars
[ https://issues.apache.org/jira/browse/SPARK-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Bailey resolved SPARK-12282. --- Resolution: Not A Problem > Document spark.jars > --- > > Key: SPARK-12282 > URL: https://issues.apache.org/jira/browse/SPARK-12282 > Project: Spark > Issue Type: Documentation > Components: Documentation >Reporter: Justin Bailey >Priority: Trivial > > The spark.jars property (as implemented in SparkSubmit.scala, > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L516) > is not documented anywhere, and should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-12282) Document spark.jars
Justin Bailey created SPARK-12282: - Summary: Document spark.jars Key: SPARK-12282 URL: https://issues.apache.org/jira/browse/SPARK-12282 Project: Spark Issue Type: Documentation Reporter: Justin Bailey The spark.jars property (as implemented in SparkSubmit.scala, https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L516) is not documented anywhere, and should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-12282) Document spark.jars
[
https://issues.apache.org/jira/browse/SPARK-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15053676#comment-15053676
]
Justin Bailey edited comment on SPARK-12282 at 12/11/15 10:18 PM:
--
If you pass {{--conf spark.jars=".."}}, you can set this flag, which is
actually pretty useful (its a consistent way to set configuration).
So maybe {{spark-submit}} should warn or throw if this configuration is
included?
was (Author: m4dc4p):
If you pass `--conf spark.jars=".."`, you can set this flag, which is actually
pretty useful (its a consistent way to set configuration).
So maybe spark-submit should warn or throw if this configuration is included?
> Document spark.jars
> ---
>
> Key: SPARK-12282
> URL: https://issues.apache.org/jira/browse/SPARK-12282
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Reporter: Justin Bailey
>Priority: Trivial
>
> The spark.jars property (as implemented in SparkSubmit.scala,
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L516)
> is not documented anywhere, and should be.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12282) Document spark.jars
[ https://issues.apache.org/jira/browse/SPARK-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15053676#comment-15053676 ] Justin Bailey commented on SPARK-12282: --- If you pass `--conf spark.jars=".."`, you can set this flag, which is actually pretty useful (its a consistent way to set configuration). So maybe spark-submit should warn or throw if this configuration is included? > Document spark.jars > --- > > Key: SPARK-12282 > URL: https://issues.apache.org/jira/browse/SPARK-12282 > Project: Spark > Issue Type: Documentation > Components: Documentation >Reporter: Justin Bailey >Priority: Trivial > > The spark.jars property (as implemented in SparkSubmit.scala, > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L516) > is not documented anywhere, and should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0
[
https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045325#comment-15045325
]
Justin Bailey commented on SPARK-8118:
--
Did some research and Parquet uses java.util.logging. You can disable the
chatty output by creating a properties file with these contents:
{noformat}
org.apache.parquet.handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=SEVERE
{noformat}
And then passing the path of the file to Spark when the application is
submitted. Assuming the file lives in /tmp/parquet.logging.properties (of
course, that needs to be available on all worker nodes):
{noformat}
spark-submit \
--conf
spark.driver.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties"
\
--conf
spark.executor.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties"
\
...
{noformat}
> Turn off noisy log output produced by Parquet 1.7.0
> ---
>
> Key: SPARK-8118
> URL: https://issues.apache.org/jira/browse/SPARK-8118
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 1.4.1, 1.5.0
>Reporter: Cheng Lian
>Assignee: Cheng Lian
>Priority: Minor
> Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust
> {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet
> logs via SLF4J, so that we can handle them consistently. In general these
> logs are very useful. Esp. when used to diagnosing Parquet memory issue and
> filter push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0
[
https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045363#comment-15045363
]
Justin Bailey commented on SPARK-8118:
--
I didn't know about that option but I bet it would work - sounds like a good
idea!
> Turn off noisy log output produced by Parquet 1.7.0
> ---
>
> Key: SPARK-8118
> URL: https://issues.apache.org/jira/browse/SPARK-8118
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 1.4.1, 1.5.0
>Reporter: Cheng Lian
>Assignee: Cheng Lian
>Priority: Minor
> Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust
> {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet
> logs via SLF4J, so that we can handle them consistently. In general these
> logs are very useful. Esp. when used to diagnosing Parquet memory issue and
> filter push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0
[
https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042438#comment-15042438
]
Justin Bailey commented on SPARK-8118:
--
Can this issue be re-opened? It's really terrible how much log output Parquet
produces.
> Turn off noisy log output produced by Parquet 1.7.0
> ---
>
> Key: SPARK-8118
> URL: https://issues.apache.org/jira/browse/SPARK-8118
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 1.4.1, 1.5.0
>Reporter: Cheng Lian
>Assignee: Cheng Lian
>Priority: Minor
> Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust
> {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet
> logs via SLF4J, so that we can handle them consistently. In general these
> logs are very useful. Esp. when used to diagnosing Parquet memory issue and
> filter push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0
[
https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042438#comment-15042438
]
Justin Bailey edited comment on SPARK-8118 at 12/4/15 11:48 PM:
Can this issue be re-opened? It's really terrible how much log output Parquet
produces. (Using spark 1.5.1, btw)
was (Author: m4dc4p):
Can this issue be re-opened? It's really terrible how much log output Parquet
produces.
> Turn off noisy log output produced by Parquet 1.7.0
> ---
>
> Key: SPARK-8118
> URL: https://issues.apache.org/jira/browse/SPARK-8118
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 1.4.1, 1.5.0
>Reporter: Cheng Lian
>Assignee: Cheng Lian
>Priority: Minor
> Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust
> {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet
> logs via SLF4J, so that we can handle them consistently. In general these
> logs are very useful. Esp. when used to diagnosing Parquet memory issue and
> filter push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
