[
https://issues.apache.org/jira/browse/SPARK-19466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eren Avsarogullari updated SPARK-19466:
---------------------------------------
Description:
Fair Scheduler Logging for the following cases can be useful for the user.
1- If *valid* spark.scheduler.allocation.file property is set, user can be
informed so user can aware which scheduler file is processed when SparkContext
initializes.
2- If *invalid* spark.scheduler.allocation.file property is set, currently, the
following stacktrace is shown to user. In addition to this, more meaningful
message can be shown to user by emphasizing the problem at building level of
fair scheduler and covering other potential issues at this level.
{code:xml}
Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No such
file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)
{code}
3- If spark.scheduler.allocation.file property is not set and *default* fair
scheduler file(fairscheduler.xml) is found in classpath, it will be loaded but
currently, user is not informed so logging can be useful.
4- If spark.scheduler.allocation.file property is not set and default fair
scheduler file does not exist, currently, user is not informed so logging can
be useful.
was:
Fair Scheduler Logging for the following cases can be useful for the user.
1- If valid spark.scheduler.allocation.file property is set, user can be
informed so user can aware which scheduler file is processed when SparkContext
initializes.
2- If invalid spark.scheduler.allocation.file is set, currently, the following
stacktrace is shown to user. In addition to this, more meaningful message can
be shown to user by emphasizing the problem at building level of fair scheduler
and covering other potential issues at this level.
Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No such
file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)
3- If spark.scheduler.allocation.file is not set and default fair scheduler
file(fairscheduler.xml) is found in classpath, it will be loaded but currently,
user is not informed so logging can be useful.
4- If spark.scheduler.allocation.file is not set and default fair scheduler
file does not exist, currently, user is not informed so logging can be useful.
> Improve Fair Scheduler Logging
> ------------------------------
>
> Key: SPARK-19466
> URL: https://issues.apache.org/jira/browse/SPARK-19466
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 2.2.0
> Reporter: Eren Avsarogullari
>
> Fair Scheduler Logging for the following cases can be useful for the user.
> 1- If *valid* spark.scheduler.allocation.file property is set, user can be
> informed so user can aware which scheduler file is processed when
> SparkContext initializes.
> 2- If *invalid* spark.scheduler.allocation.file property is set, currently,
> the following stacktrace is shown to user. In addition to this, more
> meaningful message can be shown to user by emphasizing the problem at
> building level of fair scheduler and covering other potential issues at this
> level.
> {code:xml}
> Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No
> such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at
> org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
> at
> org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)
> {code}
> 3- If spark.scheduler.allocation.file property is not set and *default* fair
> scheduler file(fairscheduler.xml) is found in classpath, it will be loaded
> but currently, user is not informed so logging can be useful.
> 4- If spark.scheduler.allocation.file property is not set and default fair
> scheduler file does not exist, currently, user is not informed so logging can
> be useful.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
