[jira] [Commented] (SPARK-33001) Why am I receiving this warning?
[
https://issues.apache.org/jira/browse/SPARK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17638647#comment-17638647
]
geekyouth commented on SPARK-33001:
---
i have set "spark-3.0.0-bin-hadoop3.2\conf\spark-defaults.conf" as follow:
{code:java}
spark.executor.processTreeMetrics.enabled false {code}
then restart terminal，run spark-shell， but the warn rise again
> Why am I receiving this warning?
>
>
> Key: SPARK-33001
> URL: https://issues.apache.org/jira/browse/SPARK-33001
> Project: Spark
> Issue Type: Question
> Components: Spark Core
>Affects Versions: 3.0.1
>Reporter: George Fotopoulos
>Priority: Major
>
> I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2
> with Docker 2.3.0.5
> I am running Windows 10 build 2004
> Can somebody explain me why am I receiving this warning and what can I do
> about it?
> I tried googling this warning but, all I found was people asking about it and
> no answers.
> [screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]
> {code:scala}
> WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a
> result reporting of ProcessTree metrics is stopped
> {code}
> Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33001) Why am I receiving this warning?
[
https://issues.apache.org/jira/browse/SPARK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232192#comment-17232192
]
Wing Yew Poon commented on SPARK-33001:
---
I may have been the last to touch ProcfsMetricsGetter.scala but it was authored
by [~rezasafi].
[~xorz57] and [~dannylee8], are you encountering the warning when running Spark
on Windows? The warning is harmless. ProcfsMetricsGetter is only meant to be
run on Linux machines with a /proc filesystem. The warning happened because the
command "getconf PAGESIZE" was run and it is not a valid command on Windows so
an exception was caught.
ProcfsMetricsGetter is actually only used when
spark.executor.processTreeMetrics.enabled=true. However, the class is
instantiated and the warning occurs then, even though after that the class is
not used.
Ideally, you should not see this warning. Ideally, isProcfsAvailable should be
checked before computePageSize() is called (the latter should not be called if
procfs is not available, and it is not on Windows). So it is a minor bug that
you see this warning. But it can be safely ignored.
> Why am I receiving this warning?
>
>
> Key: SPARK-33001
> URL: https://issues.apache.org/jira/browse/SPARK-33001
> Project: Spark
> Issue Type: Question
> Components: Spark Core
>Affects Versions: 3.0.1
>Reporter: George Fotopoulos
>Priority: Major
>
> I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2
> with Docker 2.3.0.5
> I am running Windows 10 build 2004
> Can somebody explain me why am I receiving this warning and what can I do
> about it?
> I tried googling this warning but, all I found was people asking about it and
> no answers.
> [screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]
> {code:scala}
> WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a
> result reporting of ProcessTree metrics is stopped
> {code}
> Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33001) Why am I receiving this warning?
[
https://issues.apache.org/jira/browse/SPARK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231749#comment-17231749
]
Danny Lee commented on SPARK-33001:
---
[~xorz57] - I followed the code back to 1305 in this file
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala]
but unfortunately even if I set my system to "isTesting" like in line 1930 the
debug output for computePageSize() only shows the number 4096. I think this is
in early stages of testing, but I dont think it was affecting my code at all
and my job was failing for a different reason (most likely memory related).
The warning just happened to show up before my exception, but I believe its
unrelated.
> Why am I receiving this warning?
>
>
> Key: SPARK-33001
> URL: https://issues.apache.org/jira/browse/SPARK-33001
> Project: Spark
> Issue Type: Question
> Components: Spark Core
>Affects Versions: 3.0.1
>Reporter: George Fotopoulos
>Priority: Major
>
> I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2
> with Docker 2.3.0.5
> I am running Windows 10 build 2004
> Can somebody explain me why am I receiving this warning and what can I do
> about it?
> I tried googling this warning but, all I found was people asking about it and
> no answers.
> [screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]
> {code:scala}
> WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a
> result reporting of ProcessTree metrics is stopped
> {code}
> Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33001) Why am I receiving this warning?
[
https://issues.apache.org/jira/browse/SPARK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231703#comment-17231703
]
Danny Lee commented on SPARK-33001:
---
[~xorz57] I'm getting the same error, right before Spark bails and throws:
20/11/13 12:41:51 ERROR MicroBatchExecution: Query [id =
32320bc7-d7ba-49b4-8a56-1166a4f2d6db, runId =
d7cc93c2-41ef-4765-aecd-9cd453c25905] terminated with error
org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut
down
Its related to line 97 in this file, but there isn't much documentation.
Executor metrics appear relatively new, even [~wypoon] n (last committer)
admits "Executor metrics are new in Spark 3.0. They lack documentation."
I'll post back if I find a solution.
> Why am I receiving this warning?
>
>
> Key: SPARK-33001
> URL: https://issues.apache.org/jira/browse/SPARK-33001
> Project: Spark
> Issue Type: Question
> Components: Spark Core
>Affects Versions: 3.0.1
>Reporter: George Fotopoulos
>Priority: Major
>
> I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2
> with Docker 2.3.0.5
> I am running Windows 10 build 2004
> Can somebody explain me why am I receiving this warning and what can I do
> about it?
> I tried googling this warning but, all I found was people asking about it and
> no answers.
> [screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]
> {code:scala}
> WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a
> result reporting of ProcessTree metrics is stopped
> {code}
> Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33001) Why am I receiving this warning?
[
https://issues.apache.org/jira/browse/SPARK-33001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202837#comment-17202837
]
George Fotopoulos commented on SPARK-33001:
---
Sure it is a question. I specified the type of this issue as a question also.
The reason why I am posting this question here is because I found the very same
question on the mailing list but with no answers.
http://apache-spark-user-list.1001560.n3.nabble.com/WARN-ProcfsMetricsGetter-Exception-when-trying-to-compute-pagesize-as-a-result-reporting-of-ProcessTd-td37812.html
> Why am I receiving this warning?
>
>
> Key: SPARK-33001
> URL: https://issues.apache.org/jira/browse/SPARK-33001
> Project: Spark
> Issue Type: Question
> Components: Spark Core
>Affects Versions: 3.0.1
>Reporter: George Fotopoulos
>Priority: Major
>
> I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2
> with Docker 2.3.0.5
> I am running Windows 10 build 2004
> Can somebody explain me why am I receiving this warning and what can I do
> about it?
> I tried googling this warning but, all I found was people asking about it and
> no answers.
> [screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]
> {code:scala}
> WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a
> result reporting of ProcessTree metrics is stopped
> {code}
> Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
