[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595256#comment-15595256
]
Chris Brown commented on SPARK-16606:
-
Awesome, thanks! Just a typo, yes, but a significant one for newbs like me who
aren't familiar with Spark internals and we get a message that translates, due
to the typo, to something like:
> Warning! You just tried to retrieve a global singleton, but you should use an
> existing SparkContext!
When in fact, what it actually means to say, once you get the verb
tense/modality right, is more like:
> You tried to get or create a new Spark context with some configuration, but
> there's already a Spark context registered, and we're gonna give you back
> that one (because THERE CAN BE ONLY ONE) regardless of whatever configuration
> you just supplied.
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Assignee: Sean Owen
>Priority: Minor
> Fix For: 2.1.0
>
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595124#comment-15595124
]
Apache Spark commented on SPARK-16606:
--
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/15586
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Assignee: Sean Owen
>Priority: Minor
> Fix For: 2.1.0
>
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595112#comment-15595112
]
Sean Owen commented on SPARK-16606:
---
Oops, yes I missed that. It's just a typo but worth touching up. One sec...
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Assignee: Sean Owen
>Priority: Minor
> Fix For: 2.1.0
>
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595098#comment-15595098
]
Chris Brown commented on SPARK-16606:
-
Hi [~sowen], I noticed your PR #14533 and am excited about a little less noise
in my logs, but it seems your PR didn't address the same typo in SparkSession?
See
https://github.com/apache/spark/search?utf8=%E2%9C%93&q=%22Use+an+existing%22&type=Code
and
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L817-L829
Is that a no-fix?
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Assignee: Sean Owen
>Priority: Minor
> Fix For: 2.1.0
>
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411328#comment-15411328
]
Apache Spark commented on SPARK-16606:
--
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/14533
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395420#comment-15395420
]
Jacek Laskowski commented on SPARK-16606:
-
Haven't really started to be honest. I'd like to work on it though so please
leave it open for a couple of days. Thanks.
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395378#comment-15395378
]
Sean Owen commented on SPARK-16606:
---
[~ja...@japila.pl] are you working on this or should I zap it?
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382346#comment-15382346
]
Jacek Laskowski commented on SPARK-16606:
-
On it. Thanks.
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16606) Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."
[
https://issues.apache.org/jira/browse/SPARK-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382224#comment-15382224
]
Sean Owen commented on SPARK-16606:
---
Agree, and this needs a fix in two places in SparkSession
> Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an
> existing SparkContext, some configuration may not take effect."
> --
>
> Key: SPARK-16606
> URL: https://issues.apache.org/jira/browse/SPARK-16606
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {{SparkContext.getOrCreate}} should really be checking whether the code gets
> the already-created instance or creating a new one.
> Just a nit-pick: the warning message should also be "Using..." not "Use"
> {code}
> scala> sc.version
> res2: String = 2.1.0-SNAPSHOT
> scala> sc
> res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> scala> SparkContext.getOrCreate
> 16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
