[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21495 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196995267
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
Really not worthy to block on this thing (if it is the only issue).
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196993585
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
We can fix them in place. There are many other style nits too. It's not
worth swiping them making the backport harder.
I am fine with ignoring such nits and they don't block this PR but it's not
something we should say the opposite side is okay.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196992852
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
Well, there's a bunch of code in Spark where both 4-space and 2-space
indentation are existed. From my thought it is not so worthy to stick to this
tiny point.
If you want to fix it, then creating a separate PR to fix all of them.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196878408
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
Guys, parameters are listed in 4 spaces and other keywords after that are
lined up with 2 spaces, which is written in
https://github.com/databricks/scala-style-guide#spacing-and-indentation
In case of two lines, it's not explicitly written but wouldn't we better
stick to the example as possible as we can?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user som-snytt commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r194294485
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
It's definitely two spaces after a period. I've been wanting to make that
joke, but held off.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r194287473
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
IIRC, four spaces is OK.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r194176806
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
I thought for `extends`, it's four spaces?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193981694
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
I don't know why it is working for me, and maybe my local deps get mess up
:(
I just updated the jline, and please try and see if it works.
Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193978714
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
--- End diff --
nit: two spaces
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193942643
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
Thank you for confirming, @som-snytt .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user som-snytt commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193938805
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
Completion was upgraded since the old days; also, other bugs required
updating jline. There is interest in upgrading to jline 3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193936286
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
Can we upgrade to the corresponding `jline` version together in this PR,
@dbtsai ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193935371
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
Yep. Of course, it's a clean clone and build of this PR. Accoring to the
error message and the followings. We need `jline-2.14.3.jar` because `scala`
uses the API of higher version of `jline`. Could you confirm this, @som-snytt ?
```
$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.18.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
$ javap -cp jline-2.12.1.jar
jline.console.completer.CandidateListCompletionHandler
Compiled from "CandidateListCompletionHandler.java"
public class jline.console.completer.CandidateListCompletionHandler
implements jline.console.completer.CompletionHandler {
public jline.console.completer.CandidateListCompletionHandler();
public boolean complete(jline.console.ConsoleReader,
java.util.List, int) throws java.io.IOException;
public static void setBuffer(jline.console.ConsoleReader,
java.lang.CharSequence, int) throws java.io.IOException;
public static void printCandidates(jline.console.ConsoleReader,
java.util.Collection) throws java.io.IOException;
}
$ javap -cp jline-2.14.3.jar
jline.console.completer.CandidateListCompletionHandler
Compiled from "CandidateListCompletionHandler.java"
public class jline.console.completer.CandidateListCompletionHandler
implements jline.console.completer.CompletionHandler {
public jline.console.completer.CandidateListCompletionHandler();
public boolean getPrintSpaceAfterFullCompletion();
public void setPrintSpaceAfterFullCompletion(boolean);
public boolean isStripAnsi();
public void setStripAnsi(boolean);
public boolean complete(jline.console.ConsoleReader,
java.util.List, int) throws java.io.IOException;
public static void setBuffer(jline.console.ConsoleReader,
java.lang.CharSequence, int) throws java.io.IOException;
public static void printCandidates(jline.console.ConsoleReader,
java.util.Collection) throws java.io.IOException;
}
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193927042
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
It is working for me. I got `scala-compiler-2.11.12.jar` in my classpath.
Can you do a clean build?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193920472
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
It looks like `jline` version mismatch.
- Spark 2.4.0-SNAPSHOT uses
[2.12.1](https://github.com/apache/spark/blob/master/pom.xml#L749)
- Scala 2.11.12 uses
[2.14.3](https://github.com/scala/scala/blob/2.11.x/versions.properties#L35)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r193919141
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
For my environment, I'm hitting `NoSuchMethodError` like the following. Did
you see something like this?
```scala
~/PR-21495:PR-21495$ bin/spark-shell
18/06/07 23:39:00 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-SNAPSHOT
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala> Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1528414746558).
Spark session available as 'spark'.
Exception in thread "main" java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
at
scala.tools.nsc.interpreter.jline.InteractiveReader.postInit(JLineReader.scala:54)
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r192961905
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter._
-class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends
IMain(settings, out) {
- self =>
+class SparkILoopInterpreter(settings: Settings, out: JPrintWriter,
initializeSpark: () => Unit)
+extends IMain(settings, out) { self =>
+
+ /**
+ * We override `initializeSynchronous` to initialize Spark *after*
`intp` is properly initialized
+ * and *before* the REPL sees any files in the private `loadInitFiles`
functions, so that
+ * the Spark context is visible in those files.
+ *
+ * This is a bit of a hack, but there isn't another hook available to us
at this point.
+ *
+ * See the discussion in Scala community
https://github.com/scala/bug/issues/10913 for detail.
+ */
+ override def initializeSynchronous(): Unit = {
+super.initializeSynchronous()
+initializeSpark()
--- End diff --
@som-snytt It's working, but I'm wondering if I'm doing it correctly.
In this case, I'll use `$intp` without checking
```scala
if (intp.reporter.hasErrors) {
echo("Interpreter encountered errors during initialization!")
null
}
```
in `iLoop.scala`.
And `intp.quietBind(NamedParam[IMain]("$intp", intp)(tagOfIMain,
classTag[IMain]))` will not be executed before our custom Spark initialization
code.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/21495 [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/dbtsai/spark scala-2.11.12 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21495.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21495 commit de790fd251ba3727bba23ceb1ca07559d25b7e87 Author: DB Tsai Date: 2018-05-29T18:03:40Z Upgrade scala to 2.11.12 and 2.12.6 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
