Re: Scala Language NPE
Ah nevermind, the fix is to get rid of "return" from my method. There's
probably a bug somewhere related to the repl taking bad input more cleanly,
but this isn't the end of the world once you figure out what the issue is.
Thanks for the time,
Andrew
On Mon, Jun 2, 2014 at 11:35 PM, Andrew Ash wrote:
> // observed in Spark 1.0
>
> Scala devs,
>
> I was observing an unusual NPE in my code recently, and came up with the
> below minimal test case:
>
> class Super extends Serializable {
> lazy val superVal: String = null
> }
>
> class Sub extends Super {
> lazy val subVal: String = {
> try {
> "literal"
> } catch {
> case _:Throwable => return superVal
> }
> }
> }
>
> Save this to a file, open the Spark shell, and load with ":l
> /tmp/test.scala"
>
> I got the below really unusual exception. It goes away though when
> removing the try/catch inside subVal and just returning either a straight
> literal or superVal.
>
>
> Is this a bug in Spark, or Scala, or my code, or what? I think it might
> be related to the Spark Repl doing magic but I'm unsure what.
>
> Cheers!
> Andrew
>
>
>
>
> scala> :l /tmp/test.scala
> Loading /tmp/test.scala...
> defined class Super
>
> while compiling:
> during phase: mixin
> library version: version 2.10.4
> compiler version: version 2.10.4
> reconstructed args:
>
> last tree to typer: Apply(constructor $read)
> symbol: constructor $read in class $read (flags:
> )
>symbol definition: def (): $line9.$read
> tpe: $line9.$read
>symbol owners: constructor $read -> class $read -> package $line9
>context owners: class iwC$Sub -> package $line9
>
> == Enclosing template or block ==
>
> Template( // val : , tree.tpe=$line9.iwC$Sub
> "$line5.$read$$iwC$$iwC$$iwC$$iwC$Super" // parents
> ValDef(
> private
> "_"
>
>
> )
> // 6 statements
> ValDef( // lazy private[this] var subVal: String
> private lazy
> "subVal "
> // tree.tpe=String
>
> )
> DefDef( // lazy val subVal(): String
> lazy
> "subVal"
> []
> List(Nil)
> // tree.tpe=String
> Block( // tree.tpe=String
> ValDef( // val nonLocalReturnKey1: Object
>
> "nonLocalReturnKey1"
> // tree.tpe=Object
> Apply( // def (): Object in class Object, tree.tpe=Object
> new Object."" // def (): Object in class Object,
> tree.tpe=()Object
> Nil
> )
> )
> Try( // tree.tpe=String
> Block( // tree.tpe=String
> Assign( // tree.tpe=Unit
> $read$$iwC$$iwC$$iwC$$iwC$Sub.this."subVal " // lazy
> private[this] var subVal: String, tree.tpe=String
> Block( // tree.tpe=String
> {}
> Apply( // final private[this] def
> liftedTree1$1(nonLocalReturnKey1$1: Object): String, tree.tpe=String
> $read$$iwC$$iwC$$iwC$$iwC$Sub.this."liftedTree1$1" //
> final private[this] def liftedTree1$1(nonLocalReturnKey1$1: Object):
> String, tree.tpe=(nonLocalReturnKey1$1: Object)String
> "nonLocalReturnKey1" // val nonLocalReturnKey1: Object,
> tree.tpe=Object
> )
> )
> )
> $read$$iwC$$iwC$$iwC$$iwC$Sub.this."subVal " // lazy
> private[this] var subVal: String, tree.tpe=String
> )
> CaseDef( // tree.tpe=String
> Bind( // val ex: runtime.NonLocalReturnControl,
> tree.tpe=runtime.NonLocalReturnControl
> "ex"
> Typed( // tree.tpe=runtime.NonLocalReturnControl
> "_" // tree.tpe=runtime.NonLocalReturnControl
>// tree.tpe=runtime.NonLocalReturnControl
> )
> )
> If( // tree.tpe=String
> Apply( // final def eq(x$1: Object): Boolean in class Object,
> tree.tpe=Boolean
> ex.key()."eq" // final def eq(x$1: Object): Boolean in class
> Object, tree.tpe=(x$1: Object)Boolean
> "nonLocalReturnKey1" // val nonLocalReturnKey1: Object,
> tree.tpe=Object
> )
> Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class
> Object, tree.tpe=String
> TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in
> class Object, tree.tpe=()String
> ex.value()."$asInstanceOf" // final def $asInstanceOf[T0
> >: ? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
> // tree.tpe=String
> )
> Nil
> )
> Throw("ex")tree.tpe=Nothing
> )
> )
> )
> )
> )
> ValDef( // protected val $outer: $line9.iwC
> protected
> "$outer "
> // tree.tpe=$line9.iwC
>
> )
> DefDef( // val $outer(): $line9.iwC
>
> "$line9$$read$$iwC$$iwC$$iwC$$iwC$Sub$$$outer"
> []
> List(Nil)
> // tree.tpe=$line9.iwC
> $read$$iwC$$iwC$$iwC$$iwC$
Scala Language NPE
// observed in Spark 1.0
Scala devs,
I was observing an unusual NPE in my code recently, and came up with the
below minimal test case:
class Super extends Serializable {
lazy val superVal: String = null
}
class Sub extends Super {
lazy val subVal: String = {
try {
"literal"
} catch {
case _:Throwable => return superVal
}
}
}
Save this to a file, open the Spark shell, and load with ":l
/tmp/test.scala"
I got the below really unusual exception. It goes away though when
removing the try/catch inside subVal and just returning either a straight
literal or superVal.
Is this a bug in Spark, or Scala, or my code, or what? I think it might be
related to the Spark Repl doing magic but I'm unsure what.
Cheers!
Andrew
scala> :l /tmp/test.scala
Loading /tmp/test.scala...
defined class Super
while compiling:
during phase: mixin
library version: version 2.10.4
compiler version: version 2.10.4
reconstructed args:
last tree to typer: Apply(constructor $read)
symbol: constructor $read in class $read (flags:
)
symbol definition: def (): $line9.$read
tpe: $line9.$read
symbol owners: constructor $read -> class $read -> package $line9
context owners: class iwC$Sub -> package $line9
== Enclosing template or block ==
Template( // val : , tree.tpe=$line9.iwC$Sub
"$line5.$read$$iwC$$iwC$$iwC$$iwC$Super" // parents
ValDef(
private
"_"
)
// 6 statements
ValDef( // lazy private[this] var subVal: String
private lazy
"subVal "
// tree.tpe=String
)
DefDef( // lazy val subVal(): String
lazy
"subVal"
[]
List(Nil)
// tree.tpe=String
Block( // tree.tpe=String
ValDef( // val nonLocalReturnKey1: Object
"nonLocalReturnKey1"
// tree.tpe=Object
Apply( // def (): Object in class Object, tree.tpe=Object
new Object."" // def (): Object in class Object,
tree.tpe=()Object
Nil
)
)
Try( // tree.tpe=String
Block( // tree.tpe=String
Assign( // tree.tpe=Unit
$read$$iwC$$iwC$$iwC$$iwC$Sub.this."subVal " // lazy
private[this] var subVal: String, tree.tpe=String
Block( // tree.tpe=String
{}
Apply( // final private[this] def
liftedTree1$1(nonLocalReturnKey1$1: Object): String, tree.tpe=String
$read$$iwC$$iwC$$iwC$$iwC$Sub.this."liftedTree1$1" // final
private[this] def liftedTree1$1(nonLocalReturnKey1$1: Object): String,
tree.tpe=(nonLocalReturnKey1$1: Object)String
"nonLocalReturnKey1" // val nonLocalReturnKey1: Object,
tree.tpe=Object
)
)
)
$read$$iwC$$iwC$$iwC$$iwC$Sub.this."subVal " // lazy
private[this] var subVal: String, tree.tpe=String
)
CaseDef( // tree.tpe=String
Bind( // val ex: runtime.NonLocalReturnControl,
tree.tpe=runtime.NonLocalReturnControl
"ex"
Typed( // tree.tpe=runtime.NonLocalReturnControl
"_" // tree.tpe=runtime.NonLocalReturnControl
// tree.tpe=runtime.NonLocalReturnControl
)
)
If( // tree.tpe=String
Apply( // final def eq(x$1: Object): Boolean in class Object,
tree.tpe=Boolean
ex.key()."eq" // final def eq(x$1: Object): Boolean in class
Object, tree.tpe=(x$1: Object)Boolean
"nonLocalReturnKey1" // val nonLocalReturnKey1: Object,
tree.tpe=Object
)
Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class
Object, tree.tpe=String
TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in
class Object, tree.tpe=()String
ex.value()."$asInstanceOf" // final def $asInstanceOf[T0 >:
? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
// tree.tpe=String
)
Nil
)
Throw("ex")tree.tpe=Nothing
)
)
)
)
)
ValDef( // protected val $outer: $line9.iwC
protected
"$outer "
// tree.tpe=$line9.iwC
)
DefDef( // val $outer(): $line9.iwC
"$line9$$read$$iwC$$iwC$$iwC$$iwC$Sub$$$outer"
[]
List(Nil)
// tree.tpe=$line9.iwC
$read$$iwC$$iwC$$iwC$$iwC$Sub.this."$outer " // protected val $outer:
$line9.iwC, tree.tpe=$line9.iwC
)
DefDef( // final private[this] def liftedTree1$1(nonLocalReturnKey1$1:
Object): String
private final
"liftedTree1"
[]
// 1 parameter list
ValDef( // nonLocalReturnKey1$1: Object
"nonLocalReturnKey1$1"
// tree.tpe=Object
)
// tree.tpe=String
Try( // tree.tpe=String
"literal"
CaseDef( // tree.tpe=Nothing
Typed( // tree.tpe=Throwable
"_" // tree.tpe=Throwable
// tree.tpe=Throwable
)
Throw( // tree.tpe=
Spark 1.1 Window and 1.0 Wrap-up
Hey All,
I wanted to announce the the Spark 1.1 release window:
June 1 - Merge window opens
July 25 - Cut-off for new pull requests
August 1 - Merge window closes (code freeze), QA period starts
August 15+ - RC's and voting
This is consistent with the "3 month" release cycle we are targeting.
I'd really encourage people submitting larger features to do so during
the month of June, as features submitted closer to the window closing
could end up getting pushed into the next release.
I wanted to reflect a bit as well on the 1.0 release. First, thanks to
everyone who was involved in this release. It was the largest release
ever and it's something we should all be proud of.
In the 1.0 release, we cleaned up and consolidated several parts of the
Spark code base. In particular, we consolidated the previously
fragmented process of submitting Spark jobs across a wide variety of
environments {YARN/Mesos/Standalone, Windows/Unix, Python/Java/Scala}.
We also brought the three language API's into much closer alignment.
These were difficult (but critical) tasks towards having a stable
deployment environment on which higher level libraries can build.
These cross-cutting changes also had associated test burden resulting
in an extended QA period. The 1.1, 1.2, 1.3, family of releases are
intended to be smaller releases, and I'd like to deliver them with
very predictable timing to the community. This will mean being fairly
strict about freezes and investing in QA infrastructure to allow us to
get through voting more quickly.
With 1.0 shipped, now is a great time to catch up on code reviews and look at
outstanding patches. Despite the large queue, we've actually been
consistently merging/closing about 80% of proposed PR's, which
is definitely good (for instance, we have 170 outstanding out of 950
proposed), but there remain a lot of people waiting on reviews, and
it's something everyone can help with!
Thanks again to everyone involved. Looking forward to more great releases!
- Patrick
Spark 1.1-snapshot: java.io.FileNotFoundException from ShuffleMapTask
Quite often I notice that shuffle file is missing thus FileNotFoundException is throws. Any idea why shuffle file will be missing ? Am I running low in memory? (I am using latest code from master branch on yarn-hadoop-2.2) -- java.io.FileNotFoundException: /var/storage/sda3/nm-local/usercache/npanj/appcache/application_1401394632504_0131/spark-local-20140603050956-6728/20/shuffle_0_2_97 (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:221) at org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:116) at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:177) at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:161) at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-1-snapshot-java-io-FileNotFoundException-from-ShuffleMapTask-tp6915.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Which version does the binary compatibility test against by default?
Yeah - check out sparkPreviousArtifact in the build: https://github.com/apache/spark/blob/master/project/SparkBuild.scala#L325 - Patrick On Mon, Jun 2, 2014 at 5:30 PM, Xiangrui Meng wrote: > Is there a way to specify the target version? -Xiangrui
Re: Eclipse Scala IDE/Scala test and Wiki
Madhu, can you send me your Wiki username? (Sending it just to me is fine.) I can add you to the list to edit it. Matei On Jun 2, 2014, at 6:27 PM, Reynold Xin wrote: > I tried but didn't find where I could add you. You probably need Matei to > help out with this. > > > > On Mon, Jun 2, 2014 at 7:43 AM, Madhu wrote: > I was able to set up Spark in Eclipse using the Spark IDE plugin. > I also got unit tests running with Scala Test, which makes development quick > and easy. > > I wanted to document the setup steps in this wiki page: > > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IDESetup > > I can't seem to edit that page. > Confluence usually has a an "Edit" button in the upper right, but it does > not appear for me, even though I am logged in. > > Am I missing something? > > > > - > -- > Madhu > https://www.linkedin.com/in/msiddalingaiah > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Eclipse-Scala-IDE-Scala-test-and-Wiki-tp6908.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. >
Re: Eclipse Scala IDE/Scala test and Wiki
I tried but didn't find where I could add you. You probably need Matei to help out with this. On Mon, Jun 2, 2014 at 7:43 AM, Madhu wrote: > I was able to set up Spark in Eclipse using the Spark IDE plugin. > I also got unit tests running with Scala Test, which makes development > quick > and easy. > > I wanted to document the setup steps in this wiki page: > > > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IDESetup > > I can't seem to edit that page. > Confluence usually has a an "Edit" button in the upper right, but it does > not appear for me, even though I am logged in. > > Am I missing something? > > > > - > -- > Madhu > https://www.linkedin.com/in/msiddalingaiah > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Eclipse-Scala-IDE-Scala-test-and-Wiki-tp6908.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. >
Which version does the binary compatibility test against by default?
Is there a way to specify the target version? -Xiangrui
Re: [VOTE] Release Apache Spark 1.0.0 (rc5)
On Mon, Jun 2, 2014 at 6:05 PM, Marcelo Vanzin wrote: > You mentioned something in your shading argument that kinda reminded > me of something. Spark currently depends on slf4j implementations and > log4j with "compile" scope. I'd argue that's the wrong approach if > we're talking about Spark being used embedded inside applications; > Spark should only depend on the slf4j API package, and let the > application provide the underlying implementation. Good idea in general; in practice, the drawback is that you can't do things like set log levels if you only depend on the SLF4J API. There are a few cases where that's nice to control, and that's only possible if you bind to a particular logger as well. You typically bundle a SLF4J binding anyway, to give a default, or else the end-user has to know to also bind some SLF4J logger to get output. Of course it does make for a bit more surgery if you want to override the binding this way. Shading can bring a whole new level of confusion; I myself would only use it where essential as a workaround. Same with trying to make more elaborate custom classloading schemes -- never in my darkest nightmares have I imagine the failure modes that probably pop up when that goes wrong. I think the library collisions will get better over time as only later versions of Hadoop are in scope, for example, and/or one build system is in play. I like tackling complexity along those lines first.
Re: [VOTE] Release Apache Spark 1.0.0 (rc5)
Hi Patrick, Thanks for all the explanations, that makes sense. @DeveloperApi worries me a little bit especially because of the things Colin mentions - it's sort of hard to make people move off of APIs, or support different versions of the same API. But maybe if expectations (or lack thereof) are set up front, there will be less issues. You mentioned something in your shading argument that kinda reminded me of something. Spark currently depends on slf4j implementations and log4j with "compile" scope. I'd argue that's the wrong approach if we're talking about Spark being used embedded inside applications; Spark should only depend on the slf4j API package, and let the application provide the underlying implementation. The assembly jars could include an implementation (since I assume those are currently targeted at cluster deployment and not embedding). That way there is less sources of conflict at runtime (i.e. the "multiple implementation jars" messages you can see when running some Spark programs). On Fri, May 30, 2014 at 10:54 PM, Patrick Wendell wrote: > 2. Many libraries like logging subsystems, configuration systems, etc > rely on static state and initialization. I'm not totally sure how e.g. > slf4j initializes itself if you have both a shaded and non-shaded copy > of slf4j present. -- Marcelo
Eclipse Scala IDE/Scala test and Wiki
I was able to set up Spark in Eclipse using the Spark IDE plugin. I also got unit tests running with Scala Test, which makes development quick and easy. I wanted to document the setup steps in this wiki page: https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IDESetup I can't seem to edit that page. Confluence usually has a an "Edit" button in the upper right, but it does not appear for me, even though I am logged in. Am I missing something? - -- Madhu https://www.linkedin.com/in/msiddalingaiah -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Eclipse-Scala-IDE-Scala-test-and-Wiki-tp6908.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
