Hi,
It's with the master built today. Why can't I call
ds.foreachPartition(println)? Is using type annotation the only way to
go forward? I'd be so sad if that's the case.
scala> ds.foreachPartition(println)
:28: error: overloaded method value foreachPartition with alternatives:
(func:
org.apa
Do you not mean ds.foreachPartition(_.foreach(println)) or similar?
On Tue, Jul 5, 2016 at 2:22 PM, Jacek Laskowski wrote:
> Hi,
>
> It's with the master built today. Why can't I call
> ds.foreachPartition(println)? Is using type annotation the only way to
> go forward? I'd be so sad if that's th
Sort of. Your example works, but could you do a mere
ds.foreachPartition(println)? Why not? What should I even see the Java
version?
scala> val ds = spark.range(10)
ds: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> ds.foreachPartition(println)
:26: error: overloaded method value foreac
A DStream is a sequence of RDDs, not of elements. I don't think I'd
expect to express an operation on a DStream as if it were elements.
On Tue, Jul 5, 2016 at 2:47 PM, Jacek Laskowski wrote:
> Sort of. Your example works, but could you do a mere
> ds.foreachPartition(println)? Why not? What shoul
ds is Dataset and the problem is that println (or any other
one-element function) would not work here (and perhaps other methods
with two variants - Java's and Scala's).
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Right, should have noticed that in your second mail. But foreach
already does what you want, right? it would be identical here.
How these two methods do conceptually different things on different
arguments. I don't think I'd expect them to accept the same functions.
On Tue, Jul 5, 2016 at 3:18 PM
Well, there is foreach for Java and another foreach for Scala. That's
what I can understand. But while supporting two language-specific APIs
-- Scala and Java -- Dataset API lost support for such simple calls
without type annotations so you have to be explicit about the variant
(since I'm using Sca
This seems like a Scala compiler bug.
On Tuesday, July 5, 2016, Jacek Laskowski wrote:
> Well, there is foreach for Java and another foreach for Scala. That's
> what I can understand. But while supporting two language-specific APIs
> -- Scala and Java -- Dataset API lost support for such simple
Hi Reynold,
Is this already reported and tracked somewhere. I'm quite sure that
people will be asking about the reasons Spark does this. Where are
such issues reported usually?
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apac
I don't think that's a scala compiler bug.
println is a valid expression that returns unit.
Unit is not a single-argument function, and does not match any of the
overloads of foreachPartition
You may be used to a conversion taking place when println is passed to
method expecting a function, but
You can file it here: https://issues.scala-lang.org/secure/Dashboard.jspa
Perhaps "bug" is not the right word, but "limitation". println accepts a
single argument of type Any and returns Unit, and it appears that Scala
fails to infer the correct overloaded method in this case.
def println() = C
I asked this question in Scala user group two years ago:
https://groups.google.com/forum/#!topic/scala-user/W4f0d8xK1nk
Take a look if you are interested in.
On Tue, Jul 5, 2016 at 1:31 PM, Reynold Xin wrote:
> You can file it here: https://issues.scala-lang.org/secure/Dashboard.jspa
>
> Perhap
Thanks Cody, Reynold, and Ryan! Learnt a lot and feel "corrected".
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Wed, Jul 6, 2016 at 2:46 AM, Shixiong(Ryan) Zhu
13 matches
Mail list logo
