Hi Akhil
I have it working now with Groovy REPL in a form similar to the one you are
mentionning. Still I dont understand why the previous form (with the Function)
is raising that exception.
Cheers
Guillaume
On 28 July 2015 at 08:56, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Did you try it with just: (comment out line 27)
>
> println "Count of spark: " + file.filter({s -> s.contains('spark')}).
> count()
>
> Thanks
> Best Regards
>
> On Sun, Jul 26, 2015 at 12:43 AM, tog <guillaume.all...@gmail.com> wrote:
>
>> Hi
>>
>> I have been using Spark for quite some time using either scala or python.
>> I wanted to give a try to groovy through scripts for small tests.
>>
>> Unfortunately I get the following exception (using that simple script
>> <https://gist.github.com/galleon/d6540327c418aa8a479f>)
>>
>> Is there anything I am not doing correctly here.
>>
>> Thanks
>>
>> tog Groovy4Spark $ groovy GroovySparkWordcount.groovy
>>
>> class org.apache.spark.api.java.JavaRDD
>>
>> true
>>
>> true
>>
>> Caught: org.apache.spark.SparkException: Task not serializable
>>
>> org.apache.spark.SparkException: Task not serializable
>>
>> at
>> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:315)
>>
>> at
>> org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:305)
>>
>> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:132)
>>
>> at org.apache.spark.SparkContext.clean(SparkContext.scala:1893)
>>
>> at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:311)
>>
>> at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:310)
>>
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
>>
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
>>
>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:286)
>>
>> at org.apache.spark.rdd.RDD.filter(RDD.scala:310)
>>
>> at org.apache.spark.api.java.JavaRDD.filter(JavaRDD.scala:78)
>>
>> at org.apache.spark.api.java.JavaRDD$filter$0.call(Unknown Source)
>>
>> at GroovySparkWordcount.run(GroovySparkWordcount.groovy:27)
>>
>> Caused by: java.io.NotSerializableException: GroovySparkWordcount
>>
>> Serialization stack:
>>
>> - object not serializable (class: GroovySparkWordcount, value:
>> GroovySparkWordcount@7eee6c13)
>>
>> - field (class: GroovySparkWordcount$1, name: this$0, type: class
>> GroovySparkWordcount)
>>
>> - object (class GroovySparkWordcount$1, GroovySparkWordcount$1@15c16f19)
>>
>> - field (class: org.apache.spark.api.java.JavaRDD$$anonfun$filter$1,
>> name: f$1, type: interface org.apache.spark.api.java.function.Function)
>>
>> - object (class org.apache.spark.api.java.JavaRDD$$anonfun$filter$1,
>> <function1>)
>>
>> at
>> org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
>>
>> at
>> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
>>
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:81)
>>
>> at
>> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:312)
>>
>> ... 12 more
>>
>>
>> --
>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>
>
>
--
PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
