Sorry I forgot to mention my environment:
mesos 0.17, spark 1.4.1, scala 2.10.4, java 1.8
On Mon, Aug 24, 2015 at 10:32 AM, David Salinas <david.salinas....@gmail.com
> wrote:
> Hi Moon,
>
> Today I cannot reproduce the bug with elementary example either but it is
> still impacting all my notebooks. The weird thing is that when calling a
> transformation with map, it takes Zeppelin Context in the closure which
> gives these java.lang.NoClassDefFoundError:
> Lorg/apache/zeppelin/spark/ZeppelinContext errors (spark shell run this
> command without any problem). I will try to find another example that is
> more persistent (it is weird this example was failing yesterday). Do you
> have any idea of what could cause Zeppelin Context to be included in the
> closure?
>
> Best,
>
> David
>
>
> On Fri, Aug 21, 2015 at 6:29 PM, moon soo Lee <m...@apache.org> wrote:
>
>> I have tested your code and can not reproduce the problem.
>>
>> Could you share your environment? how did you configure Zeppelin with
>> Spark?
>>
>> Thanks,
>> moon
>>
>> On Fri, Aug 21, 2015 at 2:25 AM David Salinas <
>> david.salinas....@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a problem when using spark closure. This error was not appearing
>>> with spark 1.2.1.
>>>
>>> I have included a reproducible example that happens when taking the
>>> closure (Zeppelin has been built with head of master with this command mvn
>>> install -DskipTests -Pspark-1.4 -Dspark.version=1.4.1
>>> -Dhadoop.version=2.2.0 -Dprotobuf.version=2.5.0). Does anyone ever
>>> encountered this problem? All my previous notebooks are broken by this :(
>>>
>>> ------------------------------
>>> val textFile = sc.textFile("hdfs://somefile.txt")
>>>
>>> val f = (s: String) => s+s
>>> textFile.map(f).count
>>> //works fine
>>> //res145: Long = 407
>>>
>>>
>>> def f(s:String) = {
>>> s+s
>>> }
>>> textFile.map(f).count
>>>
>>> //fails ->
>>>
>>> org.apache.spark.SparkException: Job aborted due to stage failure: Task
>>> 566 in stage 87.0 failed 4 times, most recent failure: Lost task 566.3 in
>>> stage 87.0 (TID 43396, XXX.com): java.lang.NoClassDefFoundError:
>>> Lorg/apache/zeppelin/spark/ZeppelinContext; at
>>> java.lang.Class.getDeclaredFields0(Native Method) at
>>> java.lang.Class.privateGetDeclaredFields(Class.java:2583) at
>>> java.lang.Class.getDeclaredField(Class.java:2068) ...
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at
>>>
>>> Best,
>>>
>>> David
>>>
>>
>
