Hi Zeppelin developers, This issue sounds very serious. Is this specific to David's use case here?
Best Regards, Jerry On Mon, Aug 24, 2015 at 1:28 PM, David Salinas <david.salinas....@gmail.com> wrote: > I have looked at the SparkInterpreter.java code and this is indeed the > issue. Whenever one uses an instruction with z.input("...") something then > no spark transformation can work as z will be shipped to the slaves where > Zeppelin is not installed as showed by the example I sent. > A workaround could be to interpret separately the variables (by defining a > map of variables before interpreting). > > Best, > > David > > > On Mon, Aug 24, 2015 at 6:45 PM, David Salinas < > david.salinas....@gmail.com> wrote: > >> Hi Moon, >> >> I found another way to reproduce the problem: >> >> //cell 1 does not work >> val file = "hdfs://someclusterfile.json" >> val s = z.input("Foo").toString >> val textFile = sc.textFile(file) >> textFile.filter(_.contains(s)).count >> //org.apache.spark.SparkException: Job aborted due to stage failure: Task >> 41 in stage 5.0 failed 4 times, most recent failure: Lost task 41.3 in >> stage 5.0 (TID 2735,XXX.com ): java.lang.NoClassDefFoundError: >> Lorg/apache/zeppelin/spark/ZeppelinContext; >> >> // cell 2 works >> val file = "hdfs://someclusterfile.json" >> val s = "Y" >> val textFile = sc.textFile(file) >> textFile.filter(_.contains(s)).count >> //res19: Long = 109 >> >> This kind of issue happens often also when using variables from other >> cells and also when taking closure for transformation. Maybe you are >> reading variables inside the transformation with something like >> "z.get("s")" which causes z to be send to the slaves as one of its member >> is used (although I also sometimes have this issue without using anything >> from other cells). >> >> Best, >> >> David >> >> >> On Mon, Aug 24, 2015 at 10:34 AM, David Salinas < >> david.salinas....@gmail.com> wrote: >> >>> Sorry I forgot to mention my environment: >>> mesos 0.17, spark 1.4.1, scala 2.10.4, java 1.8 >>> >>> On Mon, Aug 24, 2015 at 10:32 AM, David Salinas < >>> david.salinas....@gmail.com> wrote: >>> >>>> Hi Moon, >>>> >>>> Today I cannot reproduce the bug with elementary example either but it >>>> is still impacting all my notebooks. The weird thing is that when calling a >>>> transformation with map, it takes Zeppelin Context in the closure which >>>> gives these java.lang.NoClassDefFoundError: >>>> Lorg/apache/zeppelin/spark/ZeppelinContext errors (spark shell run this >>>> command without any problem). I will try to find another example that is >>>> more persistent (it is weird this example was failing yesterday). Do you >>>> have any idea of what could cause Zeppelin Context to be included in the >>>> closure? >>>> >>>> Best, >>>> >>>> David >>>> >>>> >>>> On Fri, Aug 21, 2015 at 6:29 PM, moon soo Lee <m...@apache.org> wrote: >>>> >>>>> I have tested your code and can not reproduce the problem. >>>>> >>>>> Could you share your environment? how did you configure Zeppelin with >>>>> Spark? >>>>> >>>>> Thanks, >>>>> moon >>>>> >>>>> On Fri, Aug 21, 2015 at 2:25 AM David Salinas < >>>>> david.salinas....@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a problem when using spark closure. This error was not >>>>>> appearing with spark 1.2.1. >>>>>> >>>>>> I have included a reproducible example that happens when taking the >>>>>> closure (Zeppelin has been built with head of master with this command >>>>>> mvn >>>>>> install -DskipTests -Pspark-1.4 -Dspark.version=1.4.1 >>>>>> -Dhadoop.version=2.2.0 -Dprotobuf.version=2.5.0). Does anyone ever >>>>>> encountered this problem? All my previous notebooks are broken by this :( >>>>>> >>>>>> ------------------------------ >>>>>> val textFile = sc.textFile("hdfs://somefile.txt") >>>>>> >>>>>> val f = (s: String) => s+s >>>>>> textFile.map(f).count >>>>>> //works fine >>>>>> //res145: Long = 407 >>>>>> >>>>>> >>>>>> def f(s:String) = { >>>>>> s+s >>>>>> } >>>>>> textFile.map(f).count >>>>>> >>>>>> //fails -> >>>>>> >>>>>> org.apache.spark.SparkException: Job aborted due to stage failure: >>>>>> Task 566 in stage 87.0 failed 4 times, most recent failure: Lost task >>>>>> 566.3 >>>>>> in stage 87.0 (TID 43396, XXX.com): java.lang.NoClassDefFoundError: >>>>>> Lorg/apache/zeppelin/spark/ZeppelinContext; at >>>>>> java.lang.Class.getDeclaredFields0(Native Method) at >>>>>> java.lang.Class.privateGetDeclaredFields(Class.java:2583) at >>>>>> java.lang.Class.getDeclaredField(Class.java:2068) ... >>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>>>>> at >>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>>>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at >>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) >>>>>> at >>>>>> >>>>>> Best, >>>>>> >>>>>> David >>>>>> >>>>> >>>> >>> >> >