Thanks! I changed it to an object with explicit main function and that works ... But what's the problem behind this? Extending App trait prevents Spark of copying the defined values outside of the closure?
2013/8/7 Ian O'Connell <[email protected]> > do you have any function inside the object? > > following a code layout like... > https://github.com/mesos/spark/blob/master/examples/src/main/scala/spark/examples/SparkLR.scala? > > > On Wed, Aug 7, 2013 at 8:17 AM, Han JU <[email protected]> wrote: > >> Thanks first. >> >> It's a scala Object extending App. >> >> >> 2013/8/7 Ian O'Connell <[email protected]> >> >>> is your code is probably part of an object? the closure cleaner doesn't >>> attempt to pull in parts of objects. >>> >>> What does the code around your sketched section look like? >>> >>> >>> On Wed, Aug 7, 2013 at 6:47 AM, Han JU <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I'm just putting my hands on Spark and I wrote a simple job in scala. >>>> It sketches like: >>>> >>>> val TAB = "\t" >>>> >>>> val support = 2 >>>> >>>> val sc = new SparkContext(...) >>>> >>>> val raw = sc.textFile(...) >>>> >>>> val filtered = raw.map( >>>> >>>> line => { >>>> >>>> val lineSplit = line.split(TAB) // TAB is null and exception is >>>> thrown during the run >>>> ... >>>> >>>> }).filter( p => p._2 >= support) // support here is 0 during the run >>>> >>>> ... >>>> >>>> I run the sbt-assembly jar like "java -cp ..." on a standalone cluster, >>>> I found out that when referenced in the RDD transformation, the 2 values, >>>> TAB and support, are set to their default values. So TAB is null, and >>>> support is 0 and no longer "\t" and 2 as they are initialized above. >>>> >>>> If the same jar is run locally (MASTER is local or local[k] instead of >>>> spark://...) on the same input, it runs perfectly. The code also runs well >>>> in spark-shell on cluster. >>>> >>>> For the jar to run correctly on cluster, I have to hard code the string >>>> literal and the number in the RDD transformation part. >>>> >>>> It really seems to me a weird bug, maybe it has something to do with >>>> the sbt-assembly jar compilation? Some suggestions? >>>> >>>> Thanks. >>>> >>>> I'm using spark version 0.7.3 and scala 2.9.3. >>>> >>>> -- >>>> *JU Han* >>>> >>>> Software Engineer Intern @ KXEN Inc. >>>> UTC - Université de Technologie de Compiègne >>>> * **GI06 - Fouille de Données et Décisionnel* >>>> >>>> +33 0619608888 >>>> >>> >>> >> >> >> -- >> *JU Han* >> >> Software Engineer Intern @ KXEN Inc. >> UTC - Université de Technologie de Compiègne >> * **GI06 - Fouille de Données et Décisionnel* >> >> +33 0619608888 >> > > -- *JU Han* Software Engineer Intern @ KXEN Inc. UTC - Université de Technologie de Compiègne * **GI06 - Fouille de Données et Décisionnel* +33 0619608888
