Are you building Spark with Java 6 or Java 7. Java 6 uses the extended Zip format and Java 7 uses Zip64. I think we've tried to add some build warnings if Java 7 is used, for this reason:
https://github.com/apache/spark/blob/master/make-distribution.sh#L102 Any luck if you use JDK 6 to compile? On Mon, Jun 2, 2014 at 12:03 PM, Xu (Simon) Chen <xche...@gmail.com> wrote: > OK, my colleague found this: > https://mail.python.org/pipermail/python-list/2014-May/671353.html > > And my jar file has 70011 files. Fantastic.. > > > > > On Mon, Jun 2, 2014 at 2:34 PM, Xu (Simon) Chen <xche...@gmail.com> wrote: >> >> I asked several people, no one seems to believe that we can do this: >> $ PYTHONPATH=/path/to/assembly/jar python >> >>> import pyspark >> >> This following pull request did mention something about generating a zip >> file for all python related modules: >> https://www.mail-archive.com/reviews@spark.apache.org/msg08223.html >> >> I've tested that zipped modules can as least be imported via zipimport. >> >> Any ideas? >> >> -Simon >> >> >> >> On Mon, Jun 2, 2014 at 11:50 AM, Andrew Or <and...@databricks.com> wrote: >>> >>> Hi Simon, >>> >>> You shouldn't have to install pyspark on every worker node. In YARN mode, >>> pyspark is packaged into your assembly jar and shipped to your executors >>> automatically. This seems like a more general problem. There are a few >>> things to try: >>> >>> 1) Run a simple pyspark shell with yarn-client, and do >>> "sc.parallelize(range(10)).count()" to see if you get the same error >>> 2) If so, check if your assembly jar is compiled correctly. Run >>> >>> $ jar -tf <path/to/assembly/jar> pyspark >>> $ jar -tf <path/to/assembly/jar> py4j >>> >>> to see if the files are there. For Py4j, you need both the python files >>> and the Java class files. >>> >>> 3) If the files are there, try running a simple python shell (not pyspark >>> shell) with the assembly jar on the PYTHONPATH: >>> >>> $ PYTHONPATH=/path/to/assembly/jar python >>> >>> import pyspark >>> >>> 4) If that works, try it on every worker node. If it doesn't work, there >>> is probably something wrong with your jar. >>> >>> There is a known issue for PySpark on YARN - jars built with Java 7 >>> cannot be properly opened by Java 6. I would either verify that the >>> JAVA_HOME set on all of your workers points to Java 7 (by setting >>> SPARK_YARN_USER_ENV), or simply build your jar with Java 6: >>> >>> $ cd /path/to/spark/home >>> $ JAVA_HOME=/path/to/java6 ./make-distribution --with-yarn --hadoop >>> 2.3.0-cdh5.0.0 >>> >>> 5) You can check out >>> http://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application, >>> which has more detailed information about how to debug running an >>> application on YARN in general. In my experience, the steps outlined there >>> are quite useful. >>> >>> Let me know if you get it working (or not). >>> >>> Cheers, >>> Andrew >>> >>> >>> >>> 2014-06-02 17:24 GMT+02:00 Xu (Simon) Chen <xche...@gmail.com>: >>> >>>> Hi folks, >>>> >>>> I have a weird problem when using pyspark with yarn. I started ipython >>>> as follows: >>>> >>>> IPYTHON=1 ./pyspark --master yarn-client --executor-cores 4 >>>> --num-executors 4 --executor-memory 4G >>>> >>>> When I create a notebook, I can see workers being created and indeed I >>>> see spark UI running on my client machine on port 4040. >>>> >>>> I have the following simple script: >>>> """ >>>> import pyspark >>>> data = sc.textFile("hdfs://test/tmp/data/*").cache() >>>> oneday = data.map(lambda line: line.split(",")).\ >>>> map(lambda f: (f[0], float(f[1]))).\ >>>> filter(lambda t: t[0] >= "2013-01-01" and t[0] < >>>> "2013-01-02").\ >>>> map(lambda t: (parser.parse(t[0]), t[1])) >>>> oneday.take(1) >>>> """ >>>> >>>> By executing this, I see that it is my client machine (where ipython is >>>> launched) is reading all the data from HDFS, and produce the result of >>>> take(1), rather than my worker nodes... >>>> >>>> When I do "data.count()", things would blow up altogether. But I do see >>>> in the error message something like this: >>>> """ >>>> >>>> Error from python worker: >>>> /usr/bin/python: No module named pyspark >>>> >>>> """ >>>> >>>> >>>> Am I supposed to install pyspark on every worker node? >>>> >>>> >>>> Thanks. >>>> >>>> -Simon >>> >>> >> >
