[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589913#comment-14589913 ]
Harry Brundage edited comment on SPARK-7009 at 6/17/15 3:22 PM: ---------------------------------------------------------------- [~sowen] I wrote a script to test that we can all run to prove if it's importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1 hadoop-2.6 artifact works fine for me on my machine. Script attached. If this is the case for others, then we're in a spot of trouble, no? was (Author: airhorns): [~sowen] I wrote a script to test that we can all run to prove if it's importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1 hadoop-2.6 artifact works fine for me on my machine. Script attached. If this is the case for other, then we're in a spot of trouble, no? > Build assembly JAR via ant to avoid zip64 problems > -------------------------------------------------- > > Key: SPARK-7009 > URL: https://issues.apache.org/jira/browse/SPARK-7009 > Project: Spark > Issue Type: Improvement > Components: Build > Affects Versions: 1.3.0 > Environment: Java 7+ > Reporter: Steve Loughran > Attachments: check_spark_python.sh > > Original Estimate: 2h > Remaining Estimate: 2h > > SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a > format incompatible with Java and pyspark. > Provided the total number of .class files+resources is <64K, ant can be used > to make the final JAR instead, perhaps by unzipping the maven-generated JAR > then rezipping it with zip64=never, before publishing the artifact via maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org