[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502740#comment-14502740 ]
Steve Loughran commented on SPARK-7009: --------------------------------------- The problem we are seeing is pyspark related; (SPARK-1753 ?) it doesn't like Java7-built artifacts, and appears to handle that ugly hack in late java6 JVMs. Java7 is/should be the runtime to build things (even that's going to be hard to D/L soon), and as more things go to java7+ binaries (e.g. Hadoop 2.7+), the option of building on Java6 goes away. If pyspark doesn't like the java7 assembly JAR, then the options are limited # fix python's zip stuff # fix ant to replicate the java 6 hack # fix openjdk # something ugly involving a build with java7, unjar and then a rejar with java6 short term, option #4 is the only one that works; that can live outside the mvn build itself. #2 is doable; I could probably get that patch in by the end of the week, then after ant 1.9.5 ships tweak the mvn:antrun plugin to use the later dependency. openjdk is probably too hard, leaving pyspark the final fix point. > Build assembly JAR via ant to avoid zip64 problems > -------------------------------------------------- > > Key: SPARK-7009 > URL: https://issues.apache.org/jira/browse/SPARK-7009 > Project: Spark > Issue Type: Improvement > Components: Build > Affects Versions: 1.3.0 > Environment: Java 7+ > Reporter: Steve Loughran > Original Estimate: 2h > Remaining Estimate: 2h > > SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a > format incompatible with Java and pyspark. > Provided the total number of .class files+resources is <64K, ant can be used > to make the final JAR instead, perhaps by unzipping the maven-generated JAR > then rezipping it with zip64=never, before publishing the artifact via maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org