[
https://issues.apache.org/jira/browse/SPARK-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989892#comment-13989892
]
Sean Owen commented on SPARK-1718:
----------------------------------
I could be mistaken here. The primary problem combination is building with 7
and running with 6. But I also thought I understood that building or running
with an older JDK 6 could be a problem too. (That, I am not 100% sure of.)
If you are running with 6, then you definitely don't want to build with 7. (The
source/target can't be set to 7 in this case; either you build with 6 and it
balks, or, you successfully build with 7 but at runtime, 6 won't accept the
bytecode.) It sounds like you are building with 7 then? but is that your Mac
build? if your RedHat build is using JDK 7, then I think this is just the same
problem as in SPARK-1520 and you should use JDK 6 to build on that machine.
(keep in mind that unzipping / rezipping, and unjarring / rejarring, might
affect the result, as it affects the format of the .jar file! Worth noting
whether that alone is causing or solving the issue.)
If you are sure you're building with 6, then my next question would be whether
it's actually building with an older JDK 6, and whether that can be upgraded
perhaps, and whether that resolves it.
Running on JDK 7 should be fine either way. I wasn't clear whether Andrew was
saying that didn't work either:
https://github.com/apache/spark/pull/30#issuecomment-42057384 But I assume
the question is how to get it running on 6.
> pyspark doesn't work with assembly jar containing over 65536 files/dirs built
> on redhat
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-1718
> URL: https://issues.apache.org/jira/browse/SPARK-1718
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
>
> Recently pyspark was ported to yarn (pr 30), but when I went to try it I
> couldn't get it work. I was building on a redhat 6 box. I figured out that
> if the assembly jar file contained over 65536 files/directories then it
> wouldn't work. If I unjarred the assembly and removed some stuff to get it
> under 65536 and jarred it back up, then it would work.
> It appears to only be an issue when building on a redhat box as I can build
> on my mac and it works just fine there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
