[
https://issues.apache.org/jira/browse/SPARK-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989906#comment-13989906
]
Sean Owen commented on SPARK-1718:
----------------------------------
Yeah I may not be adding anything here. I suppose I just advise to double-check
what's being used to build, to run, and anything in between (like zip or jar).
Like, does the python-related build zip or jar anything? (I don't know that
part of the build.) That could reintroduce the problem if something outside of
Java land is not using the zip64 format.
> pyspark doesn't work with assembly jar containing over 65536 files/dirs built
> on redhat
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-1718
> URL: https://issues.apache.org/jira/browse/SPARK-1718
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
>
> Recently pyspark was ported to yarn (pr 30), but when I went to try it I
> couldn't get it work. I was building on a redhat 6 box. I figured out that
> if the assembly jar file contained over 65536 files/directories then it
> wouldn't work. If I unjarred the assembly and removed some stuff to get it
> under 65536 and jarred it back up, then it would work.
> It appears to only be an issue when building on a redhat box as I can build
> on my mac and it works just fine there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
