[
https://issues.apache.org/jira/browse/SPARK-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989854#comment-13989854
]
Thomas Graves commented on SPARK-1718:
--------------------------------------
I can build the jar on my mac, copy it over to the same redhat boxes I run on
and it works fine. If it was the runtime environment was using jdk6 then that
wouldn't work. I assume you are saying that if you build the jar with jdk6 and
try to run on jdk7 it has the same issue?
I also checked the MANIFEST to verify it was build with jdk7.
$ cat MANIFEST.MF
Build-Jdk: 1.7.0_25
I also went in and changed the pom.xml to use java version 1.7 as source and
target, but that doesn't look like its working as when I check the .class files
for the major version it comes back as 50 (jdk6) so perhaps this is what is
causing the issue.
It could still be possible there is something in my environment causing it but
as of yet haven't figured out what so wanted to file a jira to track the issue.
I took Andrew's comment as he also tried it and ran into the same issue but
perhaps I misunderstood.
Do you happen to have redhat box you could try it on?
> pyspark doesn't work with assembly jar containing over 65536 files/dirs built
> on redhat
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-1718
> URL: https://issues.apache.org/jira/browse/SPARK-1718
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
>
> Recently pyspark was ported to yarn (pr 30), but when I went to try it I
> couldn't get it work. I was building on a redhat 6 box. I figured out that
> if the assembly jar file contained over 65536 files/directories then it
> wouldn't work. If I unjarred the assembly and removed some stuff to get it
> under 65536 and jarred it back up, then it would work.
> It appears to only be an issue when building on a redhat box as I can build
> on my mac and it works just fine there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
