[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893074#comment-15893074
]
Andrew Otto commented on SPARK-1693:
We just upgraded to CDH 5.10, which has Spark 1.6.0, Hadoop 2.6.0, Hive 1.1.0,
and Oozie 4.1.0.
We are having trouble running Spark jobs that use HiveContext from Oozie. They
run perfectly fine from the CLI with spark-submit, just not in Oozie. We
aren't certain that HiveContext is related, but we can reproduce regularly with
a job that uses HiveContext.
Anyway, I post this here, because the error we are getting is the same that
started this issue:
{code}class "javax.servlet.FilterRegistration"'s signer information does not
match signer information of other classes in the same package{code}
I've noticed that the Oozie sharelib includes
javax.servlet-3.0.0.v201112011016.jar. I also see that spark-assembly.jar
includes a javax.servlet.FilterRegistration class, although its hard for me to
tell which version. The jetty pom.xml files in spark-assembly.jar seem to say
{{javax.servlet.*;version="2.6.0"}}, but I'm a little green on how all these
dependencies get resolved. I don't see any javax.servlet .jars in any of
/usr/lib/hadoop* (where CDH installs hadoop jars).
Help! :) If this is not related to this issue, I'll open a new one.
> Dependent on multiple versions of servlet-api jars lead to throw an
> SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
>
>
> Key: SPARK-1693
> URL: https://issues.apache.org/jira/browse/SPARK-1693
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Guoqiang Li
>Assignee: Guoqiang Li
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: log.txt
>
>
> {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 >
> log.txt{code}
> The log:
> {code}
> UnpersistSuite:
> - unpersist RDD *** FAILED ***
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s
> signer information does not match signer information of other classes in the
> same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089936#comment-15089936
]
Owen O'Malley commented on SPARK-1693:
--
Can you explain what the problem is and how to fix it? We are hitting the same
problem on the hive on spark work.
> Dependent on multiple versions of servlet-api jars lead to throw an
> SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
>
>
> Key: SPARK-1693
> URL: https://issues.apache.org/jira/browse/SPARK-1693
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Guoqiang Li
>Assignee: Guoqiang Li
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: log.txt
>
>
> {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 >
> log.txt{code}
> The log:
> {code}
> UnpersistSuite:
> - unpersist RDD *** FAILED ***
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s
> signer information does not match signer information of other classes in the
> same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089945#comment-15089945
]
Sean Owen commented on SPARK-1693:
--
As I recall this happens because the official JavaEE servlet API jar includes
MANIFEST.MF entries with a signature. If these classes are repackaged in an
assembly jar with these same MANIFEST.MF entries, you get this failure. There's
a lot of discussion above, but one quick fix is to manage to not include the
signatures from anywhere.
> Dependent on multiple versions of servlet-api jars lead to throw an
> SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
>
>
> Key: SPARK-1693
> URL: https://issues.apache.org/jira/browse/SPARK-1693
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Guoqiang Li
>Assignee: Guoqiang Li
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: log.txt
>
>
> {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 >
> log.txt{code}
> The log:
> {code}
> UnpersistSuite:
> - unpersist RDD *** FAILED ***
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s
> signer information does not match signer information of other classes in the
> same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14728669#comment-14728669
]
Hudson commented on SPARK-1693:
---
SUCCESS: Integrated in gora-trunk #1610 (See
[https://builds.apache.org/job/gora-trunk/1610/])
servlet dependency is excluded from spark, not ant. Without excluding it throws
a java security exception. Motivation for it:
https://issues.apache.org/jira/browse/SPARK-1693 and (furkankamaci: rev
ad4f5601ebd52e182bf79ba4032dcb85d918b925)
* gora-core/pom.xml
* pom.xml
> Dependent on multiple versions of servlet-api jars lead to throw an
> SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
>
>
> Key: SPARK-1693
> URL: https://issues.apache.org/jira/browse/SPARK-1693
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Guoqiang Li
>Assignee: Guoqiang Li
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: log.txt
>
>
> {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 >
> log.txt{code}
> The log:
> {code}
> UnpersistSuite:
> - unpersist RDD *** FAILED ***
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s
> signer information does not match signer information of other classes in the
> same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562555#comment-14562555
]
Michael Schmeißer commented on SPARK-1693:
--
I think the fix version of this issue is incorrect, I could reproduce the
problem with spark-core 1.1.0-cdh5.2.1. With version 1.3.0-cdh5.4.2, the same
code works fine in local execution.
Dependent on multiple versions of servlet-api jars lead to throw an
SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
Key: SPARK-1693
URL: https://issues.apache.org/jira/browse/SPARK-1693
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Priority: Blocker
Fix For: 1.0.0
Attachments: log.txt
{code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0
log.txt{code}
The log:
{code}
UnpersistSuite:
- unpersist RDD *** FAILED ***
java.lang.SecurityException: class javax.servlet.FilterRegistration's
signer information does not match signer information of other classes in the
same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988618#comment-13988618
]
Guoqiang Li commented on SPARK-1693:
[~srowen] You're right,preferably only one version exists.
Dependent on multiple versions of servlet-api jars lead to throw an
SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
Key: SPARK-1693
URL: https://issues.apache.org/jira/browse/SPARK-1693
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Priority: Blocker
Fix For: 1.0.0
Attachments: log.txt
{code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0
log.txt{code}
The log:
{code}
UnpersistSuite:
- unpersist RDD *** FAILED ***
java.lang.SecurityException: class javax.servlet.FilterRegistration's
signer information does not match signer information of other classes in the
same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986656#comment-13986656
]
Guoqiang Li commented on SPARK-1693:
Thanks Sean Owen .
Code has been submitted to [PR 590|https://github.com/apache/spark/pull/590]
Dependent on multiple versions of servlet-api jars lead to throw an
SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
Key: SPARK-1693
URL: https://issues.apache.org/jira/browse/SPARK-1693
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Attachments: log.txt
{code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0
log.txt{code}
The log:
{code}
UnpersistSuite:
- unpersist RDD *** FAILED ***
java.lang.SecurityException: class javax.servlet.FilterRegistration's
signer information does not match signer information of other classes in the
same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987316#comment-13987316
]
Patrick Wendell commented on SPARK-1693:
I'm confused about something [~sowen]: If Hadoop 2.3/2.4 wants version 3 of the
servlet-api dependency, why does the Hadoop build declare version 2.5 as a
dependency in their POM's?
http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.3.0/hadoop-project-2.3.0.pom
http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.4.0/hadoop-project-2.4.0.pom
Also - is there not a risk of bumping this version to 3.X when users are
building against older hadoop versions? I noticed the older hadoop versions
depend 2.X versions of this... will it break them if 2.X and 3.X are present at
runtime?
[~sowen] If you could look over this PR it would be great. It's really scary
merging a big change like this to the build at the 11th hour. So the more eyes
the better.
Dependent on multiple versions of servlet-api jars lead to throw an
SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
Key: SPARK-1693
URL: https://issues.apache.org/jira/browse/SPARK-1693
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Attachments: log.txt
{code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0
log.txt{code}
The log:
{code}
UnpersistSuite:
- unpersist RDD *** FAILED ***
java.lang.SecurityException: class javax.servlet.FilterRegistration's
signer information does not match signer information of other classes in the
same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987323#comment-13987323
]
Guoqiang Li commented on SPARK-1693:
Now, the main problem is to ensure that hadoop 2.3, 2.4 can work in this
version.But I don't have Hadoop 2.3, 2.4 clusters.
Dependent on multiple versions of servlet-api jars lead to throw an
SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
Key: SPARK-1693
URL: https://issues.apache.org/jira/browse/SPARK-1693
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Priority: Blocker
Fix For: 1.0.0
Attachments: log.txt
{code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0
log.txt{code}
The log:
{code}
UnpersistSuite:
- unpersist RDD *** FAILED ***
java.lang.SecurityException: class javax.servlet.FilterRegistration's
signer information does not match signer information of other classes in the
same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
