[
https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-11081:
-------------------------------
Description:
As seen from this thread
(https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E),
Spark is incompatible with Jersey 2 especially when Spark is embedded in an
application running with Jersey.
There was an in-depth discussion on options for shading and making it easier
for users to be able to use Jersey 2 with Spark applications:
https://github.com/apache/spark/pull/9615
To recap the discussion, Jersey 1 has two issues:
1. It has classes listed in META-INF/services/ files that would be loaded even
if Jersey 2 was being loaded on the classpath in a higher precedence. This
means that Jersey 2 would attempt to use Jersey 1 implementations in some
places regardless of user attempts to override the dependency with things like
userClassPathFirst.
2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to
exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is
compatible with.
Also discussed was the fact that plain old shading doesn't work here, since you
would need to shade lines in META-INF/services as well, not just classes. Not
to mention that shading JAX-RS annotations is tricky as well.
To recap the discussion as what needs to happen Spark-side, we need to:
1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is the
Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually
shade/namespace the classes that way, just the artifact name)
2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact,
say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files live
in this artifact as well)
3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
4. Spark assembly excludes META-INF/services/javax.ws.rs.*
Now a user that wants to depend on Jersey 2, and is depending on Spark maven
artifacts, would do the following in their application
1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs
dependencies
2. In my application's dependencies, exclude org.spark-project.javax.ws.rs from
spark-core
3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new
javax.ws.rs jars to the executors
was:
As seen from this thread
(https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E),
Spark is incompatible with Jersey 2 especially when Spark is embedded in an
application running with Jersey.
There was an in-depth discussion on options for shading and making it easier
for users to be able to use Jersey 2 with Spark applications:
https://github.com/apache/spark/pull/9615
To recap the discussion, Jersey 1 has two issues:
1. It has classes listed in META-INF/services/ files that would be loaded even
if Jersey 2 was being loaded on the classpath in a higher precedence. This
means that Jersey 2 would attempt to use Jersey 1 implementations in some
places regardless of user attempts to override the dependency with things like
userClassPathFirst.
2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to
exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is
compatible with.
To recap the discussion as what needs to happen Spark-side, we need to:
1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is the
Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually
shade/namespace the classes that way, just the artifact name)
2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact,
say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files live
in this artifact as well)
3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
4. Spark assembly excludes META-INF/services/javax.ws.rs.*
Now a user that wants to depend on Jersey 2, and is depending on Spark maven
artifacts, would do the following in their application
1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs
dependencies
2. In my application's dependencies, exclude org.spark-project.javax.ws.rs from
spark-core
3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new
javax.ws.rs jars to the executors
> Make spark-core pull in Jersey and javax.ws.rs dependencies separately for
> easier overriding
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-11081
> URL: https://issues.apache.org/jira/browse/SPARK-11081
> Project: Spark
> Issue Type: Improvement
> Components: Build, Spark Core
> Reporter: Mingyu Kim
>
> As seen from this thread
> (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E),
> Spark is incompatible with Jersey 2 especially when Spark is embedded in an
> application running with Jersey.
> There was an in-depth discussion on options for shading and making it easier
> for users to be able to use Jersey 2 with Spark applications:
> https://github.com/apache/spark/pull/9615
> To recap the discussion, Jersey 1 has two issues:
> 1. It has classes listed in META-INF/services/ files that would be loaded
> even if Jersey 2 was being loaded on the classpath in a higher precedence.
> This means that Jersey 2 would attempt to use Jersey 1 implementations in
> some places regardless of user attempts to override the dependency with
> things like userClassPathFirst.
> 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to
> exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is
> compatible with.
> Also discussed was the fact that plain old shading doesn't work here, since
> you would need to shade lines in META-INF/services as well, not just classes.
> Not to mention that shading JAX-RS annotations is tricky as well.
> To recap the discussion as what needs to happen Spark-side, we need to:
> 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is
> the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually
> shade/namespace the classes that way, just the artifact name)
> 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact,
> say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files
> live in this artifact as well)
> 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
> 4. Spark assembly excludes META-INF/services/javax.ws.rs.*
> Now a user that wants to depend on Jersey 2, and is depending on Spark maven
> artifacts, would do the following in their application
> 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs
> dependencies
> 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs
> from spark-core
> 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new
> javax.ws.rs jars to the executors
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
