[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
[ https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042516#comment-15042516 ] Matt Cheah commented on SPARK-11081: Upgrading to Jersey 2 definitely sounds more reasonable. Perhaps we can discard this ticket and just open a "Upgrade to Jersey 2" ticket targeting Spark 2.0, what does everyone think? > Make spark-core pull in Jersey and javax.ws.rs dependencies separately for > easier overriding > > > Key: SPARK-11081 > URL: https://issues.apache.org/jira/browse/SPARK-11081 > Project: Spark > Issue Type: Improvement > Components: Build, Spark Core >Reporter: Mingyu Kim > > As seen from this thread > (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E), > Spark is incompatible with Jersey 2 especially when Spark is embedded in an > application running with Jersey. > There was an in-depth discussion on options for shading and making it easier > for users to be able to use Jersey 2 with Spark applications: > https://github.com/apache/spark/pull/9615 > To recap the discussion, Jersey 1 has two issues: > 1. It has classes listed in META-INF/services/ files that would be loaded > even if Jersey 2 was being loaded on the classpath in a higher precedence. > This means that Jersey 2 would attempt to use Jersey 1 implementations in > some places regardless of user attempts to override the dependency with > things like userClassPathFirst. > 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to > exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is > compatible with. > Also discussed was the fact that plain old shading doesn't work here, since > you would need to shade lines in META-INF/services as well, not just classes. > Not to mention that shading JAX-RS annotations is tricky as well. > To recap the discussion as what needs to happen Spark-side, we need to: > 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is > the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually > shade/namespace the classes that way, just the artifact name) > 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, > say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files > live in this artifact as well) > 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2 > 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out > these files aren't actually necessary for Jersey 1 to function properly in > general (we need to test this more however) > Now a user that wants to depend on Jersey 2, and is depending on Spark maven > artifacts, would do the following in their application > 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs > dependencies > 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs > from spark-core. We keep org.spark-project.jersey because spark-core needs > it, but it will use the javax.ws.rs classes that my application is providing. > 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new > javax.ws.rs jars to the executors -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
[ https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042519#comment-15042519 ] Marcelo Vanzin commented on SPARK-11081: Opening a new one or re-purposing this one, either is fine. > Make spark-core pull in Jersey and javax.ws.rs dependencies separately for > easier overriding > > > Key: SPARK-11081 > URL: https://issues.apache.org/jira/browse/SPARK-11081 > Project: Spark > Issue Type: Improvement > Components: Build, Spark Core >Reporter: Mingyu Kim > > As seen from this thread > (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E), > Spark is incompatible with Jersey 2 especially when Spark is embedded in an > application running with Jersey. > There was an in-depth discussion on options for shading and making it easier > for users to be able to use Jersey 2 with Spark applications: > https://github.com/apache/spark/pull/9615 > To recap the discussion, Jersey 1 has two issues: > 1. It has classes listed in META-INF/services/ files that would be loaded > even if Jersey 2 was being loaded on the classpath in a higher precedence. > This means that Jersey 2 would attempt to use Jersey 1 implementations in > some places regardless of user attempts to override the dependency with > things like userClassPathFirst. > 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to > exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is > compatible with. > Also discussed was the fact that plain old shading doesn't work here, since > you would need to shade lines in META-INF/services as well, not just classes. > Not to mention that shading JAX-RS annotations is tricky as well. > To recap the discussion as what needs to happen Spark-side, we need to: > 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is > the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually > shade/namespace the classes that way, just the artifact name) > 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, > say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files > live in this artifact as well) > 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2 > 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out > these files aren't actually necessary for Jersey 1 to function properly in > general (we need to test this more however) > Now a user that wants to depend on Jersey 2, and is depending on Spark maven > artifacts, would do the following in their application > 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs > dependencies > 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs > from spark-core. We keep org.spark-project.jersey because spark-core needs > it, but it will use the javax.ws.rs classes that my application is providing. > 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new > javax.ws.rs jars to the executors -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
[
https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042045#comment-15042045
]
Marcelo Vanzin commented on SPARK-11081:
BTW, given all the Spark 2.0 talk, if might be a good idea to just upgrade
Jersey to a newer version that doesn't have all the {{META-INF/services}}
baggage to clean up.
> Make spark-core pull in Jersey and javax.ws.rs dependencies separately for
> easier overriding
>
>
> Key: SPARK-11081
> URL: https://issues.apache.org/jira/browse/SPARK-11081
> Project: Spark
> Issue Type: Improvement
> Components: Build, Spark Core
>Reporter: Mingyu Kim
>
> As seen from this thread
> (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E),
> Spark is incompatible with Jersey 2 especially when Spark is embedded in an
> application running with Jersey.
> There was an in-depth discussion on options for shading and making it easier
> for users to be able to use Jersey 2 with Spark applications:
> https://github.com/apache/spark/pull/9615
> To recap the discussion, Jersey 1 has two issues:
> 1. It has classes listed in META-INF/services/ files that would be loaded
> even if Jersey 2 was being loaded on the classpath in a higher precedence.
> This means that Jersey 2 would attempt to use Jersey 1 implementations in
> some places regardless of user attempts to override the dependency with
> things like userClassPathFirst.
> 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to
> exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is
> compatible with.
> Also discussed was the fact that plain old shading doesn't work here, since
> you would need to shade lines in META-INF/services as well, not just classes.
> Not to mention that shading JAX-RS annotations is tricky as well.
> To recap the discussion as what needs to happen Spark-side, we need to:
> 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is
> the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually
> shade/namespace the classes that way, just the artifact name)
> 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact,
> say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files
> live in this artifact as well)
> 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
> 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out
> these files aren't actually necessary for Jersey 1 to function properly in
> general (we need to test this more however)
> Now a user that wants to depend on Jersey 2, and is depending on Spark maven
> artifacts, would do the following in their application
> 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs
> dependencies
> 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs
> from spark-core. We keep org.spark-project.jersey because spark-core needs
> it, but it will use the javax.ws.rs classes that my application is providing.
> 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new
> javax.ws.rs jars to the executors
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
[ https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042048#comment-15042048 ] Andrew Ash commented on SPARK-11081: Agreed, Spark 2.0 is the main opportunity to bump as many dependency versions as possible. Some of these constraints are imposed on Spark by its transitive dependencies, not the direct ones though. But I totally support doing a dependency inspection during the 2.0 phase. > Make spark-core pull in Jersey and javax.ws.rs dependencies separately for > easier overriding > > > Key: SPARK-11081 > URL: https://issues.apache.org/jira/browse/SPARK-11081 > Project: Spark > Issue Type: Improvement > Components: Build, Spark Core >Reporter: Mingyu Kim > > As seen from this thread > (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E), > Spark is incompatible with Jersey 2 especially when Spark is embedded in an > application running with Jersey. > There was an in-depth discussion on options for shading and making it easier > for users to be able to use Jersey 2 with Spark applications: > https://github.com/apache/spark/pull/9615 > To recap the discussion, Jersey 1 has two issues: > 1. It has classes listed in META-INF/services/ files that would be loaded > even if Jersey 2 was being loaded on the classpath in a higher precedence. > This means that Jersey 2 would attempt to use Jersey 1 implementations in > some places regardless of user attempts to override the dependency with > things like userClassPathFirst. > 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to > exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is > compatible with. > Also discussed was the fact that plain old shading doesn't work here, since > you would need to shade lines in META-INF/services as well, not just classes. > Not to mention that shading JAX-RS annotations is tricky as well. > To recap the discussion as what needs to happen Spark-side, we need to: > 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is > the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually > shade/namespace the classes that way, just the artifact name) > 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, > say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files > live in this artifact as well) > 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2 > 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out > these files aren't actually necessary for Jersey 1 to function properly in > general (we need to test this more however) > Now a user that wants to depend on Jersey 2, and is depending on Spark maven > artifacts, would do the following in their application > 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs > dependencies > 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs > from spark-core. We keep org.spark-project.jersey because spark-core needs > it, but it will use the javax.ws.rs classes that my application is providing. > 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new > javax.ws.rs jars to the executors -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
[ https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037040#comment-15037040 ] Matt Cheah commented on SPARK-11081: Quick question as I develop this as I'm fairly new to Maven - how is Spark publishing org.spark-project.akka? I presume that this is a version of Akka that we're somehow modifying and we're publishing it (for Hadoop 1 builds), and it would be helpful to use that as a starting point for publishing org.spark-project.jersey. > Make spark-core pull in Jersey and javax.ws.rs dependencies separately for > easier overriding > > > Key: SPARK-11081 > URL: https://issues.apache.org/jira/browse/SPARK-11081 > Project: Spark > Issue Type: Improvement > Components: Build, Spark Core >Reporter: Mingyu Kim > > As seen from this thread > (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E), > Spark is incompatible with Jersey 2 especially when Spark is embedded in an > application running with Jersey. > There was an in-depth discussion on options for shading and making it easier > for users to be able to use Jersey 2 with Spark applications: > https://github.com/apache/spark/pull/9615 > To recap the discussion, Jersey 1 has two issues: > 1. It has classes listed in META-INF/services/ files that would be loaded > even if Jersey 2 was being loaded on the classpath in a higher precedence. > This means that Jersey 2 would attempt to use Jersey 1 implementations in > some places regardless of user attempts to override the dependency with > things like userClassPathFirst. > 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to > exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is > compatible with. > Also discussed was the fact that plain old shading doesn't work here, since > you would need to shade lines in META-INF/services as well, not just classes. > Not to mention that shading JAX-RS annotations is tricky as well. > To recap the discussion as what needs to happen Spark-side, we need to: > 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is > the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually > shade/namespace the classes that way, just the artifact name) > 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, > say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files > live in this artifact as well) > 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2 > 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out > these files aren't actually necessary for Jersey 1 to function properly in > general (we need to test this more however) > Now a user that wants to depend on Jersey 2, and is depending on Spark maven > artifacts, would do the following in their application > 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs > dependencies > 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs > from spark-core. We keep org.spark-project.jersey because spark-core needs > it, but it will use the javax.ws.rs classes that my application is providing. > 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new > javax.ws.rs jars to the executors -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
