[ 
https://issues.apache.org/jira/browse/HADOOP-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555973#comment-17555973
 ] 

Viraj Jasani commented on HADOOP-18033:
---------------------------------------

{quote}The duplicate class exception that we saw here was actually an alarm 
here that these two dependencies can't stay in peace together, but we got away 
with that by an exclude...
{quote}
I don't think it was as simple as "completely removing jsr311-api from Hadoop" 
would allow us to exclude shading javax.ws.rs-api from both 
hadoop-client-minicluster and hadoop-client-runtime. I have already tried this 
before, it doesn't work AFAIK. At least, one of them would have to keep the 
exclusion on (just like say for instance, hadoop-annotation I guess).

 

HADOOP-15983 has upgraded all com.sun.jersey dependencies (jersey-core, 
jersey-servlet etc) to the latest version and still the latest version of 
jersey-core pulls-in jsr311-api with it:

 
{code:java}
[INFO] |  +- com.sun.jersey:jersey-core:jar:1.19.4:compile
[INFO] |  |  \- javax.ws.rs:jsr311-api:jar:1.1.1:compile {code}
I don't think without exclusion (and maybe some additional code change, if 
JAX-RS 1.x and 2.x incompatibilities are in use), we might be able to get rid 
of jsr311-api. I would expect the same for Tez and other dependencies as well. 
Tez and other dependencies also can explore the similar path of excluding 
jsr311-api completely and only rely on JAX-RS 2.x based javax.ws.rs-api.

 

jsr311-api is the official spec jar for the JAX-RS 1.x line and the latest 
central release available is from Nov, 2009 (too old) 
[https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api]

whereas javax.ws.rs-api is jar for JAX-RS 2.x line 
[https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api] (latest 
version from 2018), hence we can expect more upgraded thirdparty libraries 
(just like Jackson2) having dependency on javax.ws.rs-api and less on 
jsr311-api. So all downstreamers (Tez, Hadoop, Spark) should try to get rid of 
jsr311-api anyways, totally agree here. One dependency doesn't necessarily have 
to wait for another to remove it, for instance, Tez can go ahead with exclusion 
of jsr311-api even before upgrading to Hadoop 3.3.4 because if not Hadoop, some 
of it's other dependencies (like jersey-core latest version, as mentioned 
above) would likely anyways pull it in transitively.

 

On the other hand, let Hadoop also get rid of jsr311-api. But I am pretty sure, 
removing it won't solve shading issue completely. Will come up with patch soon 
because I do recall I have already tried this as part of this Jira only. We can 
also run full build QA (all modules) and I can manually verify HDFS, MapReduce 
and ATSv2 working on pseudo-distributed mode.

> Upgrade fasterxml Jackson to 2.13.0
> -----------------------------------
>
>                 Key: HADOOP-18033
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18033
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Akira Ajisaka
>            Assignee: Viraj Jasani
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0, 3.3.2
>
>          Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest 
> as of now) or upper.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to