[ https://issues.apache.org/jira/browse/HADOOP-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555973#comment-17555973 ]
Viraj Jasani commented on HADOOP-18033: --------------------------------------- {quote}The duplicate class exception that we saw here was actually an alarm here that these two dependencies can't stay in peace together, but we got away with that by an exclude... {quote} I don't think it was as simple as "completely removing jsr311-api from Hadoop" would allow us to exclude shading javax.ws.rs-api from both hadoop-client-minicluster and hadoop-client-runtime. I have already tried this before, it doesn't work AFAIK. At least, one of them would have to keep the exclusion on (just like say for instance, hadoop-annotation I guess). HADOOP-15983 has upgraded all com.sun.jersey dependencies (jersey-core, jersey-servlet etc) to the latest version and still the latest version of jersey-core pulls-in jsr311-api with it: {code:java} [INFO] | +- com.sun.jersey:jersey-core:jar:1.19.4:compile [INFO] | | \- javax.ws.rs:jsr311-api:jar:1.1.1:compile {code} I don't think without exclusion (and maybe some additional code change, if JAX-RS 1.x and 2.x incompatibilities are in use), we might be able to get rid of jsr311-api. I would expect the same for Tez and other dependencies as well. Tez and other dependencies also can explore the similar path of excluding jsr311-api completely and only rely on JAX-RS 2.x based javax.ws.rs-api. jsr311-api is the official spec jar for the JAX-RS 1.x line and the latest central release available is from Nov, 2009 (too old) [https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api] whereas javax.ws.rs-api is jar for JAX-RS 2.x line [https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api] (latest version from 2018), hence we can expect more upgraded thirdparty libraries (just like Jackson2) having dependency on javax.ws.rs-api and less on jsr311-api. So all downstreamers (Tez, Hadoop, Spark) should try to get rid of jsr311-api anyways, totally agree here. One dependency doesn't necessarily have to wait for another to remove it, for instance, Tez can go ahead with exclusion of jsr311-api even before upgrading to Hadoop 3.3.4 because if not Hadoop, some of it's other dependencies (like jersey-core latest version, as mentioned above) would likely anyways pull it in transitively. On the other hand, let Hadoop also get rid of jsr311-api. But I am pretty sure, removing it won't solve shading issue completely. Will come up with patch soon because I do recall I have already tried this as part of this Jira only. We can also run full build QA (all modules) and I can manually verify HDFS, MapReduce and ATSv2 working on pseudo-distributed mode. > Upgrade fasterxml Jackson to 2.13.0 > ----------------------------------- > > Key: HADOOP-18033 > URL: https://issues.apache.org/jira/browse/HADOOP-18033 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Reporter: Akira Ajisaka > Assignee: Viraj Jasani > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest > as of now) or upper. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org