Thanks guys for the follow up. I'm fine with upgrading to Hadoop 3.0.0 for 1.0.0 given the explanations you have given. I'll provide further feedback about amending the patch a bit on the JIRA.
Colm. On Mon, Jan 8, 2018 at 5:03 PM, larry mccay <[email protected]> wrote: > Hi Colm - > > I just responded to a similar question on the KNOX-1161 JIRA itself. > I think we probably should have had a DISCUSS thread on this upgrade but > we can discuss here or on the JIRA. > > To me, the fact that the two releases 0.14.0 and 1.0.0 will largely be the > same feature set that are differentiated by an uptake of Hadoop 3 is really > natural. > It gives folks the opportunity to migrate to Hadoop 3 or not depending on > their comfort with doing so and not miss out on new features in Knox. > > I definitely agree on the points about the pom cleanup. > I don't believe that jackson2.version is required either - that should be > removed. > > The other cleanup tasks/questions seem valid and likely just a result of > growing needs and not realizing that there were previous action being taken > on the same dependencies - such as Kirby. > > KNOX-1161 could certainly be reopened to do something of this cleanup. > > Please let me know what you think about my assertion regarding the version > boundary of the 1.0.0 release is a natural place to do this split. > > thanks, > > --larry > > > On Mon, Jan 8, 2018 at 8:41 AM, Sandeep More <[email protected]> > wrote: > >> Hello Colm, >> >> Interesting, I think of this patch as an maintenance upgrade to Knox >> 1.0.0 >> build, this should not affect the way Knox works, infact there was just >> one >> line trivial code change in this patch. The rational behind this is to >> upgrade Hadoop dependencies that Knox uses to be in-line with the latest >> Hadoop release so we can easily leverage new features and improvements >> going forward. Also, feature wise 1.0.0 is still close to 0.14.0, this >> patch does not add, remove or improve any feature. >> >> About jackson 2.2.2 in root pom, you might be right, this could be an >> artifact of my testing, let me try removing it again and see. >> >> The Kerby version rolled with Hadoop 3 has some significant api changes, >> which if included would likely have us revisit the SecureClusterTest code. >> So the patch uses the version of Kerby that does not need code changes to >> our test classes. My aim was to keep Knox code changes as minimum as >> possible. >> >> "Beanutils is downgraded from 1.9.3 to 1.9.2." >> On the contrary Beanutils was upgraded from 1.9.2 to 1.9.3 [1] >> >> - <version>1.9.2</version> >> + <version>${commons-beanutils-version}</version> >> >> >> "Also we are defining lower versions of Jackson and Mockito in the >> gateway-test-release pom." >> >> Do you mean lower version than before ? I do recall I had trouble with >> higher version of jackson with our tests. Through trial and error I found >> these were the versions that played nice with our tests. >> >> I decided to stick with these versions since I did not think changing the >> Unit tests was a good idea at this point. >> >> Thanks for taking a look at the patch and let me know your thoughts. >> >> Best, >> Sandeep >> >> [1] >> https://git1-us-west.apache.org/repos/asf?p=knox.git;a=blob; >> f=pom.xml;h=6bd93962f30fb52fb2273ba5294bae3140eb9119;hb=772bc33d#l125 >> >> >> >> On Mon, Jan 8, 2018 at 4:52 AM, Colm O hEigeartaigh <[email protected]> >> wrote: >> >> > Hi Sandeep, >> > >> > I'm wondering if this (major) upgrade is not breaking the idea of >> trying to >> > change as little as possible for 1.0.0 from 0.14.0? >> > >> > I think "jackson2.version" added to the root pom with version 2.2.2 >> could >> > be removed, as it is not used anywhere. Why do we exclude Kerby from the >> > Hadoop pom and then include it as a dependency again? >> > >> > Also, is it really necessary to "downgrade" various dependencies just >> > because they are used by Hadoop? Beanutils is downgraded from 1.9.3 to >> > 1.9.2. Also we are defining lower versions of Jackson and Mockito in the >> > gateway-test-release pom. >> > >> > Colm. >> > >> > On Fri, Jan 5, 2018 at 7:40 PM, <[email protected]> wrote: >> > >> > > Repository: knox >> > > Updated Branches: >> > > refs/heads/master 6d4756f3d -> 772bc33d4 >> > > >> > > >> > > KNOX-1161 - Update hadoop dependencies to hadoop 3 >> > > >> > > >> > > Project: http://git-wip-us.apache.org/repos/asf/knox/repo >> > > Commit: http://git-wip-us.apache.org/repos/asf/knox/commit/772bc33d >> > > Tree: http://git-wip-us.apache.org/repos/asf/knox/tree/772bc33d >> > > Diff: http://git-wip-us.apache.org/repos/asf/knox/diff/772bc33d >> > > >> > > Branch: refs/heads/master >> > > Commit: 772bc33d48dd37be5cd098992df3e50db24326f3 >> > > Parents: 6d4756f >> > > Author: Sandeep More <[email protected]> >> > > Authored: Fri Jan 5 14:38:24 2018 -0500 >> > > Committer: Sandeep More <[email protected]> >> > > Committed: Fri Jan 5 14:38:24 2018 -0500 >> > > >> > > ------------------------------------------------------------ >> ---------- >> > > .../filter/PortMappingHelperHandler.java | 2 +- >> > > gateway-test-release/pom.xml | 77 >> > ++++++++++++++++++++ >> > > pom.xml | 31 +++++++- >> > > 3 files changed, 107 insertions(+), 3 deletions(-) >> > > ------------------------------------------------------------ >> ---------- >> > > >> > > >> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/ >> > > gateway-server/src/main/java/org/apache/hadoop/gateway/filter/ >> > > PortMappingHelperHandler.java >> > > ------------------------------------------------------------ >> ---------- >> > > diff --git a/gateway-server/src/main/java/org/apache/hadoop/ >> > > gateway/filter/PortMappingHelperHandler.java >> b/gateway-server/src/main/ >> > > java/org/apache/hadoop/gateway/filter/PortMappingHelperHandler.java >> > > index ea3efc4..06d9668 100644 >> > > --- a/gateway-server/src/main/java/org/apache/hadoop/gateway/filter/ >> > > PortMappingHelperHandler.java >> > > +++ b/gateway-server/src/main/java/org/apache/hadoop/gateway/filter/ >> > > PortMappingHelperHandler.java >> > > @@ -96,7 +96,7 @@ public class PortMappingHelperHandler extends >> > > HandlerWrapper { >> > > throws IOException, ServletException { >> > > >> > > String newTarget = target; >> > > - String baseURI = baseRequest.getUri().toString(); >> > > + String baseURI = baseRequest.getRequestURI(); >> > > >> > > // If Port Mapping feature enabled >> > > if (config.isGatewayPortMappingEnabled()) { >> > > >> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/ >> > > gateway-test-release/pom.xml >> > > ------------------------------------------------------------ >> ---------- >> > > diff --git a/gateway-test-release/pom.xml >> b/gateway-test-release/pom.xml >> > > index e61e0c8..48def02 100644 >> > > --- a/gateway-test-release/pom.xml >> > > +++ b/gateway-test-release/pom.xml >> > > @@ -34,7 +34,62 @@ >> > > <module>webhdfs-test</module> >> > > </modules> >> > > >> > > + <properties> >> > > + <jetty.version>9.3.19.v20170502</jetty.version> >> > > + <mockito.version>1.8.4</mockito.version> >> > > + <jackson2.version>2.7.8</jackson2.version> >> > > + </properties> >> > > + >> > > + >> > > <dependencies> >> > > + <!-- Hadoop 3.0 Deps. --> >> > > + <dependency> >> > > + <groupId>com.fasterxml.jackson.core</groupId> >> > > + <artifactId>jackson-databind</artifactId> >> > > + <version>${jackson2.version}</version> >> > > + </dependency> >> > > + >> > > + <dependency> >> > > + <groupId>org.mockito</groupId> >> > > + <artifactId>mockito-all</artifactId> >> > > + <version>${mockito.version}</version> >> > > + <scope>test</scope> >> > > + </dependency> >> > > + >> > > + <dependency> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>jetty-server</artifactId> >> > > + <version>${jetty.version}</version> >> > > + <exclusions> >> > > + <exclusion> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>javax.servlet-api</artifactId> >> > > + </exclusion> >> > > + </exclusions> >> > > + </dependency> >> > > + <dependency> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>jetty-util</artifactId> >> > > + <version>${jetty.version}</version> >> > > + </dependency> >> > > + <dependency> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>jetty-servlet</artifactId> >> > > + <version>${jetty.version}</version> >> > > + </dependency> >> > > + <dependency> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>jetty-webapp</artifactId> >> > > + <version>${jetty.version}</version> >> > > + </dependency> >> > > + <dependency> >> > > + <groupId>org.eclipse.jetty</groupId> >> > > + <artifactId>jetty-util-ajax</artifactId> >> > > + <version>${jetty.version}</version> >> > > + </dependency> >> > > + >> > > + >> > > + >> > > <dependency> >> > > <groupId>javax.servlet</groupId> >> > > <artifactId>javax.servlet-api</artifactId> >> > > @@ -118,10 +173,27 @@ >> > > <groupId>org.apache.directory.server</groupId> >> > > <artifactId>apacheds-all</artifactId> >> > > </exclusion> >> > > + >> > > + <exclusion> >> > > + <groupId>commons-configuration</groupId> >> > > + <artifactId>commons-configuration</artifactId> >> > > + </exclusion> >> > > + >> > > + <exclusion> >> > > + <groupId>com.fasterxml.jackson.core</groupId> >> > > + <artifactId>jackson-databind</artifactId> >> > > + </exclusion> >> > > + >> > > </exclusions> >> > > </dependency> >> > > >> > > <dependency> >> > > + <groupId>commons-configuration</groupId> >> > > + <artifactId>commons-configuration</artifactId> >> > > + <version>1.10</version> >> > > + </dependency> >> > > + >> > > + <dependency> >> > > <groupId>org.hamcrest</groupId> >> > > <artifactId>hamcrest-library</artifactId> >> > > <scope>test</scope> >> > > @@ -152,6 +224,11 @@ >> > > </dependency> >> > > >> > > <dependency> >> > > + <groupId>org.apache.kerby</groupId> >> > > + <artifactId>kerb-simplekdc</artifactId> >> > > + </dependency> >> > > + >> > > + <dependency> >> > > <groupId>junit</groupId> >> > > <artifactId>junit</artifactId> >> > > <scope>test</scope> >> > > >> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/pom.xml >> > > ------------------------------------------------------------ >> ---------- >> > > diff --git a/pom.xml b/pom.xml >> > > index fd7f62b..6bd9396 100644 >> > > --- a/pom.xml >> > > +++ b/pom.xml >> > > @@ -112,7 +112,7 @@ >> > > <gateway-version>1.0.0-SNAPSHOT</gateway-version> >> > > <gateway-group>org.apache.knox</gateway-group> >> > > <groovy-version>2.4.6</groovy-version> >> > > - <hadoop-version>2.7.3</hadoop-version> >> > > + <hadoop-version>3.0.0</hadoop-version> >> > > <jackson.version>2.8.10</jackson.version> >> > > <jetty-version>9.2.15.v20160210</jetty-version> >> > > <surefire-version>2.16</surefire-version> >> > > @@ -121,6 +121,9 @@ >> > > <javax-websocket-version>1.1</javax-websocket-version> >> > > <metrics-version>3.1.2</metrics-version> >> > > <shiro.version>1.2.6</shiro.version> >> > > + <kerb-simplekdc-version>1.0.0-RC2</kerb-simplekdc-version> >> > > + <commons-beanutils-version>1.9.3</commons-beanutils-version> >> > > + <jackson2.version>2.2.2</jackson2.version> >> > > </properties> >> > > >> > > <licenses> >> > > @@ -1015,10 +1018,34 @@ >> > > <groupId>xmlenc</groupId> >> > > <artifactId>xmlenc</artifactId> >> > > </exclusion> >> > > + >> > > + <exclusion> >> > > + <groupId>com.sun.jersey</groupId> >> > > + <artifactId>jersey-servlet</artifactId> >> > > + </exclusion> >> > > + >> > > + <exclusion> >> > > + <groupId>org.apache.kerby</groupId> >> > > + <artifactId>kerb-simplekdc</artifactId> >> > > + </exclusion> >> > > + >> > > + <!-- >> > > + <exclusion> >> > > + <groupId>com.fasterxml.jackson >> .core</groupId> >> > > + <artifactId>jackson-databind</artifactId> >> > > + </exclusion> >> > > + --> >> > > + >> > > </exclusions> >> > > </dependency> >> > > >> > > <dependency> >> > > + <groupId>org.apache.kerby</groupId> >> > > + <artifactId>kerb-simplekdc</artifactId> >> > > + <version>${kerb-simplekdc-version}</version> >> > > + </dependency> >> > > + >> > > + <dependency> >> > > <groupId>com.fasterxml.jackson.core</groupId> >> > > <artifactId>jackson-databind</artifactId> >> > > <version>${jackson.version}</version> >> > > @@ -1057,7 +1084,7 @@ >> > > <dependency> >> > > <groupId>commons-beanutils</groupId> >> > > <artifactId>commons-beanutils</artifactId> >> > > - <version>1.9.2</version> >> > > + <version>${commons-beanutils-version}</version> >> > > </dependency> >> > > <dependency> >> > > <groupId>org.apache.commons</groupId> >> > > >> > > >> > >> > >> > -- >> > Colm O hEigeartaigh >> > >> > Talend Community Coder >> > http://coders.talend.com >> > >> > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
