Hi Colm -
I just responded to a similar question on the KNOX-1161 JIRA itself.
I think we probably should have had a DISCUSS thread on this upgrade but we
can discuss here or on the JIRA.
To me, the fact that the two releases 0.14.0 and 1.0.0 will largely be the
same feature set that are differentiated by an uptake of Hadoop 3 is really
natural.
It gives folks the opportunity to migrate to Hadoop 3 or not depending on
their comfort with doing so and not miss out on new features in Knox.
I definitely agree on the points about the pom cleanup.
I don't believe that jackson2.version is required either - that should be
removed.
The other cleanup tasks/questions seem valid and likely just a result of
growing needs and not realizing that there were previous action being taken
on the same dependencies - such as Kirby.
KNOX-1161 could certainly be reopened to do something of this cleanup.
Please let me know what you think about my assertion regarding the version
boundary of the 1.0.0 release is a natural place to do this split.
thanks,
--larry
On Mon, Jan 8, 2018 at 8:41 AM, Sandeep More <moresand...@gmail.com> wrote:
> Hello Colm,
>
> Interesting, I think of this patch as an maintenance upgrade to Knox 1.0.0
> build, this should not affect the way Knox works, infact there was just one
> line trivial code change in this patch. The rational behind this is to
> upgrade Hadoop dependencies that Knox uses to be in-line with the latest
> Hadoop release so we can easily leverage new features and improvements
> going forward. Also, feature wise 1.0.0 is still close to 0.14.0, this
> patch does not add, remove or improve any feature.
>
> About jackson 2.2.2 in root pom, you might be right, this could be an
> artifact of my testing, let me try removing it again and see.
>
> The Kerby version rolled with Hadoop 3 has some significant api changes,
> which if included would likely have us revisit the SecureClusterTest code.
> So the patch uses the version of Kerby that does not need code changes to
> our test classes. My aim was to keep Knox code changes as minimum as
> possible.
>
> "Beanutils is downgraded from 1.9.3 to 1.9.2."
> On the contrary Beanutils was upgraded from 1.9.2 to 1.9.3 [1]
>
> - <version>1.9.2</version>
> + <version>${commons-beanutils-version}</version>
>
>
> "Also we are defining lower versions of Jackson and Mockito in the
> gateway-test-release pom."
>
> Do you mean lower version than before ? I do recall I had trouble with
> higher version of jackson with our tests. Through trial and error I found
> these were the versions that played nice with our tests.
>
> I decided to stick with these versions since I did not think changing the
> Unit tests was a good idea at this point.
>
> Thanks for taking a look at the patch and let me know your thoughts.
>
> Best,
> Sandeep
>
> [1]
> https://git1-us-west.apache.org/repos/asf?p=knox.git;a=blob;f=pom.xml;h=
> 6bd93962f30fb52fb2273ba5294bae3140eb9119;hb=772bc33d#l125
>
>
>
> On Mon, Jan 8, 2018 at 4:52 AM, Colm O hEigeartaigh <cohei...@apache.org>
> wrote:
>
> > Hi Sandeep,
> >
> > I'm wondering if this (major) upgrade is not breaking the idea of trying
> to
> > change as little as possible for 1.0.0 from 0.14.0?
> >
> > I think "jackson2.version" added to the root pom with version 2.2.2 could
> > be removed, as it is not used anywhere. Why do we exclude Kerby from the
> > Hadoop pom and then include it as a dependency again?
> >
> > Also, is it really necessary to "downgrade" various dependencies just
> > because they are used by Hadoop? Beanutils is downgraded from 1.9.3 to
> > 1.9.2. Also we are defining lower versions of Jackson and Mockito in the
> > gateway-test-release pom.
> >
> > Colm.
> >
> > On Fri, Jan 5, 2018 at 7:40 PM, <m...@apache.org> wrote:
> >
> > > Repository: knox
> > > Updated Branches:
> > > refs/heads/master 6d4756f3d -> 772bc33d4
> > >
> > >
> > > KNOX-1161 - Update hadoop dependencies to hadoop 3
> > >
> > >
> > > Project: http://git-wip-us.apache.org/repos/asf/knox/repo
> > > Commit: http://git-wip-us.apache.org/repos/asf/knox/commit/772bc33d
> > > Tree: http://git-wip-us.apache.org/repos/asf/knox/tree/772bc33d
> > > Diff: http://git-wip-us.apache.org/repos/asf/knox/diff/772bc33d
> > >
> > > Branch: refs/heads/master
> > > Commit: 772bc33d48dd37be5cd098992df3e50db24326f3
> > > Parents: 6d4756f
> > > Author: Sandeep More <m...@apache.org>
> > > Authored: Fri Jan 5 14:38:24 2018 -0500
> > > Committer: Sandeep More <m...@apache.org>
> > > Committed: Fri Jan 5 14:38:24 2018 -0500
> > >
> > > ----------------------------------------------------------------------
> > > .../filter/PortMappingHelperHandler.java | 2 +-
> > > gateway-test-release/pom.xml | 77
> > ++++++++++++++++++++
> > > pom.xml | 31 +++++++-
> > > 3 files changed, 107 insertions(+), 3 deletions(-)
> > > ----------------------------------------------------------------------
> > >
> > >
> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/
> > > gateway-server/src/main/java/org/apache/hadoop/gateway/filter/
> > > PortMappingHelperHandler.java
> > > ----------------------------------------------------------------------
> > > diff --git a/gateway-server/src/main/java/org/apache/hadoop/
> > > gateway/filter/PortMappingHelperHandler.java
> b/gateway-server/src/main/
> > > java/org/apache/hadoop/gateway/filter/PortMappingHelperHandler.java
> > > index ea3efc4..06d9668 100644
> > > --- a/gateway-server/src/main/java/org/apache/hadoop/gateway/filter/
> > > PortMappingHelperHandler.java
> > > +++ b/gateway-server/src/main/java/org/apache/hadoop/gateway/filter/
> > > PortMappingHelperHandler.java
> > > @@ -96,7 +96,7 @@ public class PortMappingHelperHandler extends
> > > HandlerWrapper {
> > > throws IOException, ServletException {
> > >
> > > String newTarget = target;
> > > - String baseURI = baseRequest.getUri().toString();
> > > + String baseURI = baseRequest.getRequestURI();
> > >
> > > // If Port Mapping feature enabled
> > > if (config.isGatewayPortMappingEnabled()) {
> > >
> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/
> > > gateway-test-release/pom.xml
> > > ----------------------------------------------------------------------
> > > diff --git a/gateway-test-release/pom.xml
> b/gateway-test-release/pom.xml
> > > index e61e0c8..48def02 100644
> > > --- a/gateway-test-release/pom.xml
> > > +++ b/gateway-test-release/pom.xml
> > > @@ -34,7 +34,62 @@
> > > <module>webhdfs-test</module>
> > > </modules>
> > >
> > > + <properties>
> > > + <jetty.version>9.3.19.v20170502</jetty.version>
> > > + <mockito.version>1.8.4</mockito.version>
> > > + <jackson2.version>2.7.8</jackson2.version>
> > > + </properties>
> > > +
> > > +
> > > <dependencies>
> > > + <!-- Hadoop 3.0 Deps. -->
> > > + <dependency>
> > > + <groupId>com.fasterxml.jackson.core</groupId>
> > > + <artifactId>jackson-databind</artifactId>
> > > + <version>${jackson2.version}</version>
> > > + </dependency>
> > > +
> > > + <dependency>
> > > + <groupId>org.mockito</groupId>
> > > + <artifactId>mockito-all</artifactId>
> > > + <version>${mockito.version}</version>
> > > + <scope>test</scope>
> > > + </dependency>
> > > +
> > > + <dependency>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>jetty-server</artifactId>
> > > + <version>${jetty.version}</version>
> > > + <exclusions>
> > > + <exclusion>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>javax.servlet-api</artifactId>
> > > + </exclusion>
> > > + </exclusions>
> > > + </dependency>
> > > + <dependency>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>jetty-util</artifactId>
> > > + <version>${jetty.version}</version>
> > > + </dependency>
> > > + <dependency>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>jetty-servlet</artifactId>
> > > + <version>${jetty.version}</version>
> > > + </dependency>
> > > + <dependency>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>jetty-webapp</artifactId>
> > > + <version>${jetty.version}</version>
> > > + </dependency>
> > > + <dependency>
> > > + <groupId>org.eclipse.jetty</groupId>
> > > + <artifactId>jetty-util-ajax</artifactId>
> > > + <version>${jetty.version}</version>
> > > + </dependency>
> > > +
> > > +
> > > +
> > > <dependency>
> > > <groupId>javax.servlet</groupId>
> > > <artifactId>javax.servlet-api</artifactId>
> > > @@ -118,10 +173,27 @@
> > > <groupId>org.apache.directory.server</groupId>
> > > <artifactId>apacheds-all</artifactId>
> > > </exclusion>
> > > +
> > > + <exclusion>
> > > + <groupId>commons-configuration</groupId>
> > > + <artifactId>commons-configuration</artifactId>
> > > + </exclusion>
> > > +
> > > + <exclusion>
> > > + <groupId>com.fasterxml.jackson.core</groupId>
> > > + <artifactId>jackson-databind</artifactId>
> > > + </exclusion>
> > > +
> > > </exclusions>
> > > </dependency>
> > >
> > > <dependency>
> > > + <groupId>commons-configuration</groupId>
> > > + <artifactId>commons-configuration</artifactId>
> > > + <version>1.10</version>
> > > + </dependency>
> > > +
> > > + <dependency>
> > > <groupId>org.hamcrest</groupId>
> > > <artifactId>hamcrest-library</artifactId>
> > > <scope>test</scope>
> > > @@ -152,6 +224,11 @@
> > > </dependency>
> > >
> > > <dependency>
> > > + <groupId>org.apache.kerby</groupId>
> > > + <artifactId>kerb-simplekdc</artifactId>
> > > + </dependency>
> > > +
> > > + <dependency>
> > > <groupId>junit</groupId>
> > > <artifactId>junit</artifactId>
> > > <scope>test</scope>
> > >
> > > http://git-wip-us.apache.org/repos/asf/knox/blob/772bc33d/pom.xml
> > > ----------------------------------------------------------------------
> > > diff --git a/pom.xml b/pom.xml
> > > index fd7f62b..6bd9396 100644
> > > --- a/pom.xml
> > > +++ b/pom.xml
> > > @@ -112,7 +112,7 @@
> > > <gateway-version>1.0.0-SNAPSHOT</gateway-version>
> > > <gateway-group>org.apache.knox</gateway-group>
> > > <groovy-version>2.4.6</groovy-version>
> > > - <hadoop-version>2.7.3</hadoop-version>
> > > + <hadoop-version>3.0.0</hadoop-version>
> > > <jackson.version>2.8.10</jackson.version>
> > > <jetty-version>9.2.15.v20160210</jetty-version>
> > > <surefire-version>2.16</surefire-version>
> > > @@ -121,6 +121,9 @@
> > > <javax-websocket-version>1.1</javax-websocket-version>
> > > <metrics-version>3.1.2</metrics-version>
> > > <shiro.version>1.2.6</shiro.version>
> > > + <kerb-simplekdc-version>1.0.0-RC2</kerb-simplekdc-version>
> > > + <commons-beanutils-version>1.9.3</commons-beanutils-version>
> > > + <jackson2.version>2.2.2</jackson2.version>
> > > </properties>
> > >
> > > <licenses>
> > > @@ -1015,10 +1018,34 @@
> > > <groupId>xmlenc</groupId>
> > > <artifactId>xmlenc</artifactId>
> > > </exclusion>
> > > +
> > > + <exclusion>
> > > + <groupId>com.sun.jersey</groupId>
> > > + <artifactId>jersey-servlet</artifactId>
> > > + </exclusion>
> > > +
> > > + <exclusion>
> > > + <groupId>org.apache.kerby</groupId>
> > > + <artifactId>kerb-simplekdc</artifactId>
> > > + </exclusion>
> > > +
> > > + <!--
> > > + <exclusion>
> > > + <groupId>com.fasterxml.jackson.core</groupId>
> > > + <artifactId>jackson-databind</artifactId>
> > > + </exclusion>
> > > + -->
> > > +
> > > </exclusions>
> > > </dependency>
> > >
> > > <dependency>
> > > + <groupId>org.apache.kerby</groupId>
> > > + <artifactId>kerb-simplekdc</artifactId>
> > > + <version>${kerb-simplekdc-version}</version>
> > > + </dependency>
> > > +
> > > + <dependency>
> > > <groupId>com.fasterxml.jackson.core</groupId>
> > > <artifactId>jackson-databind</artifactId>
> > > <version>${jackson.version}</version>
> > > @@ -1057,7 +1084,7 @@
> > > <dependency>
> > > <groupId>commons-beanutils</groupId>
> > > <artifactId>commons-beanutils</artifactId>
> > > - <version>1.9.2</version>
> > > + <version>${commons-beanutils-version}</version>
> > > </dependency>
> > > <dependency>
> > > <groupId>org.apache.commons</groupId>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
