[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580435#comment-17580435 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran opened a new pull request, #985: URL: https://github.com/apache/parquet-mr/pull/985 Hadoop 3.3.3 moved to reload4j for logging to stop shipping a version of log4j with known (albeit unused) CVEs. This bypasses the existing exclusion code used to keep hadoop's SLF4J dependency off the classpaths, and by adding a new jar, breaks parquet-cli build. Make sure you have checked _all_ steps below. ### Jira - [X] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR" - https://issues.apache.org/jira/browse/PARQUET-XXX - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: The testing is regression testing "does the build work?", "does a test run complete without SLF4J warnings of duplicates?". done manually with `-Dhadoop.version=3.3.4` ### Commits - [X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain Javadoc that explain what it does > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580443#comment-17580443 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1217104595 i've also built against the next release of hadoop, and of 3.4.0-SNAPSHOT. the parquet build fails there as jackson 1 is purged from the hadoop classpath, breaking the japicmp plugin. ``` Execution default of goal com.github.siom79.japicmp:japicmp-maven-plugin:0.14.2:cmp failed: Could not load 'org.codehaus.jackson.type.TypeReference ``` > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582610#comment-17582610 ] ASF GitHub Bot commented on PARQUET-2173: - shangxinli commented on code in PR #985: URL: https://github.com/apache/parquet-mr/pull/985#discussion_r950886634 ## pom.xml: ## @@ -160,7 +160,11 @@ org.slf4j -slf4j-log4j12 +* Review Comment: '*' might be too broad > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583039#comment-17583039 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran commented on code in PR #985: URL: https://github.com/apache/parquet-mr/pull/985#discussion_r951589888 ## pom.xml: ## @@ -160,7 +160,11 @@ org.slf4j -slf4j-log4j12 +* Review Comment: it means that 1. classic log4j is excluded 2. slf4j-reload4j is excluded 3. when the hadoop move to log4j2 is finally shipped, its slf4j bindings will be excluded too. exclusions 1 and 2 could be done explicitly, but #3 is unclear until something ships. (hadoop trunk/3.4 is on log4j2, but the move isn't complete) > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642886#comment-17642886 ] ASF GitHub Bot commented on PARQUET-2173: - shangxinli commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1336214427 cc @ggershinsky @wgtmac let me know if you have concern to merge. > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643740#comment-17643740 ] ASF GitHub Bot commented on PARQUET-2173: - wgtmac commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1339006198 > cc @ggershinsky @wgtmac let me know if you have concern to merge. Thanks for pinging me! I don't have any concern for now. > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643762#comment-17643762 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1339052381 be good to get this in. FWIW i've been trying to build lots of things with the current smoke-build of a hadoop 3.3.5 RC. There's some aspects of maven playing up and I can't get parquet to collect the JARs from the asf staging repo, even with a profile in ~/.m2/settings.xml for it. I will probably have to add an explicit profile for that in the parquet build. > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643848#comment-17643848 ] ASF GitHub Bot commented on PARQUET-2173: - ggershinsky commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1339220517 > > cc @ggershinsky @wgtmac let me know if you have concern to merge. > > Thanks for pinging me! I don't have any concern for now. Same here > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682474#comment-17682474 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1410083313 any plans to merge now? > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682805#comment-17682805 ] ASF GitHub Bot commented on PARQUET-2173: - wgtmac commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1411336266 It looks good to me but I don't have the privilege to merge. May I request your help? @ggershinsky @shangxinli @gszadovszky > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683250#comment-17683250 ] ASF GitHub Bot commented on PARQUET-2173: - gszadovszky merged PR #985: URL: https://github.com/apache/parquet-mr/pull/985 > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2173) Fix parquet build against hadoop 3.3.3+
[ https://issues.apache.org/jira/browse/PARQUET-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683359#comment-17683359 ] ASF GitHub Bot commented on PARQUET-2173: - steveloughran commented on PR #985: URL: https://github.com/apache/parquet-mr/pull/985#issuecomment-1413592127 thanks; closed the jira > Fix parquet build against hadoop 3.3.3+ > --- > > Key: PARQUET-2173 > URL: https://issues.apache.org/jira/browse/PARQUET-2173 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > Fix For: 1.13.0 > > > parquet won't build against hadoop 3.3.3+ because it swapped out log4j 1.17 > for reload4j, and this creates maven dependency problems in parquet cli > {code} > [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ parquet-cli > --- > [WARNING] Used undeclared dependencies found: > [WARNING]ch.qos.reload4j:reload4j:jar:1.2.22:provided > {code} > the hadoop common dependencies need to exclude this jar and any changed slf4j > ones. -- This message was sent by Atlassian Jira (v8.20.10#820010)