[jira] [Resolved] (HADOOP-19170) Fixes compilation issues on Mac
[
https://issues.apache.org/jira/browse/HADOOP-19170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-19170.
--
Fix Version/s: 3.5.0
Resolution: Fixed
> Fixes compilation issues on Mac
> ---
>
> Key: HADOOP-19170
> URL: https://issues.apache.org/jira/browse/HADOOP-19170
> Project: Hadoop Common
> Issue Type: Bug
> Environment: OS: macOS Catalina 10.15.7
> compiler: clang 12.0.0
> cmake: 3.24.0
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> When I build hadoop-common native in Mac OS, I found this error:
> {code:java}
> /x/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:114:50:
> error: function-like macro '__GLIBC_PREREQ' is not defined
> #if defined(__sun) || defined(__GLIBC_PREREQ) && __GLIBC_PREREQ(2, 32) {code}
> The reason is that Mac OS does not support glibc. And C conditional
> compilation requires validation of all expressions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17647) Release Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17647. -- Assignee: Wei-Chiu Chuang Resolution: Done The release was published on June 15 2021. > Release Hadoop 3.3.1 > > > Key: HADOOP-17647 > URL: https://issues.apache.org/jira/browse/HADOOP-17647 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > File this jira to track the release work of Hadoop 3.3.1 > Release dashboard: > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19035) CrcUtil/CrcComposer should not throw IOException for non-IO
[ https://issues.apache.org/jira/browse/HADOOP-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-19035. -- Fix Version/s: 3.5.0 Resolution: Fixed > CrcUtil/CrcComposer should not throw IOException for non-IO > --- > > Key: HADOOP-19035 > URL: https://issues.apache.org/jira/browse/HADOOP-19035 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > CrcUtil and CrcComposer should throw specific exceptions for non-IO cases > - IllegalArgumentException: invalid arguments > - ArrayIndexOutOfBoundsException: index exceeds array size > - IllegalStateException: unexpected computation state -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18956) Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider
[ https://issues.apache.org/jira/browse/HADOOP-18956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18956. -- Fix Version/s: 3.4.0 Resolution: Fixed > Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and > ZKSignerSecretProvider > -- > > Key: HADOOP-18956 > URL: https://issues.apache.org/jira/browse/HADOOP-18956 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zita Dombi >Assignee: István Fajth >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HADOOP-18709 added support for Zookeeper to communicate with SSL/TLS enabled > in hadoop-common. With those changes we have the necessary parameters, that > we need to set to enable SSL/TLS in a ZK Client. That change also did changes > in ZKCuratorManager, so with that it is easy to set the SSL/TLS, for Yarn it > was done in YARN-11468. > In DelegationTokenAuthenticationFilter currently we are using > CuratorFrameworkFactory, it'd be good to change it to use ZKCuratorManager > and with that we should support SSL/TLS enablement. > *UPDATE* > So as I investigated this a bit more, it wouldn't be so easy to move to using > ZKCuratorManager. > DelegationTokenAuthenticationFilter uses ZK from two places: in > ZKDelegationTokenSecretManager and in ZKSignerSecretProvider. In both places > it uses CuratorFrameworkFactory, but the attributes and creation > differentiates from ZKCuratorManager. > In ZKDelegationTokenSecretManager it would be easy to add the new config and > based on that create ZK with CuratorFrameworkFactory. But > ZKSignerSecretProvider is in hadoop-auth module and with my change it would > need hadoop-common, so it would introduce circular dependency between modules > 'hadoop-auth' and 'hadoop-common'. I'm still working on a straightforward > solution. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
Wei-Chiu Chuang created HADOOP-18843: Summary: Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) Key: HADOOP-18843 URL: https://issues.apache.org/jira/browse/HADOOP-18843 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18766) 3.3.6 Dependency vulnerability check
[ https://issues.apache.org/jira/browse/HADOOP-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18766. -- Fix Version/s: 3.3.6 Resolution: Done > 3.3.6 Dependency vulnerability check > > > Key: HADOOP-18766 > URL: https://issues.apache.org/jira/browse/HADOOP-18766 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.6 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.3.6 > > Attachments: dependency-check-report.html > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18765) Release 3.3.6
[ https://issues.apache.org/jira/browse/HADOOP-18765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18765. -- Fix Version/s: 3.3.6 Resolution: Done > Release 3.3.6 > - > > Key: HADOOP-18765 > URL: https://issues.apache.org/jira/browse/HADOOP-18765 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.5 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.3.6 > > > * Move out all incomplete jiras > * Branching > * Unit test and verifications > * License check > * Produce signed artifacts and source tarball. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18769) Upgrade hadoop3 docker scripts to use 3.3.6
[ https://issues.apache.org/jira/browse/HADOOP-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18769. -- Fix Version/s: 3.3.6 Resolution: Fixed > Upgrade hadoop3 docker scripts to use 3.3.6 > --- > > Key: HADOOP-18769 > URL: https://issues.apache.org/jira/browse/HADOOP-18769 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.6 > > > Similar to what was done in HADOOP-18681 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18780) Add tags 3.3/3.3.5 to the official Hadoop docker image
[ https://issues.apache.org/jira/browse/HADOOP-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18780. -- Resolution: Fixed > Add tags 3.3/3.3.5 to the official Hadoop docker image > -- > > Key: HADOOP-18780 > URL: https://issues.apache.org/jira/browse/HADOOP-18780 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > > Our official docker repository has just two tags for the images: 2 and 3. > Shall we add more tags to the images? For example Hadoop 3 has two release > lines: 3.2 and 3.3. And probably even maintenance release tags: 3.3.5, 3.3.6. > This way, downstream users will not have surprises when we make a new release. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18780) Add tag 3.3/3.3.5 for the official Hadoop docker image
Wei-Chiu Chuang created HADOOP-18780: Summary: Add tag 3.3/3.3.5 for the official Hadoop docker image Key: HADOOP-18780 URL: https://issues.apache.org/jira/browse/HADOOP-18780 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Our official docker repository has just two tags for the images: 2 and 3. Shall we add more tags to the images? For example Hadoop 3 has two release lines: 3.2 and 3.3. And probably even maintenance release tags: 3.3.5, 3.3.6. This way, downstream users will not have surprises when we make a new release. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18761) Remove mysql-connector-java
[ https://issues.apache.org/jira/browse/HADOOP-18761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18761. -- Fix Version/s: 3.4.0 3.3.6 Resolution: Fixed > Remove mysql-connector-java > --- > > Key: HADOOP-18761 > URL: https://issues.apache.org/jira/browse/HADOOP-18761 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > While preparing for 3.3.6 RC, I realized the mysql-connector-java dependency > added by HADOOP-18535 is GPL licensed. > Source: https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE > See legal discussion at LEGAL-423. > I looked at the original jira and github PR and I don't think the license > issue was noticed. > Is it possible to get rid of the mysql connector dependency? As far as I can > tell the dependency is very limited. > If not, I guess I'll have to revert the commits for now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18459) scan detected CVE-2021-37136 and CVE-2021-37137 in netty.io_netty_codec
[ https://issues.apache.org/jira/browse/HADOOP-18459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18459. -- Resolution: Won't Fix snakeyaml is updated in 3.3.x. netty3 is removed in trunk. I'll resolve this as a duplicate. > scan detected CVE-2021-37136 and CVE-2021-37137 in netty.io_netty_codec > --- > > Key: HADOOP-18459 > URL: https://issues.apache.org/jira/browse/HADOOP-18459 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: deepagkanaka >Assignee: Ashutosh Gupta >Priority: Minor > > Our security scan detected CVE-2021-37136 and CVE-2021-37137 in io.netty_netty > > |Component|Version|CVE|Fixed in| > |io.netty_netty|3.10.6|CVE-2021-37136|4.1.68| > |io.netty_netty|3.10.6|CVE-2021-37137|4.1.68| > |org.yaml_snakeyaml|1.26|CVE-2022-25857|1.31| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18123) Netty version 3.10.6 to be upgraded to handle CVE-2021-43797
[ https://issues.apache.org/jira/browse/HADOOP-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18123. -- Resolution: Duplicate Netty3 is removed in Hadoop 3.4 in trunk. Resolve as a duplicate. > Netty version 3.10.6 to be upgraded to handle CVE-2021-43797 > > > Key: HADOOP-18123 > URL: https://issues.apache.org/jira/browse/HADOOP-18123 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Sushanta Sen >Priority: Major > > Netty version 3.10.6 to be upgraded to handle CVE-2021-43797, even Netty-4 > upgraded to handle this issue. > Please feedback -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13413) Update ZooKeeper version to 3.4.9
[ https://issues.apache.org/jira/browse/HADOOP-13413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-13413. -- Resolution: Won't Fix Stale jira; clean up. > Update ZooKeeper version to 3.4.9 > - > > Key: HADOOP-13413 > URL: https://issues.apache.org/jira/browse/HADOOP-13413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Tsuyoshi Ozawa >Priority: Major > > Just a reminder not to update ZooKeeper's version to 3.4.9 for syncing up > with netty's version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18769) Upgrade hadoop3 docker scripts to use 3.3.6
Wei-Chiu Chuang created HADOOP-18769: Summary: Upgrade hadoop3 docker scripts to use 3.3.6 Key: HADOOP-18769 URL: https://issues.apache.org/jira/browse/HADOOP-18769 Project: Hadoop Common Issue Type: Sub-task Reporter: Wei-Chiu Chuang Similar to what was done in HADOOP-18681 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18766) 3.3.6 Dependency vulnerability check
Wei-Chiu Chuang created HADOOP-18766: Summary: 3.3.6 Dependency vulnerability check Key: HADOOP-18766 URL: https://issues.apache.org/jira/browse/HADOOP-18766 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.6 Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18765) Release 3.3.6
Wei-Chiu Chuang created HADOOP-18765: Summary: Release 3.3.6 Key: HADOOP-18765 URL: https://issues.apache.org/jira/browse/HADOOP-18765 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang * Move out all incomplete jiras * Unit test and verifications * License check * Produce signed artifacts and source tarball. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18761) Revert HADOOP-18535 because mysql-connector-java is GPL
Wei-Chiu Chuang created HADOOP-18761: Summary: Revert HADOOP-18535 because mysql-connector-java is GPL Key: HADOOP-18761 URL: https://issues.apache.org/jira/browse/HADOOP-18761 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang While preparing for 3.3.6 RC, I realized the mysql-connector-java dependency added by HADOOP-18535 is GPL licensed. Source: https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE See legal discussion at LEGAL-423. I looked at the original jira and github PR and I don't think the license issue was noticed. Is it possible to get rid of the mysql connector dependency? As far as I can tell the dependency is very limited. If not, I guess I'll have to revert the commits for now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18760) 3.3.6 Release NOTICE and LICENSE file update
Wei-Chiu Chuang created HADOOP-18760: Summary: 3.3.6 Release NOTICE and LICENSE file update Key: HADOOP-18760 URL: https://issues.apache.org/jira/browse/HADOOP-18760 Project: Hadoop Common Issue Type: Task Affects Versions: 3.3.6 Reporter: Wei-Chiu Chuang As far as I can tell looking at hadoop-project/pom.xml the only difference between 3.3.5 and 3.3.6 from a dependency point of view is mysql connector (HADOOP-18535) derby (HADOOP-18535, HADOOP-18693). Json-smart, snakeyaml and jetty, jettison are updated in LICENSE-binary already. grizzly was used in test scope only so its removal doesn't matter. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11219) [Umbrella] Upgrade to netty 4
[ https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-11219. -- Fix Version/s: 3.4.0 Resolution: Fixed This is done. Would be nice to cherrypick the YARN shuffle handler change into branch-3.3. But at least this is finished in trunk so we can get rid of netty3 in Hadoop 3.4.0 > [Umbrella] Upgrade to netty 4 > - > > Key: HADOOP-11219 > URL: https://issues.apache.org/jira/browse/HADOOP-11219 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Major > Fix For: 3.4.0 > > > This is an umbrella jira to track the effort of upgrading to Netty 4. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18731) Backport HADOOP-18671 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18731. -- Resolution: Done > Backport HADOOP-18671 to branch-3.2 > --- > > Key: HADOOP-18731 > URL: https://issues.apache.org/jira/browse/HADOOP-18731 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18730) Backport HADOOP-18671 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-18730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18730. -- Resolution: Done > Backport HADOOP-18671 to branch-3.3 > --- > > Key: HADOOP-18730 > URL: https://issues.apache.org/jira/browse/HADOOP-18730 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18731) Backport HADOOP-18671 to branch-3.2
Wei-Chiu Chuang created HADOOP-18731: Summary: Backport HADOOP-18671 to branch-3.2 Key: HADOOP-18731 URL: https://issues.apache.org/jira/browse/HADOOP-18731 Project: Hadoop Common Issue Type: Sub-task Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18730) Backport HADOOP-18671 to branch-3.3
Wei-Chiu Chuang created HADOOP-18730: Summary: Backport HADOOP-18671 to branch-3.3 Key: HADOOP-18730 URL: https://issues.apache.org/jira/browse/HADOOP-18730 Project: Hadoop Common Issue Type: Sub-task Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18702) Remove ozone mailing lists from Hadoop website
[ https://issues.apache.org/jira/browse/HADOOP-18702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18702. -- Resolution: Fixed > Remove ozone mailing lists from Hadoop website > -- > > Key: HADOOP-18702 > URL: https://issues.apache.org/jira/browse/HADOOP-18702 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Ozone is its own TLP. Drop its information from Hadoop website. > https://hadoop.apache.org/mailing_lists.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18702) Remove ozone mailing list from Hadoop website
Wei-Chiu Chuang created HADOOP-18702: Summary: Remove ozone mailing list from Hadoop website Key: HADOOP-18702 URL: https://issues.apache.org/jira/browse/HADOOP-18702 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Ozone is its own TLP. Drop its information from Hadoop website. https://hadoop.apache.org/mailing_lists.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18548) Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems
[
https://issues.apache.org/jira/browse/HADOOP-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-18548.
--
Fix Version/s: 3.4.0
Resolution: Fixed
Thanks [~sgal] for the PR and [~cnauroth] for the review!
> Hadoop Archive tool (HAR) should acquire delegation tokens from source and
> destination file systems
> ---
>
> Key: HADOOP-18548
> URL: https://issues.apache.org/jira/browse/HADOOP-18548
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Reporter: Wei-Chiu Chuang
>Assignee: Szabolcs Gál
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Running Hadoop Archive tool (HAR) against source/dest that are not HDFS does
> not work today.
> Example:
> {noformat}
> hadoop archive -archiveName foo.har -p ofs://ozone1/vol2/bucket2/src/
> ofs://ozone1/vol2/bucket2/dst/
> {noformat}
> The command fails because it's essentially a MapReduce job and for some
> reason it does not obtain Ozone delegation token (only HDFS and KMS)
> The failure:
> {noformat}
> 22/11/30 18:12:10 INFO mapreduce.Job: Job job_1669745757520_0001 failed with
> state FAILED due to: Job setup failed : java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> ...
> {noformat}
> The workaround is to add the ozone path as the default file system:
> {noformat}
> export HADOOP_OPTS="$HADOOP_OPTS -Dfs.defaultFS=ofs://ozone1/vol2/bucket2"
> {noformat}
> A proper fix should make sure HAR requests delegation tokens from src/dest
> file system too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18668) Path capability probe for truncate is only honored by RawLocalFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18668. -- Fix Version/s: 3.4.0 Resolution: Fixed > Path capability probe for truncate is only honored by RawLocalFileSystem > > > Key: HADOOP-18668 > URL: https://issues.apache.org/jira/browse/HADOOP-18668 > Project: Hadoop Common > Issue Type: Bug > Components: fs, httpfs, viewfs >Affects Versions: 3.3.4 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > FileSystem#hasPathCapability returns true for probing > "fs.capability.paths.truncate" only by RawLocalFileSystem. It should be > honored by all file system implementations that support truncate. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18671) Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem
Wei-Chiu Chuang created HADOOP-18671: Summary: Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem Key: HADOOP-18671 URL: https://issues.apache.org/jira/browse/HADOOP-18671 Project: Hadoop Common Issue Type: New Feature Components: fs Reporter: Wei-Chiu Chuang We are in the midst of enabling HBase and Solr to run on Ozone. An obstacle is that HBase relies heavily on HDFS APIs and semantics for its Write Ahead Log (WAL) file (similarly, for Solr's transaction log). We propose to push up these HDFS APIs, i.e. recoverLease(), setSafeMode(), isFileClosed() to FileSystem abstraction so that HBase and other applications do not need to take on Ozone dependency at compile time. This work will (hopefully) enable HBase to run on other storage system implementations in the future. There are other HDFS features that HBase uses, including hedged read and favored nodes. Those are FS-specific optimizations and are not critical to enable HBase on Ozone. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18648) Avoid loading kms log4j properties dynamically by KMSWebServer
[ https://issues.apache.org/jira/browse/HADOOP-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18648. -- Fix Version/s: 3.4.0 Resolution: Fixed > Avoid loading kms log4j properties dynamically by KMSWebServer > -- > > Key: HADOOP-18648 > URL: https://issues.apache.org/jira/browse/HADOOP-18648 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Log4j2 does not support loading of log4j properties (/xml/json/yaml) > dynamically by applications. It no longer supports overriding the loading of > properties using "log4j.defaultInitOverride" the way log4j1 does. > For KMS, instead of loading the properties file dynamically, we should add > the log4j properties file as part of HADOOP_OPTS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18631) Migrate Async appenders to log4j properties
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18631. -- Fix Version/s: 3.4.0 Resolution: Fixed > Migrate Async appenders to log4j properties > --- > > Key: HADOOP-18631 > URL: https://issues.apache.org/jira/browse/HADOOP-18631 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Before we can upgrade to log4j2, we need to migrate async appenders that we > add "dynamically in the code" to the log4j.properties file. Instead of using > core/hdfs site configs, log4j properties or system properties should be used > to determine if the given logger should use async appender. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18206. -- Fix Version/s: 3.4.0 Resolution: Fixed > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18620) Avoid using grizzly-http-* APIs
[ https://issues.apache.org/jira/browse/HADOOP-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18620. -- Fix Version/s: 3.4.0 Resolution: Fixed > Avoid using grizzly-http-* APIs > --- > > Key: HADOOP-18620 > URL: https://issues.apache.org/jira/browse/HADOOP-18620 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > As discussed on the parent Jira HADOOP-15984, we do not have any > grizzly-http-servlet version available that uses Jersey 2 dependencies. > version 2.4.4 contains Jersey 1 artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/2.4.4/grizzly-http-servlet-2.4.4.pom] > The next higher version available is 3.0.0-M1 and it contains Jersey 3 > artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/3.0.0-M1/grizzly-http-servlet-3.0.0-M1.pom] > > Moreover, we do not use grizzly-http-* modules extensively. We use them only > for few tests such that we don't have to implement all the methods of > HttpServletResponse for our custom test classes. > We should get rid of grizzly-http-servlet, grizzly-http and > grizzly-http-server artifacts of org.glassfish.grizzly and rather implement > HttpServletResponse directly to avoid having to depend on grizzly upgrades as > part of overall Jersey upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18584) [NFS GW] Fix regression after netty4 migration
[ https://issues.apache.org/jira/browse/HADOOP-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18584. -- Fix Version/s: 3.4.0 Resolution: Fixed > [NFS GW] Fix regression after netty4 migration > -- > > Key: HADOOP-18584 > URL: https://issues.apache.org/jira/browse/HADOOP-18584 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Two bugs were found recently. Both related to hadoop portmap utility. > (1) The ChannelHandler that can be added to a ChannelPipeline in Netty4 must > be Shareable. However, IdleStateHandler, used by Portmap is not. We need to > fix it otherwise hadoop portmap may fail to start. > (2) RpcUdpResponseStage does not get the request from client (hadoop mountd) > because of response type mismatch, and therefore it does not respond back to > client request. > This is a regression from HADOOP-11245 but I consider it not a blocker, > because most recent Linux distro supports rpcbind and there's little reason > to use Hadoop's portmap. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16876) KMS delegation tokens are memory expensive
[ https://issues.apache.org/jira/browse/HADOOP-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-16876. -- Resolution: Cannot Reproduce > KMS delegation tokens are memory expensive > -- > > Key: HADOOP-16876 > URL: https://issues.apache.org/jira/browse/HADOOP-16876 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Wei-Chiu Chuang >Priority: Major > Attachments: Screen Shot 2020-02-20 at 5.04.12 PM.png > > > We recently saw a number of users reporting high memory consumption in KMS. > Part of the reason being HADOOP-14445. Without that, the number of kms > delegation tokens that zookeeper stores is proportional to the number of KMS > servers. > There are two problems: > (1) it exceeds zookeeper jute buffer length and operations fail. > (2) KMS uses more heap memory to store KMS DTs. > But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap > dump from KMS, the majority of the heap is occupied by znode and KMS DT > objects. With the growing number of encrypted clusters and use cases, this is > increasingly a problem our users encounter. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18584) [NFS GW] IdleStateHandler is not Shareable
Wei-Chiu Chuang created HADOOP-18584: Summary: [NFS GW] IdleStateHandler is not Shareable Key: HADOOP-18584 URL: https://issues.apache.org/jira/browse/HADOOP-18584 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.4.0 Reporter: Wei-Chiu Chuang The ChannelHandler that can be added to a ChannelPipeline in Netty4 must be Shareable. However, IdleStateHandler, used by Portmap is not. We need to fix it otherwise hadoop portmap may fail to start. This is a regression from HADOOP-11245 but I consider it not a blocker, because most recent Linux distro supports rpcbind and there's little reason to use Hadoop's portmap. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18548) Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems
Wei-Chiu Chuang created HADOOP-18548:
Summary: Hadoop Archive tool (HAR) should acquire delegation
tokens from source and destination file systems
Key: HADOOP-18548
URL: https://issues.apache.org/jira/browse/HADOOP-18548
Project: Hadoop Common
Issue Type: Bug
Components: tools
Reporter: Wei-Chiu Chuang
Running Hadoop Archive tool (HAR) against source/dest that are not HDFS does
not work today.
Example:
{noformat}
hadoop archive -archiveName foo.har -p ofs://ozone1/vol2/bucket2/src/
ofs://ozone1/vol2/bucket2/dst/
{noformat}
The command fails because it's essentially a MapReduce job and for some reason
it does not obtain Ozone delegation token (only HDFS and KMS)
The failure:
{noformat}
22/11/30 18:12:10 INFO mapreduce.Job: Job job_1669745757520_0001 failed with
state FAILED due to: Job setup failed : java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:789)
at java.security.AccessController.doPrivileged(Native Method)
...
{noformat}
The workaround is to add the ozone path as the default file system:
{noformat}
export HADOOP_OPTS="$HADOOP_OPTS -Dfs.defaultFS=ofs://ozone1/vol2/bucket2"
{noformat}
A proper fix should make sure HAR requests delegation tokens from src/dest file
system too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18489) Cherrypick HADOOP-11245 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18489. -- Fix Version/s: 3.2.5 Resolution: Fixed > Cherrypick HADOOP-11245 to branch-3.2 > - > > Key: HADOOP-18489 > URL: https://issues.apache.org/jira/browse/HADOOP-18489 > Project: Hadoop Common > Issue Type: Sub-task > Components: nfs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.2.5 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18489) Cherrypick HADOOP-11245 to branch-3.2
Wei-Chiu Chuang created HADOOP-18489: Summary: Cherrypick HADOOP-11245 to branch-3.2 Key: HADOOP-18489 URL: https://issues.apache.org/jira/browse/HADOOP-18489 Project: Hadoop Common Issue Type: Sub-task Components: nfs Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18488) Cherrypick HADOOP-11245 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-18488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18488. -- Fix Version/s: 3.3.9 Resolution: Fixed > Cherrypick HADOOP-11245 to branch-3.3 > - > > Key: HADOOP-18488 > URL: https://issues.apache.org/jira/browse/HADOOP-18488 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18488) Cherrypick HADOOP-11245 to branch-3.3
Wei-Chiu Chuang created HADOOP-18488: Summary: Cherrypick HADOOP-11245 to branch-3.3 Key: HADOOP-18488 URL: https://issues.apache.org/jira/browse/HADOOP-18488 Project: Hadoop Common Issue Type: Sub-task Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18118) Fix KMS Accept Queue Size default value to 500
[ https://issues.apache.org/jira/browse/HADOOP-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18118. -- Fix Version/s: 3.4.0 Resolution: Fixed > Fix KMS Accept Queue Size default value to 500 > -- > > Key: HADOOP-18118 > URL: https://issues.apache.org/jira/browse/HADOOP-18118 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > From HADOOP-15638,`hadoop.http.socket.backlog.size` was set as 500 by default > ,we can change code default value to keep consistent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18333. -- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded > jetty > --- > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and > impacts 9.4.0 thru 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18079) Upgrade Netty to 4.1.77.Final
[ https://issues.apache.org/jira/browse/HADOOP-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18079. -- Resolution: Fixed > Upgrade Netty to 4.1.77.Final > - > > Key: HADOOP-18079 > URL: https://issues.apache.org/jira/browse/HADOOP-18079 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.3 >Reporter: Renukaprasad C >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4, 3.2.5 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > h4. Netty version - 4.1.71 has fix some CVEs. > CVE-2019-20444, > CVE-2019-20445 > CVE-2022-24823 > Upgrade to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18289) Remove WhiteBox in hadoop-kms module.
[ https://issues.apache.org/jira/browse/HADOOP-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18289. -- Fix Version/s: 3.4.0 Resolution: Fixed Done. Thanks [~slfan1989] > Remove WhiteBox in hadoop-kms module. > - > > Key: HADOOP-18289 > URL: https://issues.apache.org/jira/browse/HADOOP-18289 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > WhiteBox is deprecated, try to remove this method in hadoop-kms. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18140) Update default value of hadoop.ssl.enabled.protocols in the EncryptedShuffle doc
[ https://issues.apache.org/jira/browse/HADOOP-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18140. -- Resolution: Duplicate > Update default value of hadoop.ssl.enabled.protocols in the EncryptedShuffle > doc > > > Key: HADOOP-18140 > URL: https://issues.apache.org/jira/browse/HADOOP-18140 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0, 2.8.4 >Reporter: Wei-Chiu Chuang >Assignee: Samrat Deb >Priority: Trivial > Labels: newbie > > The default value of hadoop.ssl.enabled.protocols was updated several times > but our doc was never updated. > https://hadoop.apache.org/docs/r3.1.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18245) Extend KMS related exceptions that get mapped to ConnectException
[ https://issues.apache.org/jira/browse/HADOOP-18245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18245. -- Fix Version/s: 3.4.0 Resolution: Fixed The PR is merged in trunk. Thanks [~kerneltime]! > Extend KMS related exceptions that get mapped to ConnectException > -- > > Key: HADOOP-18245 > URL: https://issues.apache.org/jira/browse/HADOOP-18245 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Ritesh H Shukla >Assignee: Ritesh H Shukla >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Based on production workload, we found that it is not enough to map just > SSLHandshakeException to ConnectException in Loadbalancing KMS Client but > that needs to be extended to SSLExceptions and SocketExceptions. > Sample JDK code that can raise these exceptions: > https://github.com/openjdk/jdk/blob/jdk-18%2B32/src/java.base/share/classes/sun/security/ssl/SSLSocketImpl.java#L1409-L1428 > Sample Exception backtrace: > 22/04/13 16:25:53 WARN kms.LoadBalancingKMSClientProvider: KMS provider at > [https://bdgtr041x10h5.nam.nsroot.net:16001/kms/v1/] threw an IOException: > javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake > at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1470) > at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1298) > at > sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1199) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:373) > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:587) > at > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDe > Caused by: java.io.EOFException: SSL peer shut down incorrectly > at > sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:480) > at > sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:469) > ... 59 more -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17526. -- Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~zhangduo] for the work. Merged the PR into trunk. > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18140) Update default value of hadoop.ssl.enabled.protocols in the EncryptedShuffle doc
Wei-Chiu Chuang created HADOOP-18140: Summary: Update default value of hadoop.ssl.enabled.protocols in the EncryptedShuffle doc Key: HADOOP-18140 URL: https://issues.apache.org/jira/browse/HADOOP-18140 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.8.4, 2.9.0 Reporter: Wei-Chiu Chuang The default value of hadoop.ssl.enabled.protocols was updated several times but our doc was never updated. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18125) Utility to identify git commit / Jira fixVersion discrepancies for RC preparation
[ https://issues.apache.org/jira/browse/HADOOP-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18125. -- Fix Version/s: 3.4.0 2.10.2 3.2.3 3.3.3 Resolution: Fixed > Utility to identify git commit / Jira fixVersion discrepancies for RC > preparation > - > > Key: HADOOP-18125 > URL: https://issues.apache.org/jira/browse/HADOOP-18125 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > As part of RC preparation, we need to identify all git commits that landed > on release branch, however their corresponding Jira is either not resolved > yet or does not contain expected fixVersions. Only when we have git commits > and corresponding Jiras with expected fixVersion resolved, we get all such > Jiras included in auto-generated CHANGES.md as per Yetus changelog generator. > Proposal of this Jira is to provide such script that can be useful for all > upcoming RC preparations and list down all Jiras where we need manual > intervention. This utility script should use Jira API to retrieve individual > fields and use git log to loop through commit history. > The script should identify these issues: > # commit is reverted as per commit message > # commit does not contain Jira number format (e.g. HADOOP- / HDFS- > etc) in message > # Jira does not have expected fixVersion > # Jira has expected fixVersion, but it is not yet resolved > # Jira has release corresponding fixVersion and is resolved, but no > corresponding commit yet found > It can take inputs as: > # First commit hash to start excluding commits from history > # Fix Version > # JIRA Project Name > # Path of project's working dir > # Jira server url -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18098) Basic verification for the release candidate vote
[ https://issues.apache.org/jira/browse/HADOOP-18098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18098. -- Fix Version/s: 3.4.0 Resolution: Fixed Merged into trunk. Thanks [~vjasani] > Basic verification for the release candidate vote > - > > Key: HADOOP-18098 > URL: https://issues.apache.org/jira/browse/HADOOP-18098 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We should provide script for the basic sanity of Hadoop release candidates. > It should include: > * Signature > * Checksum > * Rat check > * Build from src > * Build tarball from src > > Although we can include unit test as well, but overall unit test run is going > to be significantly higher, and precommit Jenkins builds provide better view > of UT sanity. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18089) Test coverage for Async profiler servlets
[ https://issues.apache.org/jira/browse/HADOOP-18089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18089. -- Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~vjasani] for the patch and [~aajisaka] for the review! > Test coverage for Async profiler servlets > - > > Key: HADOOP-18089 > URL: https://issues.apache.org/jira/browse/HADOOP-18089 > Project: Hadoop Common > Issue Type: Test >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > As discussed in HADOOP-18077, we should provide sufficient test coverage to > discover any potential regression in async profiler servlets: ProfileServlet > and ProfileOutputServlet. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18088) Replace log4j 1.x with reload4j
Wei-Chiu Chuang created HADOOP-18088: Summary: Replace log4j 1.x with reload4j Key: HADOOP-18088 URL: https://issues.apache.org/jira/browse/HADOOP-18088 Project: Hadoop Common Issue Type: Improvement Reporter: Wei-Chiu Chuang As proposed in the dev mailing list (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18077) ProfileOutputServlet unable to proceed due to NPE
[
https://issues.apache.org/jira/browse/HADOOP-18077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-18077.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> ProfileOutputServlet unable to proceed due to NPE
> -
>
> Key: HADOOP-18077
> URL: https://issues.apache.org/jira/browse/HADOOP-18077
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> ProfileOutputServlet context doesn't have Hadoop configs available and hence
> async profiler redirection to output servlet is failing to identify if admin
> access is allowed:
> {code:java}
> HTTP ERROR 500 java.lang.NullPointerException
> URI: /prof-output-hadoop/async-prof-pid-98613-cpu-2.html
> STATUS: 500
> MESSAGE: java.lang.NullPointerException
> SERVLET: org.apache.hadoop.http.ProfileOutputServlet-58c34bb3
> CAUSED BY: java.lang.NullPointerException
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1619)
> at
> org.apache.hadoop.http.ProfileOutputServlet.doGet(ProfileOutputServlet.java:51)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18044) Hadoop - Upgrade to JQuery 3.6.0
[ https://issues.apache.org/jira/browse/HADOOP-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18044. -- Resolution: Fixed > Hadoop - Upgrade to JQuery 3.6.0 > > > Key: HADOOP-18044 > URL: https://issues.apache.org/jira/browse/HADOOP-18044 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yuan Luo >Assignee: Yuan Luo >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > jQuery 3.6.0 has been released few months ago - > [http://blog.jquery.com/2021/03/02/jquery-3-6-0-released/ > |http://blog.jquery.com/2021/03/02/jquery-3-6-0-released/,] > We can upgrade jquery-3.5.1.min.js to jquery-3.6.0.min.js in hadoop project. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18050) Document Hadoop's stance on the log4jshell vulnerability
[ https://issues.apache.org/jira/browse/HADOOP-18050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18050. -- Resolution: Done > Document Hadoop's stance on the log4jshell vulnerability > > > Key: HADOOP-18050 > URL: https://issues.apache.org/jira/browse/HADOOP-18050 > Project: Hadoop Common > Issue Type: Task > Components: documentation >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > As of today, Hadoop relies on log4j-1, not log4j2. It is understood that the > log4jshell vulnerability (CVE-2021-44228) does not impact log4j-1. Given the > widespread attention to the incidence, we should make it clear that Hadoop is > not susceptible to the attack. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18050) Document Hadoop's stance on the log4jshell vulnerability
Wei-Chiu Chuang created HADOOP-18050: Summary: Document Hadoop's stance on the log4jshell vulnerability Key: HADOOP-18050 URL: https://issues.apache.org/jira/browse/HADOOP-18050 Project: Hadoop Common Issue Type: Task Components: documentation Reporter: Wei-Chiu Chuang As of today, Hadoop relies on log4j-1, not log4j2. It is understood that the log4jshell vulnerability (CVE-2021-44228) does not impact log4j-1. Given the widespread attention to the incidence, we should make it clear that Hadoop is not susceptible to the attack. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17982) OpensslCipher initialization error should log a WARN message
Wei-Chiu Chuang created HADOOP-17982:
Summary: OpensslCipher initialization error should log a WARN
message
Key: HADOOP-17982
URL: https://issues.apache.org/jira/browse/HADOOP-17982
Project: Hadoop Common
Issue Type: Improvement
Components: kms, security
Reporter: Wei-Chiu Chuang
We spent months troubleshooting a RangerKMS performance problem, only to
realize that the openssl library wasn't even loaded properly.
The failure to load openssl lib is currently logged as a debug message during
initialization. We really should upgrade it to at least INFO/WARN.
{code}
static {
String loadingFailure = null;
try {
if (!NativeCodeLoader.buildSupportsOpenssl()) {
PerformanceAdvisory.LOG.debug("Build does not support openssl");
loadingFailure = "build does not support openssl.";
} else {
initIDs();
}
} catch (Throwable t) {
loadingFailure = t.getMessage();
LOG.debug("Failed to load OpenSSL Cipher.", t);
} finally {
loadingFailureReason = loadingFailure;
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17946) Update commons-lang to latest 3.x
[ https://issues.apache.org/jira/browse/HADOOP-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17946. -- Resolution: Fixed > Update commons-lang to latest 3.x > - > > Key: HADOOP-17946 > URL: https://issues.apache.org/jira/browse/HADOOP-17946 > Project: Hadoop Common > Issue Type: Task >Reporter: Sean Busbey >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > our commons-lang3 dependency is currently 3.7, which is nearly 4 years old. > latest right now is 3.12 and there are at least some fixes that would make us > more robust on JDKs newer than openjdk8 (e.g. LANG-1384. [release notes > indicate 3.9 is the first to support > jdk11|https://commons.apache.org/proper/commons-lang/changes-report.html]). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17929) implement non-guava Precondition checkArgument
[
https://issues.apache.org/jira/browse/HADOOP-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17929.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> implement non-guava Precondition checkArgument
> --
>
> Key: HADOOP-17929
> URL: https://issues.apache.org/jira/browse/HADOOP-17929
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0, 3.2.3, 3.3.2
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to add the implementation {{checkArgument}} to the existing
> class {{org.apache.hadoop.util.Preconditions}}
> +The plan is as follows+
> * implement {{org.apache.hadoop.util.Preconditions.checkArgument}} with the
> minimum set of interface used in the current hadoop repo.
> * we can replace {{guava.Preconditions}} by
> {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been
> implemented.
> * We need the change to be easily to be backported in 3.x.
> A previous jira HADOOP-17126 was created to replace CheckNotNull.
> HADOOP-17930 is created to implement checkState.
> CC: [~ste...@apache.org], [~vjasani]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17938) Print lockWarningThreshold in InstrumentedLock#logWarning and InstrumentedLock#logWaitWarning
[ https://issues.apache.org/jira/browse/HADOOP-17938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17938. -- Fix Version/s: 3.4.0 Resolution: Fixed > Print lockWarningThreshold in InstrumentedLock#logWarning and > InstrumentedLock#logWaitWarning > - > > Key: HADOOP-17938 > URL: https://issues.apache.org/jira/browse/HADOOP-17938 > Project: Hadoop Common > Issue Type: Improvement >Reporter: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Print lockWarningThreshold in InstrumentedLock#logWarning and > InstrumentedLock#logWaitWarning. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17875) [KMS] FileNotFoundException due to inability to access zookeeper znode
Wei-Chiu Chuang created HADOOP-17875: Summary: [KMS] FileNotFoundException due to inability to access zookeeper znode Key: HADOOP-17875 URL: https://issues.apache.org/jira/browse/HADOOP-17875 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Wei-Chiu Chuang A mysterious FileNotFoundException was thrown by KMS client. The KMS server did not log anything related to it. After extensive investigation it was realized it could be caused by either of the two Zookeeper errors: (1) unable to access the znode /zkdtsm due to Zookeeper ACL or other problems. (2) if there's a runtime dependency mismatch. In in case, our RangerKMS (where the HadoopKMS runtime is embedded), threw a NoSuchMethodError during curator initialization (the RangerKMS used an incompatible curator-recipes version at runtime). Curator was not initialized successfully and therefore unable to access the znode. In any case, it would be great to surface up the zookeeper issue to make troubleshooting easier. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17865) [KMS] Some error messages are not logged in the KMS server
Wei-Chiu Chuang created HADOOP-17865: Summary: [KMS] Some error messages are not logged in the KMS server Key: HADOOP-17865 URL: https://issues.apache.org/jira/browse/HADOOP-17865 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Wei-Chiu Chuang While troubleshooting a recent KMS issue, it was found a certain exceptions are not logged in the server side. When client got an error, for example, "token xxx not found in the cache", we wanted to find out more details from the server side (stack trace, exact exception...etc) but it was not found. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17837. -- Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~bbeaudreault]. The PR is merged. > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17796) Update Jetty to 9.4.41 or above
[ https://issues.apache.org/jira/browse/HADOOP-17796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17796. -- Fix Version/s: 3.3.2 3.4.0 Resolution: Fixed > Update Jetty to 9.4.41 or above > --- > > Key: HADOOP-17796 > URL: https://issues.apache.org/jira/browse/HADOOP-17796 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.2.2, 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Renukaprasad C >Priority: Major > Labels: dependency, pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 20m > Remaining Estimate: 0h > > https://github.com/eclipse/jetty.project/security/advisories/GHSA-m6cp-vxjx-65j6 > https://github.com/eclipse/jetty.project/security/advisories/GHSA-gwcr-j4wh-j3cq -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17810) [KMS] LoadBalancingKMSClientProvider should implement toString()
Wei-Chiu Chuang created HADOOP-17810:
Summary: [KMS] LoadBalancingKMSClientProvider should implement
toString()
Key: HADOOP-17810
URL: https://issues.apache.org/jira/browse/HADOOP-17810
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 3.3.1
Reporter: Wei-Chiu Chuang
KMSClientProvider implements toString() but LoadBalancingKMSClientProvider does
not inherit from KMSClientProvider, nor does it implement toString(). So
running hadoop key command does not show key provider URL:
{noformat}
bin/hadoop key list -metadata
2021-07-21 15:26:43,856 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Listing keys for KeyProvider:
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@394a2528{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17672) Remove an invalid comment content in the FileContext class
[ https://issues.apache.org/jira/browse/HADOOP-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17672. -- Fix Version/s: 3.4.0 Resolution: Fixed Merged into trunk. Thanks [~ayushsaxena] for the review. > Remove an invalid comment content in the FileContext class > -- > > Key: HADOOP-17672 > URL: https://issues.apache.org/jira/browse/HADOOP-17672 > Project: Hadoop Common > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There is an invalid comment content in the FileContext class: >/** > * Protected Static Factory methods for getting a FileContexts > * that take a AbstractFileSystem as input. To be used for testing. > */ > It doesn't seem to have any effect. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17796) Update Jetty to 9.4.41 or above
Wei-Chiu Chuang created HADOOP-17796: Summary: Update Jetty to 9.4.41 or above Key: HADOOP-17796 URL: https://issues.apache.org/jira/browse/HADOOP-17796 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.3.1, 3.2.2, 3.4.0 Reporter: Wei-Chiu Chuang https://github.com/eclipse/jetty.project/security/advisories/GHSA-m6cp-vxjx-65j6 https://github.com/eclipse/jetty.project/security/advisories/GHSA-gwcr-j4wh-j3cq -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17780) Remove reflections used to access HDFS EC APIs
Wei-Chiu Chuang created HADOOP-17780: Summary: Remove reflections used to access HDFS EC APIs Key: HADOOP-17780 URL: https://issues.apache.org/jira/browse/HADOOP-17780 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.0-alpha1 Reporter: Wei-Chiu Chuang HDFS EC APIs exists since Hadoop 3.0. We can access them directly without reflections in HBase 3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17727) Modularize docker images
[ https://issues.apache.org/jira/browse/HADOOP-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17727. -- Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~gautham] > Modularize docker images > > > Key: HADOOP-17727 > URL: https://issues.apache.org/jira/browse/HADOOP-17727 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: build-log-centos-7.zip, build-log-centos-8.zip, > build-log-ubuntu-x64.zip > > Time Spent: 12h 40m > Remaining Estimate: 0h > > We're now creating the *Dockerfile*s for different platforms. We need a way > to manage the packages in a clean way as maintaining the packages for all the > different environments becomes cumbersome. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17008) Release Hadoop 3.3.0
[ https://issues.apache.org/jira/browse/HADOOP-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17008. -- Resolution: Done > Release Hadoop 3.3.0 > > > Key: HADOOP-17008 > URL: https://issues.apache.org/jira/browse/HADOOP-17008 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Brahma Reddy Battula >Priority: Major > > Thanks [~brahma] for making the effort to release Hadoop 3.3.0. Raise this > jira to track the progress, and use this jira to link other Apache projects' > Hadoop 3.3.0 support progress. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17739) Use hadoop-thirdparty 1.1.1
[ https://issues.apache.org/jira/browse/HADOOP-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17739. -- Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed Resolving. Thanks [~aajisaka]! > Use hadoop-thirdparty 1.1.1 > --- > > Key: HADOOP-17739 > URL: https://issues.apache.org/jira/browse/HADOOP-17739 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17563) Update Bouncy Castle to 1.68
[ https://issues.apache.org/jira/browse/HADOOP-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17563. -- Resolution: Fixed Reverted from branch-3.3.1. Resolve the jira. > Update Bouncy Castle to 1.68 > > > Key: HADOOP-17563 > URL: https://issues.apache.org/jira/browse/HADOOP-17563 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2h > Remaining Estimate: 0h > > -Bouncy Castle 1.60 has Hash Collision Vulnerability. Let's update to 1.68.- > Bouncy Castle 1.60 has the following vulnerabilities. Let's update to 1.68. > * [https://nvd.nist.gov/vuln/detail/CVE-2020-26939] > * [https://nvd.nist.gov/vuln/detail/CVE-2020-28052] > * [https://nvd.nist.gov/vuln/detail/CVE-2020-15522] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17723) [build] fix the Dockerfile for ARM
[
https://issues.apache.org/jira/browse/HADOOP-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17723.
--
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
> [build] fix the Dockerfile for ARM
> --
>
> Key: HADOOP-17723
> URL: https://issues.apache.org/jira/browse/HADOOP-17723
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Running the create-release script for Hadoop 3.3.1 on an ARM machine, docker
> image fails to build:
> {noformat}
> aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g
> -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
> -D_FORTIFY_SOURCE=2 -fPIC -Iast27/Include -I/usr/include/python3.6m -c
> ast27/Parser/acceler.c -o build/temp.linux-aarch64-3.6/ast27/Parser/acceler.o
>
> In file included from
> ast27/Parser/acceler.c:13:0:
> ast27/Parser/../Include/pgenheaders.h:8:10:
> fatal error: Python.h: No such file or directory
> #include "Python.h"
>
> ^~
> compilation terminated.
>
> error: command 'aarch64-linux-gnu-gcc' failed with exit
> status 1
> {noformat}
> The missing Python3.h requires python3-dev package:
> https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory
> The PhantomJS binary was built for Xenial, doesn't run after the Dockerfile
> migrated to Bionic/Focal. Fortunately Bionic/Focal has official PhantomJS
> packages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17733) Release hadoop-thirdparty 1.1.1
[ https://issues.apache.org/jira/browse/HADOOP-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17733. -- Resolution: Done > Release hadoop-thirdparty 1.1.1 > --- > > Key: HADOOP-17733 > URL: https://issues.apache.org/jira/browse/HADOOP-17733 > Project: Hadoop Common > Issue Type: Task >Affects Versions: thirdparty-1.1.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > Release hadoop-thirdparty 1.1.1 to address HADOOP-17730. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17739) Use hadoop-thirdparty 1.1.1
Wei-Chiu Chuang created HADOOP-17739: Summary: Use hadoop-thirdparty 1.1.1 Key: HADOOP-17739 URL: https://issues.apache.org/jira/browse/HADOOP-17739 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17734) Upgrade to hadoop-thirdparty 1.1.1
Wei-Chiu Chuang created HADOOP-17734: Summary: Upgrade to hadoop-thirdparty 1.1.1 Key: HADOOP-17734 URL: https://issues.apache.org/jira/browse/HADOOP-17734 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.1 Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17730) [thirdparty] Do not exclude error_prone
[
https://issues.apache.org/jira/browse/HADOOP-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17730.
--
Fix Version/s: thirdparty-1.1.1
thirdparty-1.2.0
Resolution: Fixed
> [thirdparty] Do not exclude error_prone
> ---
>
> Key: HADOOP-17730
> URL: https://issues.apache.org/jira/browse/HADOOP-17730
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1, thirdparty-1.1.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: pull-request-available
> Fix For: thirdparty-1.2.0, thirdparty-1.1.1
>
>
> Building Ozone on Hadoop 3.3.1-RC1, found the following error during
> compilation:
> {noformat}
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-hdds-common: Compilation failure
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR] cannot access
> org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR] class file for
> org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
> not found
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR] Consult the following stack
> trace for details.
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR]
> com.sun.tools.javac.code.Symbol$CompletionFailure: class file for
> org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
> not found
> 18:40:02 2021/05/24 10:40:02 INFO: [ERROR] -> [Help 1]
> {noformat}
> The error_prone was excluded by YARN-10195 to avoid dependency divergence
> with YARN Timeline service. However, now that Hadoop's guava is shaded, we
> shouldn't have problem with diverging dependencies. So let's add it back.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17733) Release hadoop-thirdparty 1.1.1
Wei-Chiu Chuang created HADOOP-17733: Summary: Release hadoop-thirdparty 1.1.1 Key: HADOOP-17733 URL: https://issues.apache.org/jira/browse/HADOOP-17733 Project: Hadoop Common Issue Type: Task Affects Versions: thirdparty-1.1.1 Reporter: Wei-Chiu Chuang Release hadoop-thirdparty 1.1.1 to address HADOOP-17730. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17730) [thirdparty] Do not exclude error_prone
Wei-Chiu Chuang created HADOOP-17730:
Summary: [thirdparty] Do not exclude error_prone
Key: HADOOP-17730
URL: https://issues.apache.org/jira/browse/HADOOP-17730
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Wei-Chiu Chuang
Building Ozone on Hadoop 3.3.1-RC1, found the following error during
compilation:
{noformat}
18:40:02 2021/05/24 10:40:02 INFO: [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-hdds-common: Compilation failure
18:40:02 2021/05/24 10:40:02 INFO: [ERROR] cannot access
org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
18:40:02 2021/05/24 10:40:02 INFO: [ERROR] class file for
org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
not found
18:40:02 2021/05/24 10:40:02 INFO: [ERROR] Consult the following stack
trace for details.
18:40:02 2021/05/24 10:40:02 INFO: [ERROR]
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for
org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
not found
18:40:02 2021/05/24 10:40:02 INFO: [ERROR] -> [Help 1]
{noformat}
The error_prone was excluded by YARN-10195 to avoid dependency divergence with
YARN Timeline service. However, now that Hadoop's guava is shaded, we shouldn't
have problem with diverging dependencies. So let's add it back.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17666) Update LICENSE for 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17666. -- Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed > Update LICENSE for 3.3.1 > > > Key: HADOOP-17666 > URL: https://issues.apache.org/jira/browse/HADOOP-17666 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available, release-blocker > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Before release, do another round of check for the LICENSE file to make sure > the dependency versions are updated correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17669) Port HADOOP-17079, HADOOP-17505 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HADOOP-17669: -- Looking at HADOOP-17079, there are a number of known bugs in the code. Revert it from 3.3.x > Port HADOOP-17079, HADOOP-17505 to branch-3.3 > -- > > Key: HADOOP-17669 > URL: https://issues.apache.org/jira/browse/HADOOP-17669 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17718) Explicitly set locale in the Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17718.
--
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
> Explicitly set locale in the Dockerfile
> ---
>
> Key: HADOOP-17718
> URL: https://issues.apache.org/jira/browse/HADOOP-17718
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When producing the RC bits for 3.3.1, the releasedocmaker step failed.
> {noformat}
> [INFO] --- exec-maven-plugin:1.3.1:exec (releasedocs) @ hadoop-common ---
> Traceback (most recent call last):
> File
> "/build/source/patchprocess/apache-yetus-0.13.0/bin/../lib/releasedocmaker/releasedocmaker.py",
> line 25, in
> releasedocmaker.main()
> File
> "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/__init__.py",
> line 979, in main
> JIRA_BASE_URL)
> File
> "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py",
> line 199, in write_list
> self.write_key_raw(jira.get_project(), line)
> File
> "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py",
> line 170, in write_key_raw
> self.base.write(input_string)
> UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position
> 71: ordinal not in range(128)
> {noformat}
> It turns out if the script reads jiras containing ascended characters, it
> can't write the report.
> Inside docker container, the default locale is "ANSI_X3.4-1968". Must set it
> to utf-8 to support special characters.
> Curious why it wasn't a problem before.
> More details:
> https://stackoverflow.com/questions/43356982/docker-python-set-utf-8-locale
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17723) [build] python3 migration broke Docker container image for ARM
Wei-Chiu Chuang created HADOOP-17723:
Summary: [build] python3 migration broke Docker container image
for ARM
Key: HADOOP-17723
URL: https://issues.apache.org/jira/browse/HADOOP-17723
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.3.1, 3.4.0
Reporter: Wei-Chiu Chuang
Running the create-release script for Hadoop 3.3.1 on an ARM machine, docker
image fails to build:
{noformat}
aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -fPIC -Iast27/Include -I/usr/include/python3.6m -c
ast27/Parser/acceler.c -o build/temp.linux-aarch64-3.6/ast27/Parser/acceler.o
In file included from ast27/Parser/acceler.c:13:0:
ast27/Parser/../Include/pgenheaders.h:8:10: fatal error: Python.h: No such file
or directory #include "Python.h"
^~
compilation terminated.
error: command
'aarch64-linux-gnu-gcc' failed with exit status 1
{noformat}
The missing Python3.h requires python3-dev package:
https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17718) Explicitly set locale in the Dockerfile
Wei-Chiu Chuang created HADOOP-17718:
Summary: Explicitly set locale in the Dockerfile
Key: HADOOP-17718
URL: https://issues.apache.org/jira/browse/HADOOP-17718
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.3.1
Reporter: Wei-Chiu Chuang
When producing the RC bits for 3.3.1, the releasedocmaker step failed.
{noformat}
[INFO] --- exec-maven-plugin:1.3.1:exec (releasedocs) @ hadoop-common ---
Traceback (most recent call last):
File
"/build/source/patchprocess/apache-yetus-0.13.0/bin/../lib/releasedocmaker/releasedocmaker.py",
line 25, in
releasedocmaker.main()
File
"/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/__init__.py",
line 979, in main
JIRA_BASE_URL)
File
"/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py",
line 199, in write_list
self.write_key_raw(jira.get_project(), line)
File
"/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py",
line 170, in write_key_raw
self.base.write(input_string)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 71:
ordinal not in range(128)
{noformat}
It turns out if the script reads jiras containing ascended characters, it can't
write the report.
Inside docker container, the default locale is "ANSI_X3.4-1968". Must set it to
utf-8 to support special characters.
Curious why it wasn't a problem before.
More details:
https://stackoverflow.com/questions/43356982/docker-python-set-utf-8-locale
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17717) Update wildfly openssl to 1.1.3.Final
Wei-Chiu Chuang created HADOOP-17717: Summary: Update wildfly openssl to 1.1.3.Final Key: HADOOP-17717 URL: https://issues.apache.org/jira/browse/HADOOP-17717 Project: Hadoop Common Issue Type: Bug Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang HADOOP-17649 got stalled. IMO we can bump the version to 1.1.3.Final instead, at least, for branch-3.3. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17658) Release Hadoop-Thirdparty 1.1.0
[ https://issues.apache.org/jira/browse/HADOOP-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17658. -- Resolution: Done > Release Hadoop-Thirdparty 1.1.0 > --- > > Key: HADOOP-17658 > URL: https://issues.apache.org/jira/browse/HADOOP-17658 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17671) Run jacc to check for Hadoop's Java API compat
[ https://issues.apache.org/jira/browse/HADOOP-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17671. -- Resolution: Done > Run jacc to check for Hadoop's Java API compat > -- > > Key: HADOOP-17671 > URL: https://issues.apache.org/jira/browse/HADOOP-17671 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: report.html > > > Run Java API Compliance Checker as part of release process to ensure > compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17713) Update apache/hadoop:3 docker image
Wei-Chiu Chuang created HADOOP-17713: Summary: Update apache/hadoop:3 docker image Key: HADOOP-17713 URL: https://issues.apache.org/jira/browse/HADOOP-17713 Project: Hadoop Common Issue Type: Sub-task Reporter: Wei-Chiu Chuang After the release passes the vote, update apache/hadoop:3 docker image by pointing it to 3.3.1 release bits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17669) Port HADOOP-17079, HADOOP-17505 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17669. -- Fix Version/s: 3.3.1 Resolution: Fixed > Port HADOOP-17079, HADOOP-17505 to branch-3.3 > -- > > Key: HADOOP-17669 > URL: https://issues.apache.org/jira/browse/HADOOP-17669 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17704) HADOOP-16916 changed interface SASTokenProvider fields, breaking compatibility between 3.3.0 and 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17704. -- Resolution: Won't Fix Given that the feature wasn't ready in 3.3.0, this is not a real breaking change. > HADOOP-16916 changed interface SASTokenProvider fields, breaking > compatibility between 3.3.0 and 3.3.1 > --- > > Key: HADOOP-17704 > URL: https://issues.apache.org/jira/browse/HADOOP-17704 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Priority: Blocker > > I understand HADOOP-16730/HADOOP-16916 is specifically made for Ranger, but I > am not sure how Ranger consumes this feature. The interface SASTokenProvider > has a number of member fields that changed variable names in HADOOP-16916, > breaking the compatibility between 3.3.0 and 3.3.1. > As a matter of fact, the feature HADOOP-16730 itself was merged in 3.3.0 not > 3.3.1. I just corrected it today. > Raise this jira and mark it as a blocker for 3.3.1. But if this isn't a big > deal then we can downgrade, because, well, this feature was not officially in > the 3.3.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17703) checkcompatibility.py errors out when specifying annotations
[
https://issues.apache.org/jira/browse/HADOOP-17703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17703.
--
Fix Version/s: 3.2.3
3.1.5
3.4.0
3.3.1
Resolution: Fixed
> checkcompatibility.py errors out when specifying annotations
>
>
> Key: HADOOP-17703
> URL: https://issues.apache.org/jira/browse/HADOOP-17703
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> [https://github.com/apache/hadoop/blob/trunk/dev-support/bin/checkcompatibility.py#L178]
> {code:java}
> with file(annotations_path, "w") as f: {code}
> is not a valid Pythonic code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17701) HDFS rebalance commands
[ https://issues.apache.org/jira/browse/HADOOP-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17701. -- Resolution: Invalid Please use the [u...@hadoop.apache.org|mailto:u...@hadoop.apache.org] for usage questions. > HDFS rebalance commands > --- > > Key: HADOOP-17701 > URL: https://issues.apache.org/jira/browse/HADOOP-17701 > Project: Hadoop Common > Issue Type: Bug > Components: hadoop-thirdparty >Reporter: satya kiran >Priority: Minor > > Team , > Having 3 node cluster and one of the node 100% disk utilized . when i am > trying to run rebalance commands by cli using (hdfs balancer -source > 192.168.x.x) and even from ambari its not releasing space . Could you please > help me on this -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17704) HADOOP-16916 changed interface SASTokenProvider fields, breaking compatibility between 3.3.0 and 3.3.1
Wei-Chiu Chuang created HADOOP-17704: Summary: HADOOP-16916 changed interface SASTokenProvider fields, breaking compatibility between 3.3.0 and 3.3.1 Key: HADOOP-17704 URL: https://issues.apache.org/jira/browse/HADOOP-17704 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 3.3.1 Reporter: Wei-Chiu Chuang I understand HADOOP-16730/HADOOP-16916 is specifically made for Ranger, but I am not sure how Ranger consumes this feature. The interface SASTokenProvider has a number of member fields that changed variable names in HADOOP-16916, breaking the compatibility between 3.3.0 and 3.3.1. As a matter of fact, the feature HADOOP-16730 itself was merged in 3.3.0 not 3.3.1. I just corrected it today. Raise this jira and mark it as a blocker for 3.3.1. But if this isn't a big deal then we can downgrade, because, well, this feature was not officially in the 3.3.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17703) checkcompatibility.py errors out when specifying annotations
Wei-Chiu Chuang created HADOOP-17703:
Summary: checkcompatibility.py errors out when specifying
annotations
Key: HADOOP-17703
URL: https://issues.apache.org/jira/browse/HADOOP-17703
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.1, 3.4.0
Reporter: Wei-Chiu Chuang
[https://github.com/apache/hadoop/blob/trunk/dev-support/bin/checkcompatibility.py#L178]
{code:java}
with file(annotations_path, "w") as f: {code}
is not a valid Pythonic code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17702) HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
Wei-Chiu Chuang created HADOOP-17702:
Summary: HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
Key: HADOOP-17702
URL: https://issues.apache.org/jira/browse/HADOOP-17702
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.3.1
Reporter: Wei-Chiu Chuang
AzureBlobFileSystemStore is a Public, Evolving class, by contract can't break
compatibility between maintenance releases.
HADOOP-17065 changed its constructor signature from
{noformat}
public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
Configuration configuration,
AbfsCounters abfsCounters) throws IOException
{
{noformat}
to
{noformat}
public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
Configuration configuration,
AbfsCounters abfsCounters) throws IOException
{
{noformat}
between 3.3.0 and 3.3.1.
cc: [~mehakmeetSingh], [~tmarquardt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17107) hadoop-azure parallel tests not working on recent JDKs
[ https://issues.apache.org/jira/browse/HADOOP-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17107. -- Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed > hadoop-azure parallel tests not working on recent JDKs > -- > > Key: HADOOP-17107 > URL: https://issues.apache.org/jira/browse/HADOOP-17107 > Project: Hadoop Common > Issue Type: Bug > Components: build, fs/azure >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1, 3.4.0 > > > recent JDKs are failing to run the wasb or abfs parallel test runs -unable to > instantiate the javascript engine. > Maybe it's been cut from the JVM or the ant script task can't bind to it. > Fix is as HADOOP-14696 -use our own plugin to set up the test dirs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17665) Ignore missing keystore configuration in reloading mechanism
[ https://issues.apache.org/jira/browse/HADOOP-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17665. -- Resolution: Fixed > Ignore missing keystore configuration in reloading mechanism > - > > Key: HADOOP-17665 > URL: https://issues.apache.org/jira/browse/HADOOP-17665 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Borislav Iordanov >Assignee: Borislav Iordanov >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When there is no configuration of keystore/truststore location, the reload > mechanism should be disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17142) Fix outdated properties of journal node when perform rollback
[
https://issues.apache.org/jira/browse/HADOOP-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17142.
--
Resolution: Fixed
> Fix outdated properties of journal node when perform rollback
> -
>
> Key: HADOOP-17142
> URL: https://issues.apache.org/jira/browse/HADOOP-17142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Deegue
>Priority: Minor
> Fix For: 3.3.1, 3.4.0, 3.2.2
>
>
> When rollback HDFS cluster, properties in JNStorage won't be refreshed after
> the storage dir changed. It leads to exceptions when starting namenode.
> The exception like:
> {code:java}
> 2020-07-09 19:04:12,810 FATAL [IPC Server handler 105 on 8022]
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error:
> recoverUnfinalizedSegments failed for required journal
> (JournalAndStream(mgr=QJM to [10.0.118.217:8485, 10.0.117.208:8485,
> 10.0.118.179:8485], stream=null))
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
> exceptions to achieve quorum size 2/3. 3 exceptions thrown:
> 10.0.118.217:8485: Incompatible namespaceID for journal Storage Directory
> /mnt/vdc-11176G-0/dfs/jn/nameservicetest1: NameNode has nsId 647617129 but
> storage has nsId 0
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.checkConsistentNamespace(JNStorage.java:236)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.newEpoch(Journal.java:300)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.newEpoch(JournalNodeRpcServer.java:136)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.newEpoch(QJournalProtocolServerSideTranslatorPB.java:133)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2274)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17142) Fix outdated properties of journal node when perform rollback
[
https://issues.apache.org/jira/browse/HADOOP-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reopened HADOOP-17142:
--
> Fix outdated properties of journal node when perform rollback
> -
>
> Key: HADOOP-17142
> URL: https://issues.apache.org/jira/browse/HADOOP-17142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Deegue
>Priority: Minor
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
>
> When rollback HDFS cluster, properties in JNStorage won't be refreshed after
> the storage dir changed. It leads to exceptions when starting namenode.
> The exception like:
> {code:java}
> 2020-07-09 19:04:12,810 FATAL [IPC Server handler 105 on 8022]
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error:
> recoverUnfinalizedSegments failed for required journal
> (JournalAndStream(mgr=QJM to [10.0.118.217:8485, 10.0.117.208:8485,
> 10.0.118.179:8485], stream=null))
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
> exceptions to achieve quorum size 2/3. 3 exceptions thrown:
> 10.0.118.217:8485: Incompatible namespaceID for journal Storage Directory
> /mnt/vdc-11176G-0/dfs/jn/nameservicetest1: NameNode has nsId 647617129 but
> storage has nsId 0
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.checkConsistentNamespace(JNStorage.java:236)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.newEpoch(Journal.java:300)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.newEpoch(JournalNodeRpcServer.java:136)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.newEpoch(QJournalProtocolServerSideTranslatorPB.java:133)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2274)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
