[jira] [Commented] (HADOOP-18980) S3A credential provider remapping: make extensible
[ https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816513#comment-17816513 ] Viraj Jasani commented on HADOOP-18980: --- Addressed edge cases with addendum PR: [https://github.com/apache/hadoop/pull/6546] > S3A credential provider remapping: make extensible > -- > > Key: HADOOP-18980 > URL: https://issues.apache.org/jira/browse/HADOOP-18980 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > s3afs will now remap the common com.amazonaws credential providers to > equivalents in the v2 sdk or in hadoop-aws > We could do the same for third party credential providers by taking a > key=value list in a configuration property and adding to the map. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18980) S3A credential provider remapping: make extensible
[ https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816512#comment-17816512 ] ASF GitHub Bot commented on HADOOP-18980: - virajjasani opened a new pull request, #6546: URL: https://github.com/apache/hadoop/pull/6546 Jira: HADOOP-18980 > S3A credential provider remapping: make extensible > -- > > Key: HADOOP-18980 > URL: https://issues.apache.org/jira/browse/HADOOP-18980 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > s3afs will now remap the common com.amazonaws credential providers to > equivalents in the v2 sdk or in hadoop-aws > We could do the same for third party credential providers by taking a > key=value list in a configuration property and adding to the map. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-18980. Invalid inputs for getTrimmedStringCollectionSplitByEquals (ADDENDUM) [hadoop]
virajjasani opened a new pull request, #6546: URL: https://github.com/apache/hadoop/pull/6546 Jira: HADOOP-18980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18125) Utility to identify git commit / Jira fixVersion discrepancies for RC preparation
[ https://issues.apache.org/jira/browse/HADOOP-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18125: Component/s: build Hadoop Flags: Reviewed Target Version/s: 3.3.3, 3.4.0 (was: 3.3.3) Affects Version/s: 3.3.3 3.2.3 2.10.2 3.4.0 > Utility to identify git commit / Jira fixVersion discrepancies for RC > preparation > - > > Key: HADOOP-18125 > URL: https://issues.apache.org/jira/browse/HADOOP-18125 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 3.4.0, 2.10.2, 3.2.3, 3.3.3 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > As part of RC preparation, we need to identify all git commits that landed > on release branch, however their corresponding Jira is either not resolved > yet or does not contain expected fixVersions. Only when we have git commits > and corresponding Jiras with expected fixVersion resolved, we get all such > Jiras included in auto-generated CHANGES.md as per Yetus changelog generator. > Proposal of this Jira is to provide such script that can be useful for all > upcoming RC preparations and list down all Jiras where we need manual > intervention. This utility script should use Jira API to retrieve individual > fields and use git log to loop through commit history. > The script should identify these issues: > # commit is reverted as per commit message > # commit does not contain Jira number format (e.g. HADOOP- / HDFS- > etc) in message > # Jira does not have expected fixVersion > # Jira has expected fixVersion, but it is not yet resolved > # Jira has release corresponding fixVersion and is resolved, but no > corresponding commit yet found > It can take inputs as: > # First commit hash to start excluding commits from history > # Fix Version > # JIRA Project Name > # Path of project's working dir > # Jira server url -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18014) CallerContext should not include some characters
[ https://issues.apache.org/jira/browse/HADOOP-18014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18014: Component/s: ipc Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.4.0 Affects Version/s: 3.3.5 3.4.0 > CallerContext should not include some characters > > > Key: HADOOP-18014 > URL: https://issues.apache.org/jira/browse/HADOOP-18014 > Project: Hadoop Common > Issue Type: Task > Components: ipc >Affects Versions: 3.4.0, 3.3.5 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18061) Update the year to 2022
[ https://issues.apache.org/jira/browse/HADOOP-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18061: Component/s: common Target Version/s: 3.3.2, 3.2.3, 2.10.2, 3.4.0 Affects Version/s: 3.3.2 3.2.3 2.10.2 3.4.0 > Update the year to 2022 > --- > > Key: HADOOP-18061 > URL: https://issues.apache.org/jira/browse/HADOOP-18061 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0, 2.10.2, 3.2.3, 3.3.2 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 40m > Remaining Estimate: 0h > > Update the year to 2022 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17946) Update commons-lang to 3.12.0
[ https://issues.apache.org/jira/browse/HADOOP-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17946: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > Update commons-lang to 3.12.0 > - > > Key: HADOOP-17946 > URL: https://issues.apache.org/jira/browse/HADOOP-17946 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.3.2 >Reporter: Sean Busbey >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > our commons-lang3 dependency is currently 3.7, which is nearly 4 years old. > latest right now is 3.12 and there are at least some fixes that would make us > more robust on JDKs newer than openjdk8 (e.g. LANG-1384. [release notes > indicate 3.9 is the first to support > jdk11|https://commons.apache.org/proper/commons-lang/changes-report.html]). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17955) Bump netty to the latest 4.1.68
[ https://issues.apache.org/jira/browse/HADOOP-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17955: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Bump netty to the latest 4.1.68 > --- > > Key: HADOOP-17955 > URL: https://issues.apache.org/jira/browse/HADOOP-17955 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Netty 4.1.68 fixes the following vulnerabilities. > * Bzip2Decoder doesn't allow setting size restrictions for decompressed data > (#CVE-2021-37136) > * SnappyFrameDecoder doesn't restrict chunk length any may buffer skippable > chunks in an unnecessary way (#CVE-2021-37137) > For more details: [https://netty.io/news/2021/09/09/4-1-68-Final.html] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17834) Bump aliyun-sdk-oss to 3.13.0
[ https://issues.apache.org/jira/browse/HADOOP-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17834: Component/s: build common Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Bump aliyun-sdk-oss to 3.13.0 > - > > Key: HADOOP-17834 > URL: https://issues.apache.org/jira/browse/HADOOP-17834 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Bump aliyun-sdk-oss to 3.13.0 in order to remove transitive dependency on > jdom 1.1. > Ref: > https://issues.apache.org/jira/browse/HADOOP-17820?focusedCommentId=17390206=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17390206. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17743) Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects
[ https://issues.apache.org/jira/browse/HADOOP-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17743: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, > hadoop-tools and cloud-storage projects > - > > Key: HADOOP-17743 > URL: https://issues.apache.org/jira/browse/HADOOP-17743 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17683) Update commons-io to 2.8.0
[ https://issues.apache.org/jira/browse/HADOOP-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17683: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Update commons-io to 2.8.0 > -- > > Key: HADOOP-17683 > URL: https://issues.apache.org/jira/browse/HADOOP-17683 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > https://nvd.nist.gov/vuln/detail/CVE-2021-29425 > In Apache Commons IO before 2.7, When invoking the method > FileNameUtils.normalize with an improper input string, like "//../foo", or > "\\..\foo", the result would be the same value, thus possibly providing > access to files in the parent directory, but not further above (thus > "limited" path traversal), if the calling code would use the result to > construct a path value. > We don't use this API in the Hadoop code, but it's still good to update > anyway (we're on 2.5, which is 4 years old) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17625) Update to Jetty 9.4.39
[ https://issues.apache.org/jira/browse/HADOOP-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17625: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Update to Jetty 9.4.39 > -- > > Key: HADOOP-17625 > URL: https://issues.apache.org/jira/browse/HADOOP-17625 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17624) Remove any rocksdb exclusion code
[ https://issues.apache.org/jira/browse/HADOOP-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17624: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 > Remove any rocksdb exclusion code > - > > Key: HADOOP-17624 > URL: https://issues.apache.org/jira/browse/HADOOP-17624 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > RocksDB was added as part of Ozone runtime, and we added code to exclude > RocksDB from the core runtime. Now that Ozone is a separate project, we > should remove anything referencing RocksDB now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17622) Avoid usage of deprecated IOUtils#cleanup API
[ https://issues.apache.org/jira/browse/HADOOP-17622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17622: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Avoid usage of deprecated IOUtils#cleanup API > - > > Key: HADOOP-17622 > URL: https://issues.apache.org/jira/browse/HADOOP-17622 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We can replace usage of deprecated API IOUtils#cleanup() with > IOUtils#cleanupWithLogger(). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17614) Bump netty to the latest 4.1.61
[ https://issues.apache.org/jira/browse/HADOOP-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17614: Component/s: build common Target Version/s: 3.2.3, 3.3.1, 3.4.0 > Bump netty to the latest 4.1.61 > --- > > Key: HADOOP-17614 > URL: https://issues.apache.org/jira/browse/HADOOP-17614 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0, 3.2.3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2.5h > Remaining Estimate: 0h > > For more details: https://netty.io/news/2021/03/09/4-1-60-Final.html > Actually, just yesterday there's a new version 4.1.61. > https://netty.io/news/2021/03/30/4-1-61-Final.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17030) Remove unused joda-time
[ https://issues.apache.org/jira/browse/HADOOP-17030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17030: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Remove unused joda-time > --- > > Key: HADOOP-17030 > URL: https://issues.apache.org/jira/browse/HADOOP-17030 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.4.0 > > Attachments: HADOOP-17030.001.patch > > > Joda-time is defined in the hadoop-project/pom.xml but it's not used > anywhere. It should be easy to remove it without problems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16990) Update Mockserver
[ https://issues.apache.org/jira/browse/HADOOP-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16990: Component/s: hdfs-client Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.2.2, 3.4.0 Affects Version/s: 3.3.1 3.2.2 3.4.0 > Update Mockserver > - > > Key: HADOOP-16990 > URL: https://issues.apache.org/jira/browse/HADOOP-16990 > Project: Hadoop Common > Issue Type: Task > Components: hdfs-client >Affects Versions: 3.2.2, 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Attila Doroszlai >Priority: Major > Fix For: 3.2.2, 3.3.1, 3.4.0 > > Attachments: HADOOP-16990-branch-3.1.004.patch, > HADOOP-16990-branch-3.3.002.patch, HADOOP-16990.001.patch, > HDFS-15620-branch-3.3-addendum.patch > > > We are on Mockserver 3.9.2 which is more than 5 years old. Time to update. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16866) Upgrade spotbugs to 4.0.6
[ https://issues.apache.org/jira/browse/HADOOP-16866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16866: Component/s: build command Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Upgrade spotbugs to 4.0.6 > - > > Key: HADOOP-16866 > URL: https://issues.apache.org/jira/browse/HADOOP-16866 > Project: Hadoop Common > Issue Type: Task > Components: build, command >Affects Versions: 3.4.0 >Reporter: Tsuyoshi Ozawa >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.4.0 > > > [https://github.com/spotbugs/spotbugs/releases] > spotbugs 4.0.0 is now released. > > We can upgrade spotbugs' version to: > 1. 3.1.12 (conservative option) > 2. 4.0.0 (which might includes incompatible changes, according to the > migration guide: [https://spotbugs.readthedocs.io/en/stable/migration.html]) > > Step by step approach is also acceptable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17234) Add .asf.yaml to allow github and jira integration
[ https://issues.apache.org/jira/browse/HADOOP-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17234: Component/s: build common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Add .asf.yaml to allow github and jira integration > -- > > Key: HADOOP-17234 > URL: https://issues.apache.org/jira/browse/HADOOP-17234 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > As of now the default for github is set only to worklog, To enable link and > label, We need to add this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17055) Remove residual code of Ozone
[ https://issues.apache.org/jira/browse/HADOOP-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17055: Component/s: ozone common Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Remove residual code of Ozone > - > > Key: HADOOP-17055 > URL: https://issues.apache.org/jira/browse/HADOOP-17055 > Project: Hadoop Common > Issue Type: Task > Components: common, ozone >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18005) Correct log format for LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18005: Component/s: security Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > Correct log format for LdapGroupsMapping > > > Key: HADOOP-18005 > URL: https://issues.apache.org/jira/browse/HADOOP-18005 > Project: Hadoop Common > Issue Type: Wish > Components: security >Affects Versions: 3.4.0, 3.3.2 >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Attachments: image-2021-11-12-09-20-58-968.png, > image-2021-11-12-09-22-12-454.png > > Time Spent: 50m > Remaining Estimate: 0h > > Correct log format for LdapGroupsMapping. > !image-2021-11-12-09-22-12-454.png|width=557,height=101! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15348) S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly
[
https://issues.apache.org/jira/browse/HADOOP-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-15348:
---
Assignee: Steve Loughran
> S3A Input Stream bytes read counter isn't getting through to
> StorageStatistics/instrumentation properly
> ---
>
> Key: HADOOP-15348
> URL: https://issues.apache.org/jira/browse/HADOOP-15348
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.4.0
>
>
> TL;DR: we should have common storage statistics for bytes read and bytes
> written, and S3A should use them in its instrumentation and have enum names
> to match.
> # in the S3AInputStream we call
> {{S3AInstrumentation.StreamStatistics.bytesRead(long)}}, which adds the
> amount to {{bytesRead}}, in a read(), readFully, or forward seek() reading in
> data
> # and in {{S3AInstrumentation.mergeInputStreamStatistics}}, that is pulled
> into streamBytesRead.
> # which has a Statistics name of ""stream_bytes_read"
> # but that is served up in the Storage statistics as
> "STREAM_SEEK_BYTES_READ", which is the wrong name.
> # and there isn't a common name for the counter across other filesystems.
> For now: people can use the wrong name in the enum; we may want to think
> about retaining it when adding the correct name. And maybe add a
> @Evolving/@LimitedPrivate scope pair to the enum
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[ https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816495#comment-17816495 ] ASF GitHub Bot commented on HADOOP-19047: - shameersss1 commented on PR #6468: URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1938151034 @steveloughran - Thanks a lot for review comments, I have addressed the comments with the new commit `d8db729e5568df5dc920604eff8167a575a5894c` 1. Added test in ITestTerasortOnS3A 2. Fixed import ordering 3. Addressed other minor comments > Support InMemory Tracking Of S3A Magic Commits > -- > > Key: HADOOP-19047 > URL: https://issues.apache.org/jira/browse/HADOOP-19047 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The following are the operations which happens within a Task when it uses S3A > Magic Committer. > *During closing of stream* > 1. A 0-byte file with a same name of the original file is uploaded to S3 > using PUT operation. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152] > for more information. This is done so that the downstream application like > Spark could get the size of the file which is being written. > 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176] > for more information. > *During TaskCommit* > 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number > of metadata file in S3 if a single task writes to 'x' files) are read and > rewritten to S3 as a single metadata file. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201] > for more information > Since these operations happens with the Task JVM, We could optimize as well > as save cost by storing these information in memory when Task memory usage is > not a constraint. Hence the proposal here is to introduce a new MagicCommit > Tracker called "InMemoryMagicCommitTracker" which will store the > 1. Metadata of MPU in memory till the Task is committed > 2. Store the size of the file which can be used by the downstream application > to get the file size before it is committed/visible to the output path. > This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call > given a Task writes only 1 file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on PR #6468: URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1938151034 @steveloughran - Thanks a lot for review comments, I have addressed the comments with the new commit `d8db729e5568df5dc920604eff8167a575a5894c` 1. Added test in ITestTerasortOnS3A 2. Fixed import ordering 3. Addressed other minor comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18159) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
[
https://issues.apache.org/jira/browse/HADOOP-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-18159:
---
Assignee: André F.
> Certificate doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> --
>
> Key: HADOOP-18159
> URL: https://issues.apache.org/jira/browse/HADOOP-18159
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.1, 3.3.2, 3.3.3
> Environment: hadoop 3.3.1
> httpclient 4.5.13
> JDK8
>Reporter: André F.
>Assignee: André F.
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. If you see this error message when trying to use s3a:// or gs:// URLs,
> look for copies of cos_api-bundle.jar on your classpath and remove them.
> Libraries which include shaded apache httpclient libraries
> (hadoop-client-runtime.jar, aws-java-sdk-bundle.jar,
> gcs-connector-shaded.jar, cos_api-bundle.jar) all load and use the unshaded
> resource mozilla/public-suffix-list.txt. If an out of date version of this is
> found on the classpath first, attempts to negotiate TLS connections may fail
> with the error "Certificate doesn't match any of the subject alternative
> names".
> In a hadoop installation, you can use the findclass tool to track down where
> the public-suffix-list.txt is coming from.
> {code}
> hadoop org.apache.hadoop.util.FindClass locate mozilla/public-suffix-list.txt
> {code}
> So far, the cos_api-bundle-5.6.19.jar appears to be the source of this
> problem.
>
> h2. bug report
> Trying to run any job after bumping our Spark version (which is now using
> Hadoop 3.3.1), lead us to the current exception while reading files on s3:
> {code:java}
> org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a:///.parquet: com.amazonaws.SdkClientException: Unable to
> execute HTTP request: Certificate for doesn't match
> any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]:
> Unable to execute HTTP request: Certificate for doesn't match any of
> the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3351)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4277)
> at {code}
>
> {code:java}
> Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy16.connect(Unknown Source)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at
> com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
>
[jira] [Assigned] (HADOOP-17142) Fix outdated properties of journal node when perform rollback
[
https://issues.apache.org/jira/browse/HADOOP-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-17142:
---
Assignee: Deegue
> Fix outdated properties of journal node when perform rollback
> -
>
> Key: HADOOP-17142
> URL: https://issues.apache.org/jira/browse/HADOOP-17142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Deegue
>Assignee: Deegue
>Priority: Minor
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
>
> When rollback HDFS cluster, properties in JNStorage won't be refreshed after
> the storage dir changed. It leads to exceptions when starting namenode.
> The exception like:
> {code:java}
> 2020-07-09 19:04:12,810 FATAL [IPC Server handler 105 on 8022]
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error:
> recoverUnfinalizedSegments failed for required journal
> (JournalAndStream(mgr=QJM to [10.0.118.217:8485, 10.0.117.208:8485,
> 10.0.118.179:8485], stream=null))
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
> exceptions to achieve quorum size 2/3. 3 exceptions thrown:
> 10.0.118.217:8485: Incompatible namespaceID for journal Storage Directory
> /mnt/vdc-11176G-0/dfs/jn/nameservicetest1: NameNode has nsId 647617129 but
> storage has nsId 0
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.checkConsistentNamespace(JNStorage.java:236)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.newEpoch(Journal.java:300)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.newEpoch(JournalNodeRpcServer.java:136)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.newEpoch(QJournalProtocolServerSideTranslatorPB.java:133)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2274)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18118) Fix KMS Accept Queue Size default value to 500
[ https://issues.apache.org/jira/browse/HADOOP-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18118: --- Assignee: guophilipse > Fix KMS Accept Queue Size default value to 500 > -- > > Key: HADOOP-18118 > URL: https://issues.apache.org/jira/browse/HADOOP-18118 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > From HADOOP-15638,`hadoop.http.socket.backlog.size` was set as 500 by default > ,we can change code default value to keep consistent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16930) Add com.amazonaws.auth.profile.ProfileCredentialsProvider to hadoop-aws docs
[
https://issues.apache.org/jira/browse/HADOOP-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-16930:
---
Assignee: Nicholas Chammas
> Add com.amazonaws.auth.profile.ProfileCredentialsProvider to hadoop-aws docs
>
>
> Key: HADOOP-16930
> URL: https://issues.apache.org/jira/browse/HADOOP-16930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs/s3
>Affects Versions: 3.2.1
>Reporter: Nicholas Chammas
>Assignee: Nicholas Chammas
>Priority: Minor
> Fix For: 3.4.0
>
>
> There is a very, very useful S3A authentication method that is not currently
> documented: {{com.amazonaws.auth.profile.ProfileCredentialsProvider}}
> This provider lets you source your AWS credentials from a shared credentials
> file, typically stored under {{~/.aws/credentials}}, using a [named
> profile|https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html].
> All you need is to set the {{AWS_PROFILE}} environment variable, and the
> provider will get the appropriate credentials for you.
> I discovered this from my coworkers, but cannot find it in the docs for
> hadoop-aws. I'd expect to see it at least mentioned in [this
> section|https://hadoop.apache.org/docs/r2.9.2/hadoop-aws/tools/hadoop-aws/index.html#S3A_Authentication_methods].
> It should probably be added to the docs for every minor release that
> supports it, which I'd guess includes 2.8 on up.
> (This provider should probably also be added to the default list of
> credential provider classes, but we can address that in another ticket. I can
> say that at least in 2.9.2, it's not in the default list.)
> (This is not to be confused with
> {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, which serves a
> completely different purpose.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-15327: Component/s: common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Upgrade MR ShuffleHandler to use Netty4 > --- > > Key: HADOOP-15327 > URL: https://issues.apache.org/jira/browse/HADOOP-15327 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Xiaoyu Yao >Assignee: Szilard Nemeth >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch, > HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch, > HADOOP-15327.005.patch, > getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log, > hades-results-20221108.zip, testfailure-testMapFileAccess-emptyresponse.zip, > testfailure-testReduceFromPartialMem.zip > > Time Spent: 11.5h > Remaining Estimate: 0h > > This way, we can remove the dependencies on the netty3 (jboss.netty) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17526: Component/s: common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18018) unguava: remove Preconditions from hadoop-tools modules
[ https://issues.apache.org/jira/browse/HADOOP-18018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18018: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > unguava: remove Preconditions from hadoop-tools modules > --- > > Key: HADOOP-18018 > URL: https://issues.apache.org/jira/browse/HADOOP-18018 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Replace guava Preconditions by internal implementations that rely on java8+ > APIs in the hadoop.util for all modules in hadoop-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17970) unguava: remove Preconditions from hdfs-projects module
[ https://issues.apache.org/jira/browse/HADOOP-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17970: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > unguava: remove Preconditions from hdfs-projects module > --- > > Key: HADOOP-17970 > URL: https://issues.apache.org/jira/browse/HADOOP-17970 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Replace guava Preconditions by internal implementations that rely on java8+ > APIs in the hadoop.util for all modules in hadoop-hdfs-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17952) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-common-project modules
[ https://issues.apache.org/jira/browse/HADOOP-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17952: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.2.3, 2.10.2, 3.3.1, 3.4.0 Affects Version/s: 3.2.3 2.10.2 3.3.1 3.4.0 > Replace Guava VisibleForTesting by Hadoop's own annotation in > hadoop-common-project modules > --- > > Key: HADOOP-17952 > URL: https://issues.apache.org/jira/browse/HADOOP-17952 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0, 2.10.2, 3.2.3 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 2.10.2, 3.2.3 > > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17951) AccessPoint verifyBucketExistsV2 always returns false
[ https://issues.apache.org/jira/browse/HADOOP-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17951: Component/s: fs/s3 > AccessPoint verifyBucketExistsV2 always returns false > - > > Key: HADOOP-17951 > URL: https://issues.apache.org/jira/browse/HADOOP-17951 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Bogdan Stolojan >Assignee: Bogdan Stolojan >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Turns out the implementation added for accesspoints to check that they exist > was always returning "false". (Is it an improvement from the SDK which always > returned "true"? nope). > The fix for this is trivial as we need to check for the presence of a message > "Could not access through this access point" or 404 if the AP doesn't exist. > Otherwise it does. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17930) implement non-guava Precondition checkState
[
https://issues.apache.org/jira/browse/HADOOP-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17930:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.2.4, 3.3.2, 3.4.0
> implement non-guava Precondition checkState
> ---
>
> Key: HADOOP-17930
> URL: https://issues.apache.org/jira/browse/HADOOP-17930
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Affects Versions: 3.4.0, 3.2.3, 3.3.2
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to add the implementation {{checkState}} to the existing class
> {{org.apache.hadoop.util.Preconditions}}
> +The plan is as follows+
> * implement {{org.apache.hadoop.util.Preconditions.checkState}} with the
> minimum set of interface used in the current hadoop repo.
> * we can replace {{guava.Preconditions}} by
> {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been
> implemented (both this jira and HADOOP-17929 are complete).
> * We need the change to be easily to be backported in 3.x.
> previous jiras:
> * HADOOP-17126 was created to implement CheckNotNull.
> * HADOOP-17929 implementing checkArgument.
> CC: [~ste...@apache.org], [~vjasani]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17951) AccessPoint verifyBucketExistsV2 always returns false
[ https://issues.apache.org/jira/browse/HADOOP-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17951: Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > AccessPoint verifyBucketExistsV2 always returns false > - > > Key: HADOOP-17951 > URL: https://issues.apache.org/jira/browse/HADOOP-17951 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Bogdan Stolojan >Assignee: Bogdan Stolojan >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Turns out the implementation added for accesspoints to check that they exist > was always returning "false". (Is it an improvement from the SDK which always > returned "true"? nope). > The fix for this is trivial as we need to check for the presence of a message > "Could not access through this access point" or 404 if the AP doesn't exist. > Otherwise it does. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17929) implement non-guava Precondition checkArgument
[
https://issues.apache.org/jira/browse/HADOOP-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17929:
Component/s: command
Hadoop Flags: Reviewed
Target Version/s: 3.2.4, 3.3.2, 3.4.0
> implement non-guava Precondition checkArgument
> --
>
> Key: HADOOP-17929
> URL: https://issues.apache.org/jira/browse/HADOOP-17929
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: command
>Affects Versions: 3.4.0, 3.2.3, 3.3.2
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to add the implementation {{checkArgument}} to the existing
> class {{org.apache.hadoop.util.Preconditions}}
> +The plan is as follows+
> * implement {{org.apache.hadoop.util.Preconditions.checkArgument}} with the
> minimum set of interface used in the current hadoop repo.
> * we can replace {{guava.Preconditions}} by
> {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been
> implemented.
> * We need the change to be easily to be backported in 3.x.
> A previous jira HADOOP-17126 was created to replace CheckNotNull.
> HADOOP-17930 is created to implement checkState.
> CC: [~ste...@apache.org], [~vjasani]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17910) [JDK 17] TestNetUtils fails
[
https://issues.apache.org/jira/browse/HADOOP-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17910:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> [JDK 17] TestNetUtils fails
> ---
>
> Key: HADOOP-17910
> URL: https://issues.apache.org/jira/browse/HADOOP-17910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> TestNetUtils#testInvalidAddress fails.
> {noformat}
> [INFO] Running org.apache.hadoop.net.TestNetUtils
> [ERROR] Tests run: 48, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 4.469 s <<< FAILURE! - in org.apache.hadoop.net.TestNetUtils
> [ERROR] testInvalidAddress(org.apache.hadoop.net.TestNetUtils) Time elapsed:
> 0.386 s <<< FAILURE!
> java.lang.AssertionError:
> Expected to find 'invalid-test-host:0' but got unexpected exception:
> java.net.UnknownHostException: invalid-test-host/:0
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:592)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:551)
> at
> org.apache.hadoop.net.TestNetUtils.testInvalidAddress(TestNetUtils.java:109)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> at
> org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:396)
> at
> org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:373)
> at
> org.apache.hadoop.net.TestNetUtils.testInvalidAddress(TestNetUtils.java:116)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[jira] [Updated] (HADOOP-17814) Provide fallbacks for identity/cost providers and backoff enable
[ https://issues.apache.org/jira/browse/HADOOP-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17814: Component/s: ipc Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Provide fallbacks for identity/cost providers and backoff enable > > > Key: HADOOP-17814 > URL: https://issues.apache.org/jira/browse/HADOOP-17814 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > This sub-task is to provide default properties for identity-provider.impl, > cost-provider.impl and backoff.enable such that if properties with port is > not configured, we can fallback to default property (port-less). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16272) Update HikariCP to 4.0.3
[ https://issues.apache.org/jira/browse/HADOOP-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16272: Component/s: build common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Update HikariCP to 4.0.3 > > > Key: HADOOP-16272 > URL: https://issues.apache.org/jira/browse/HADOOP-16272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17715) ABFS: Append blob tests with non HNS accounts fail
[ https://issues.apache.org/jira/browse/HADOOP-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17715: Component/s: fs/azure Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > ABFS: Append blob tests with non HNS accounts fail > -- > > Key: HADOOP-17715 > URL: https://issues.apache.org/jira/browse/HADOOP-17715 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0, 3.3.2 >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Append blob tests with non HNS accounts fail. > # The script to run the tests should ensure that append blob tests with non > HNS account don't execute > # Should have proper documentation mentioning that append blob is allowed > only for HNS accounts -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17331) [JDK 15] TestDNS fails by UncheckedIOException
[
https://issues.apache.org/jira/browse/HADOOP-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17331:
Component/s: test
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> [JDK 15] TestDNS fails by UncheckedIOException
> --
>
> Key: HADOOP-17331
> URL: https://issues.apache.org/jira/browse/HADOOP-17331
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: test
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After [JDK-8235783|https://bugs.openjdk.java.net/browse/JDK-8235783],
> DatagramSocket::connect throws UncheckedIOException if connect fails.
> {noformat}
> [INFO] Running org.apache.hadoop.net.TestDNS
> [ERROR] Tests run: 12, Failures: 0, Errors: 5, Skipped: 0, Time elapsed:
> 0.403 s <<< FAILURE! - in org.apache.hadoop.net.TestDNS
> [ERROR] testNullDnsServer(org.apache.hadoop.net.TestDNS) Time elapsed: 0.134
> s <<< ERROR!
> java.io.UncheckedIOException: java.net.SocketException: Unsupported address
> type
> at
> java.base/sun.nio.ch.DatagramSocketAdaptor.connect(DatagramSocketAdaptor.java:120)
> at java.base/java.net.DatagramSocket.connect(DatagramSocket.java:341)
> {noformat}
> Full error log:
> https://gist.github.com/aajisaka/2a24cb2b110cc3d19f7dec6256db6844
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17666) Update LICENSE for 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17666: Component/s: common > Update LICENSE for 3.3.1 > > > Key: HADOOP-17666 > URL: https://issues.apache.org/jira/browse/HADOOP-17666 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available, release-blocker > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Before release, do another round of check for the LICENSE file to make sure > the dependency versions are updated correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17666) Update LICENSE for 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17666: Affects Version/s: 3.3.1 3.4.0 > Update LICENSE for 3.3.1 > > > Key: HADOOP-17666 > URL: https://issues.apache.org/jira/browse/HADOOP-17666 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available, release-blocker > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Before release, do another round of check for the LICENSE file to make sure > the dependency versions are updated correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17665) Ignore missing keystore configuration in reloading mechanism
[ https://issues.apache.org/jira/browse/HADOOP-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17665: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Ignore missing keystore configuration in reloading mechanism > - > > Key: HADOOP-17665 > URL: https://issues.apache.org/jira/browse/HADOOP-17665 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Borislav Iordanov >Assignee: Borislav Iordanov >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When there is no configuration of keystore/truststore location, the reload > mechanism should be disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17653) Do not use guava's Files.createTempDir()
[ https://issues.apache.org/jira/browse/HADOOP-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17653: Component/s: common Target Version/s: 3.3.1, 3.4.0 > Do not use guava's Files.createTempDir() > > > Key: HADOOP-17653 > URL: https://issues.apache.org/jira/browse/HADOOP-17653 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816488#comment-17816488
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485758682
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
+
+ // stores taskAttemptId to commit data mapping
+ private static Map>
Review Comment:
ack for final.
> I do think they should use weak/soft references,
Is this required ? Given that we proactively remove the entries from HashMap
when the task commits or aborts. Since it is not referenced any where, when gc
happens it reclaims the memory.
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485758682
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
+
+ // stores taskAttemptId to commit data mapping
+ private static Map>
Review Comment:
ack for final.
> I do think they should use weak/soft references,
Is this required ? Given that we proactively remove the entries from HashMap
when the task commits or aborts. Since it is not referenced any where, when gc
happens it reclaims the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[ https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816482#comment-17816482 ] ASF GitHub Bot commented on HADOOP-19047: - shameersss1 commented on code in PR #6468: URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485746226 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTrackerUtils.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a.commit.magic; + +import org.apache.hadoop.conf.Configuration; Review Comment: ack > Support InMemory Tracking Of S3A Magic Commits > -- > > Key: HADOOP-19047 > URL: https://issues.apache.org/jira/browse/HADOOP-19047 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The following are the operations which happens within a Task when it uses S3A > Magic Committer. > *During closing of stream* > 1. A 0-byte file with a same name of the original file is uploaded to S3 > using PUT operation. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152] > for more information. This is done so that the downstream application like > Spark could get the size of the file which is being written. > 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176] > for more information. > *During TaskCommit* > 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number > of metadata file in S3 if a single task writes to 'x' files) are read and > rewritten to S3 as a single metadata file. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201] > for more information > Since these operations happens with the Task JVM, We could optimize as well > as save cost by storing these information in memory when Task memory usage is > not a constraint. Hence the proposal here is to introduce a new MagicCommit > Tracker called "InMemoryMagicCommitTracker" which will store the > 1. Metadata of MPU in memory till the Task is committed > 2. Store the size of the file which can be used by the downstream application > to get the file size before it is committed/visible to the output path. > This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call > given a Task writes only 1 file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468: URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485746226 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTrackerUtils.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a.commit.magic; + +import org.apache.hadoop.conf.Configuration; Review Comment: ack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816481#comment-17816481
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485745785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java:
##
@@ -118,76 +103,21 @@ public boolean outputImmediatelyVisible() {
/**
* Complete operation: generate the final commit data, put it.
- * @param uploadId Upload ID
- * @param parts list of parts
+ *
+ * @param uploadId Upload ID
+ * @param partslist of parts
* @param bytesWritten bytes written
* @param iostatistics nullable IO statistics
* @return false, indicating that the commit must fail.
- * @throws IOException any IO problem.
+ * @throws IOException any IO problem.
Review Comment:
ack
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485745785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java:
##
@@ -118,76 +103,21 @@ public boolean outputImmediatelyVisible() {
/**
* Complete operation: generate the final commit data, put it.
- * @param uploadId Upload ID
- * @param parts list of parts
+ *
+ * @param uploadId Upload ID
+ * @param partslist of parts
* @param bytesWritten bytes written
* @param iostatistics nullable IO statistics
* @return false, indicating that the commit must fail.
- * @throws IOException any IO problem.
+ * @throws IOException any IO problem.
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816480#comment-17816480
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485744738
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
+
+ // stores taskAttemptId to commit data mapping
Review Comment:
ack
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail:
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485744738
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
+
+ // stores taskAttemptId to commit data mapping
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816478#comment-17816478
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485743070
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
Review Comment:
ack
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485743070
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+
+import static
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTrackerUtils.extractTaskAttemptIdFromPath;
+
+/**
+ * InMemoryMagicCommitTracker stores the commit data in memory.
+ * The commit data and related data stores are flushed out from
+ * the memory when the task is committed or aborted.
+ */
+public class InMemoryMagicCommitTracker extends MagicCommitTracker {
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816475#comment-17816475
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java:
##
@@ -242,6 +242,13 @@ private CommitConstants() {
*/
public static final int DEFAULT_COMMITTER_THREADS = 32;
+
+ public static final String
FS_S3A_COMMITTER_MAGIC_TRACK_COMMITS_IN_MEMORY_ENABLED =
Review Comment:
ack
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[ https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816476#comment-17816476 ] ASF GitHub Bot commented on HADOOP-19047: - shameersss1 commented on code in PR #6468: URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740943 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/MagicCommitIntegration.java: ## @@ -20,17 +20,19 @@ import java.util.List; +import org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker; Review Comment: ack > Support InMemory Tracking Of S3A Magic Commits > -- > > Key: HADOOP-19047 > URL: https://issues.apache.org/jira/browse/HADOOP-19047 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The following are the operations which happens within a Task when it uses S3A > Magic Committer. > *During closing of stream* > 1. A 0-byte file with a same name of the original file is uploaded to S3 > using PUT operation. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152] > for more information. This is done so that the downstream application like > Spark could get the size of the file which is being written. > 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176] > for more information. > *During TaskCommit* > 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number > of metadata file in S3 if a single task writes to 'x' files) are read and > rewritten to S3 as a single metadata file. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201] > for more information > Since these operations happens with the Task JVM, We could optimize as well > as save cost by storing these information in memory when Task memory usage is > not a constraint. Hence the proposal here is to introduce a new MagicCommit > Tracker called "InMemoryMagicCommitTracker" which will store the > 1. Metadata of MPU in memory till the Task is committed > 2. Store the size of the file which can be used by the downstream application > to get the file size before it is committed/visible to the output path. > This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call > given a Task writes only 1 file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468: URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740943 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/MagicCommitIntegration.java: ## @@ -20,17 +20,19 @@ import java.util.List; +import org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker; Review Comment: ack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java:
##
@@ -242,6 +242,13 @@ private CommitConstants() {
*/
public static final int DEFAULT_COMMITTER_THREADS = 32;
+
+ public static final String
FS_S3A_COMMITTER_MAGIC_TRACK_COMMITS_IN_MEMORY_ENABLED =
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816473#comment-17816473
]
ASF GitHub Bot commented on HADOOP-19047:
-
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740055
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -3906,6 +3908,21 @@ public void access(final Path f, final FsAction mode)
@Retries.RetryTranslated
public FileStatus getFileStatus(final Path f) throws IOException {
Path path = qualify(f);
+if (isTrackMagicCommitsInMemoryEnabled(getConf()) &&
isMagicCommitPath(path)) {
+ // Some downstream apps might call getFileStatus for a magic path to get
the file size.
+ // when commit data is stored in memory construct the dummy
S3AFileStatus with correct
+ // file size fetched from the memory.
+ if
(InMemoryMagicCommitTracker.getTaskAttemptIdToBytesWritten().containsKey(path))
{
+long len =
InMemoryMagicCommitTracker.getTaskAttemptIdToBytesWritten().get(path);
+return new S3AFileStatus(len,
Review Comment:
ack
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740055
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -3906,6 +3908,21 @@ public void access(final Path f, final FsAction mode)
@Retries.RetryTranslated
public FileStatus getFileStatus(final Path f) throws IOException {
Path path = qualify(f);
+if (isTrackMagicCommitsInMemoryEnabled(getConf()) &&
isMagicCommitPath(path)) {
+ // Some downstream apps might call getFileStatus for a magic path to get
the file size.
+ // when commit data is stored in memory construct the dummy
S3AFileStatus with correct
+ // file size fetched from the memory.
+ if
(InMemoryMagicCommitTracker.getTaskAttemptIdToBytesWritten().containsKey(path))
{
+long len =
InMemoryMagicCommitTracker.getTaskAttemptIdToBytesWritten().get(path);
+return new S3AFileStatus(len,
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17109) add guava BaseEncoding to illegalClasses
[
https://issues.apache.org/jira/browse/HADOOP-17109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17109:
Component/s: build
common
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> add guava BaseEncoding to illegalClasses
>
>
> Key: HADOOP-17109
> URL: https://issues.apache.org/jira/browse/HADOOP-17109
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
>Affects Versions: 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> One important thing to not here as pointed out by [~jeagles] in [his comment
> on the parent
> task|https://issues.apache.org/jira/browse/HADOOP-17098?focusedCommentId=17147935=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17147935]
> {quote}One note to be careful about is that base64 translation is not a
> standard, so the two implementations could produce different results. This
> might matter in the case of serialization, persistence, or client server
> different versions.{quote}
> *Base64Url:*
> {code:java}
> Targets
> Occurrences of 'base64Url' in project with mask '*.java'
> Found Occurrences (6 usages found)
> org.apache.hadoop.mapreduce (3 usages found)
> CryptoUtils.java (3 usages found)
> wrapIfNecessary(Configuration, FSDataOutputStream, boolean) (1
> usage found)
> 138 + Base64.encodeBase64URLSafeString(iv) + "]");
> wrapIfNecessary(Configuration, InputStream, long) (1 usage found)
> 183 + Base64.encodeBase64URLSafeString(iv) + "]");
> wrapIfNecessary(Configuration, FSDataInputStream) (1 usage found)
> 218 + Base64.encodeBase64URLSafeString(iv) + "]");
> org.apache.hadoop.util (2 usages found)
> KMSUtil.java (2 usages found)
> toJSON(KeyVersion) (1 usage found)
> 104 Base64.encodeBase64URLSafeString(
> toJSON(EncryptedKeyVersion) (1 usage found)
> 117
> .encodeBase64URLSafeString(encryptedKeyVersion.getEncryptedKeyIv()));
> org.apache.hadoop.yarn.server.resourcemanager.webapp (1 usage found)
> TestRMWebServicesAppsModification.java (1 usage found)
> testAppSubmit(String, String) (1 usage found)
> 837 .put("test",
> Base64.encodeBase64URLSafeString("value12".getBytes("UTF8")));
> {code}
> *Base64:*
> {code:java}
> Targets
> Occurrences of 'base64;' in project with mask '*.java'
> Found Occurrences (51 usages found)
> org.apache.hadoop.crypto.key.kms (1 usage found)
> KMSClientProvider.java (1 usage found)
> 20 import org.apache.commons.codec.binary.Base64;
> org.apache.hadoop.crypto.key.kms.server (1 usage found)
> KMS.java (1 usage found)
> 22 import org.apache.commons.codec.binary.Base64;
> org.apache.hadoop.fs (2 usages found)
> XAttrCodec.java (2 usages found)
> 23 import org.apache.commons.codec.binary.Base64;
> 56 BASE64;
> org.apache.hadoop.fs.azure (3 usages found)
> AzureBlobStorageTestAccount.java (1 usage found)
> 23 import com.microsoft.azure.storage.core.Base64;
> BlockBlobAppendStream.java (1 usage found)
> 50 import org.apache.commons.codec.binary.Base64;
> ITestBlobDataValidation.java (1 usage found)
> 50 import com.microsoft.azure.storage.core.Base64;
> org.apache.hadoop.fs.azurebfs (2 usages found)
> AzureBlobFileSystemStore.java (1 usage found)
> 99 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> TestAbfsConfigurationFieldsValidation.java (1 usage found)
> 34 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> org.apache.hadoop.fs.azurebfs.diagnostics (2 usages found)
> Base64StringConfigurationBasicValidator.java (1 usage found)
> 26 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> TestConfigurationValidators.java (1 usage found)
> 25 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> org.apache.hadoop.fs.azurebfs.extensions (2 usages found)
> MockDelegationSASTokenProvider.java (1 usage found)
> 37 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> MockSASTokenProvider.java (1 usage found)
> 27 import org.apache.hadoop.fs.azurebfs.utils.Base64;
> org.apache.hadoop.fs.azurebfs.services (1 usage found)
> SharedKeyCredentials.java (1 usage found)
> 47 import org.apache.hadoop.fs.azurebfs.utils.Base64;
>
[jira] [Updated] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[ https://issues.apache.org/jira/browse/HADOOP-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17038: Component/s: fs/azure Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Support disabling buffered reads in ABFS positional reads > - > > Key: HADOOP-17038 > URL: https://issues.apache.org/jira/browse/HADOOP-17038 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1, 3.4.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Labels: HBase, abfsactive, pull-request-available > Fix For: 3.3.1, 3.4.0 > > Attachments: HBase Perf Test Report.xlsx, screenshot-1.png > > Time Spent: 9h 10m > Remaining Estimate: 0h > > Right now it will do a seek to the position , read and then seek back to the > old position. (As per the impl in the super class) > In HBase kind of workloads we rely mostly on short preads. (like 64 KB size > by default). So would be ideal to support a pure pos read API which will not > even keep the data in a buffer but will only read the required data as what > is asked for by the caller. (Not reading ahead more data as per the read size > config) > Allow an optional boolean config to be specified while opening file for read > using which buffered pread can be disabled. > FutureDataInputStreamBuilder openFile(Path path) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17288) Use shaded guava from thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-17288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17288: Component/s: common hadoop-thirdparty Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Use shaded guava from thirdparty > > > Key: HADOOP-17288 > URL: https://issues.apache.org/jira/browse/HADOOP-17288 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, hadoop-thirdparty >Affects Versions: 3.3.1, 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Use the shaded version of guava in hadoop-thirdparty -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16915) ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance
[ https://issues.apache.org/jira/browse/HADOOP-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16915: Component/s: fs/azure test Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > ABFS: Test failure > ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance > --- > > Key: HADOOP-16915 > URL: https://issues.apache.org/jira/browse/HADOOP-16915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.1, 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: abfsactive > Fix For: 3.3.1, 3.4.0 > > > Ref: https://issues.apache.org/jira/browse/HADOOP-16890 > The following test fails randomly. This test compares the perf between Non > HNS account against WASB. > ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17099) Replace Guava Predicate with Java8+ Predicate
[
https://issues.apache.org/jira/browse/HADOOP-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17099:
Component/s: build
Hadoop Flags: Reviewed
Target Version/s: 3.3.1, 3.2.2, 3.4.0
Affects Version/s: 3.3.1
3.2.2
3.4.0
> Replace Guava Predicate with Java8+ Predicate
> -
>
> Key: HADOOP-17099
> URL: https://issues.apache.org/jira/browse/HADOOP-17099
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.2.2, 3.3.1, 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
> Attachments: HADOOP-17099.004.patch, HADOOP-17099.005.patch,
> HADOOP-17099.006.patch, HADOOP-17099.007.patch
>
>
> {{com.google.common.base.Predicate}} can be replaced with
> {{java.util.function.Predicate}}.
> The change involving 9 occurrences is straightforward:
> {code:java}
> Targets
> Occurrences of 'com.google.common.base.Predicate' in project with mask
> '*.java'
> Found Occurrences (9 usages found)
> org.apache.hadoop.hdfs.server.blockmanagement (1 usage found)
> CombinedHostFileManager.java (1 usage found)
> 43 import com.google.common.base.Predicate;
> org.apache.hadoop.hdfs.server.namenode (1 usage found)
> NameNodeResourceChecker.java (1 usage found)
> 38 import com.google.common.base.Predicate;
> org.apache.hadoop.hdfs.server.namenode.snapshot (1 usage found)
> Snapshot.java (1 usage found)
> 41 import com.google.common.base.Predicate;
> org.apache.hadoop.metrics2.impl (2 usages found)
> MetricsRecords.java (1 usage found)
> 21 import com.google.common.base.Predicate;
> TestMetricsSystemImpl.java (1 usage found)
> 41 import com.google.common.base.Predicate;
> org.apache.hadoop.yarn.logaggregation (1 usage found)
> AggregatedLogFormat.java (1 usage found)
> 77 import com.google.common.base.Predicate;
> org.apache.hadoop.yarn.logaggregation.filecontroller (1 usage found)
> LogAggregationFileController.java (1 usage found)
> 22 import com.google.common.base.Predicate;
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile (1 usage
> found)
> LogAggregationIndexedFileController.java (1 usage found)
> 22 import com.google.common.base.Predicate;
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation
> (1 usage found)
> AppLogAggregatorImpl.java (1 usage found)
> 75 import com.google.common.base.Predicate;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17100) Replace Guava Supplier with Java8+ Supplier in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17100:
Component/s: build
Target Version/s: 3.3.1, 3.2.2, 3.4.0
Affects Version/s: 3.3.1
3.2.2
3.4.0
> Replace Guava Supplier with Java8+ Supplier in Hadoop
> -
>
> Key: HADOOP-17100
> URL: https://issues.apache.org/jira/browse/HADOOP-17100
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.2.2, 3.3.1, 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
> Attachments: HADOOP-17100.001.patch, HADOOP-17100.002.patch,
> HADOOP-17100.003.patch, HADOOP-17100.006.patch,
> HADOOP-17100.branch-3.1.006.patch, HADOOP-17100.branch-3.2.006.patch,
> HADOOP-17100.branch-3.3.006.patch
>
>
> Replacing Usage of {{guava.Supplier<>}} are in Unit tests
> {{GenereicTestUtils.waitFor()}} in Hadoop project.
> * To make things more convenient for reviewers, I decided:
> ** Not to replace Object instantiation by lambda expressions because this
> will increase the patch size significantly and require code adjustments to
> pass the checkstyle scripts.
> ** Not to refactor the imports because this will make reading the patch more
> difficult.
> * Merge should be done to the following branches: trunk, branch-3.3,
> branch-3.2, branch-3.1
> The task is straightforward because {{java.util.Supplier}} has the same API
> as {{guava.Supplier<>}} and the vast majority of usage comes from Test-units.
> Therefore, we need only to do the following a "one-line" change in all 147
> files.
> {code:bash}
>
> -import com.google.common.base.Supplier;
> +import java.util.function.Supplier;
> {code}
> The code change needs to be applied to the following list of files:
> {code:java}
>
> Targets
> Occurrences of 'com.google.common.base.Supplier' in project with mask
> '*.java'
> Found Occurrences (146 usages found)
> org.apache.hadoop.conf (1 usage found)
> TestReconfiguration.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.crypto.key.kms.server (1 usage found)
> TestKMS.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.fs (2 usages found)
> FCStatisticsBaseTest.java (1 usage found)
> 40 import com.google.common.base.Supplier;
> TestEnhancedByteBufferAccess.java (1 usage found)
> 75 import com.google.common.base.Supplier;
> org.apache.hadoop.fs.viewfs (1 usage found)
> TestViewFileSystemWithTruncate.java (1 usage found)
> 23 import com.google.common.base.Supplier;
> org.apache.hadoop.ha (1 usage found)
> TestZKFailoverController.java (1 usage found)
> 25 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs (20 usages found)
> DFSTestUtil.java (1 usage found)
> 79 import com.google.common.base.Supplier;
> MiniDFSCluster.java (1 usage found)
> 78 import com.google.common.base.Supplier;
> TestBalancerBandwidth.java (1 usage found)
> 29 import com.google.common.base.Supplier;
> TestClientProtocolForPipelineRecovery.java (1 usage found)
> 30 import com.google.common.base.Supplier;
> TestDatanodeRegistration.java (1 usage found)
> 44 import com.google.common.base.Supplier;
> TestDataTransferKeepalive.java (1 usage found)
> 47 import com.google.common.base.Supplier;
> TestDeadNodeDetection.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestDecommission.java (1 usage found)
> 41 import com.google.common.base.Supplier;
> TestDFSShell.java (1 usage found)
> 37 import com.google.common.base.Supplier;
> TestEncryptedTransfer.java (1 usage found)
> 35 import com.google.common.base.Supplier;
> TestEncryptionZonesWithKMS.java (1 usage found)
> 22 import com.google.common.base.Supplier;
> TestFileCorruption.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestLeaseRecovery2.java (1 usage found)
> 32 import com.google.common.base.Supplier;
> TestLeaseRecoveryStriped.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestMaintenanceState.java (1 usage found)
> 63 import com.google.common.base.Supplier;
> TestPread.java (1 usage found)
> 61 import com.google.common.base.Supplier;
>
[jira] [Updated] (HADOOP-17101) Replace Guava Function with Java8+ Function
[
https://issues.apache.org/jira/browse/HADOOP-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17101:
Component/s: build
Hadoop Flags: Reviewed
Target Version/s: 3.3.1, 3.2.2, 3.4.0
Affects Version/s: 3.3.1
3.2.2
3.4.0
> Replace Guava Function with Java8+ Function
> ---
>
> Key: HADOOP-17101
> URL: https://issues.apache.org/jira/browse/HADOOP-17101
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.2.2, 3.3.1, 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
> Attachments: HADOOP-17101.005.patch, HADOOP-17101.006.patch,
> HADOOP-17101.008.patch
>
>
> {code:java}
> Targets
> Occurrences of 'com.google.common.base.Function'
> Found Occurrences (7 usages found)
> hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff (1 usage found)
> Apache_Hadoop_HDFS_2.6.0.xml (1 usage found)
> 13603 type="com.google.common.base.Function"
> org.apache.hadoop.hdfs.server.blockmanagement (1 usage found)
> HostSet.java (1 usage found)
> 20 import com.google.common.base.Function;
> org.apache.hadoop.hdfs.server.datanode.checker (1 usage found)
> AbstractFuture.java (1 usage found)
> 58 * (ListenableFuture, com.google.common.base.Function)
> Futures.transform}
> org.apache.hadoop.hdfs.server.namenode.ha (1 usage found)
> HATestUtil.java (1 usage found)
> 40 import com.google.common.base.Function;
> org.apache.hadoop.hdfs.server.protocol (1 usage found)
> RemoteEditLog.java (1 usage found)
> 20 import com.google.common.base.Function;
> org.apache.hadoop.mapreduce.lib.input (1 usage found)
> TestFileInputFormat.java (1 usage found)
> 58 import com.google.common.base.Function;
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb (1 usage found)
> GetApplicationsRequestPBImpl.java (1 usage found)
> 38 import com.google.common.base.Function;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17111) Replace Guava Optional with Java8+ Optional
[
https://issues.apache.org/jira/browse/HADOOP-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17111:
Component/s: build
common
Target Version/s: 3.3.1, 3.4.0
Affects Version/s: 3.3.1
3.4.0
Description:
{code:java}
Targets
Occurrences of 'com.google.common.base.Optional' in project with mask
'*.java'
Found Occurrences (3 usages found)
org.apache.hadoop.yarn.server.nodemanager (2 usages found)
DefaultContainerExecutor.java (1 usage found)
71 import com.google.common.base.Optional;
LinuxContainerExecutor.java (1 usage found)
22 import com.google.common.base.Optional;
org.apache.hadoop.yarn.server.resourcemanager.recovery (1 usage found)
TestZKRMStateStorePerf.java (1 usage found)
21 import com.google.common.base.Optional;
{code}
was:
{code:java}
Targets
Occurrences of 'com.google.common.base.Optional' in project with mask
'*.java'
Found Occurrences (3 usages found)
org.apache.hadoop.yarn.server.nodemanager (2 usages found)
DefaultContainerExecutor.java (1 usage found)
71 import com.google.common.base.Optional;
LinuxContainerExecutor.java (1 usage found)
22 import com.google.common.base.Optional;
org.apache.hadoop.yarn.server.resourcemanager.recovery (1 usage found)
TestZKRMStateStorePerf.java (1 usage found)
21 import com.google.common.base.Optional;
{code}
> Replace Guava Optional with Java8+ Optional
> ---
>
> Key: HADOOP-17111
> URL: https://issues.apache.org/jira/browse/HADOOP-17111
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HADOOP-17111.001.patch, HADOOP-17111.002.patch
>
>
> {code:java}
> Targets
> Occurrences of 'com.google.common.base.Optional' in project with mask
> '*.java'
> Found Occurrences (3 usages found)
> org.apache.hadoop.yarn.server.nodemanager (2 usages found)
> DefaultContainerExecutor.java (1 usage found)
> 71 import com.google.common.base.Optional;
> LinuxContainerExecutor.java (1 usage found)
> 22 import com.google.common.base.Optional;
> org.apache.hadoop.yarn.server.resourcemanager.recovery (1 usage found)
> TestZKRMStateStorePerf.java (1 usage found)
> 21 import com.google.common.base.Optional;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17868) Add more test for the BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17868: Component/s: test Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Add more test for the BuiltInGzipCompressor > --- > > Key: HADOOP-17868 > URL: https://issues.apache.org/jira/browse/HADOOP-17868 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 3.4.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > We added BuiltInGzipCompressor recently. It is better to add more > compatibility tests for the compressor. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17936) TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after reverting HADOOP-16878
[
https://issues.apache.org/jira/browse/HADOOP-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17936:
Component/s: test
Target Version/s: 3.3.2, 3.4.0
Affects Version/s: 3.3.2
3.4.0
> TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after
> reverting HADOOP-16878
>
>
> Key: HADOOP-17936
> URL: https://issues.apache.org/jira/browse/HADOOP-17936
> Project: Hadoop Common
> Issue Type: Test
> Components: test
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> After reverting HADOOP-16878 from branch-3.3, test
> {{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started
> to fail because it expects an exception but the copying succeeded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17717) Update wildfly openssl to 1.1.3.Final
[ https://issues.apache.org/jira/browse/HADOOP-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17717: Component/s: build common Hadoop Flags: Reviewed Affects Version/s: 3.3.5 3.4.0 > Update wildfly openssl to 1.1.3.Final > - > > Key: HADOOP-17717 > URL: https://issues.apache.org/jira/browse/HADOOP-17717 > Project: Hadoop Common > Issue Type: Bug > Components: build, common >Affects Versions: 3.4.0, 3.3.5 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > HADOOP-17649 got stalled. IMO we can bump the version to 1.1.3.Final instead, > at least, for branch-3.3. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17518) Usage of incorrect regex range A-z
[
https://issues.apache.org/jira/browse/HADOOP-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17518:
Component/s: httpfs
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Usage of incorrect regex range A-z
> --
>
> Key: HADOOP-17518
> URL: https://issues.apache.org/jira/browse/HADOOP-17518
> Project: Hadoop Common
> Issue Type: Bug
> Components: httpfs
>Affects Versions: 3.4.0
>Reporter: Marcono1234
>Assignee: Nishtha Shah
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> There are two cases where the regex {{A-z}} is used. I assume that is a typo
> (and should be {{A-Z}}) because {{A-z}} matches:
> - {{A-Z}}
> - {{\[}}, {{}}, {{\]}}, {{^}}, {{_}}, {{`}}
> - {{a-z}}
> Affected:
> -
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L109
> (and
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L115)
> -
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/resourcetypes/ResourceTypesTestHelper.java#L38
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18100) Change scope of inner classes in InodeTree to make them accessible outside package
[ https://issues.apache.org/jira/browse/HADOOP-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18100: Component/s: viewfs Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.2.4, 3.4.0 Affects Version/s: 3.3.5 3.2.4 3.4.0 > Change scope of inner classes in InodeTree to make them accessible outside > package > -- > > Key: HADOOP-18100 > URL: https://issues.apache.org/jira/browse/HADOOP-18100 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.4.0, 3.2.4, 3.3.5 >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.5 > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18129) Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-18129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18129: Component/s: viewfs Hadoop Flags: Reviewed Target Version/s: 3.3.5, 2.10.2, 3.4.0 Affects Version/s: 3.3.5 2.10.2 3.4.0 > Change URI[] in INodeLink to String[] to reduce memory footprint of > ViewFileSystem > -- > > Key: HADOOP-18129 > URL: https://issues.apache.org/jira/browse/HADOOP-18129 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.4.0, 2.10.2, 3.3.5 >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.3.5 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Around 40k instances of INodeLink each of which is taking memory ranging from > 1680bytes to 1160bytes. Multiplying 40k with 1160bytes will give us > approximate 45mb. > With changing from URI to String in INodeLink the memory consumed by each of > the INodeLink objects has reduced from ~1160 bytes to ~320 bytes. Overall > size becomes (40k X 320) 12mb -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18096) Distcp: Sync moves filtered file to home directory rather than deleting
[ https://issues.apache.org/jira/browse/HADOOP-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18096: Component/s: tools/distcp Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > Distcp: Sync moves filtered file to home directory rather than deleting > --- > > Key: HADOOP-18096 > URL: https://issues.apache.org/jira/browse/HADOOP-18096 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.4.0, 3.3.2 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Distcp sync with snapshot, if the file being copied is renamed to a path > which is in the exclusion filter, tries to delete the file. > But instead of deleting, it moves the file to home directory > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18101) Bump aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1
[
https://issues.apache.org/jira/browse/HADOOP-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18101:
Component/s: build
common
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Bump aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1
> --
>
> Key: HADOOP-18101
> URL: https://issues.apache.org/jira/browse/HADOOP-18101
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, common
>Affects Versions: 3.4.0
>Reporter: Aswin Shakil
>Assignee: Aswin Shakil
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The current aliyun-sdk-oss 3.13.0 is affected by
> [CVE-2021-33813|https://github.com/advisories/GHSA-2363-cqg2-863c] due to
> jdom 2.0.6. maven-shade-plugin is also affected by the CVE.
> Bumping aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1 will resolve this issue
> {code:java}
> [INFO] +- org.apache.maven.plugins:maven-shade-plugin:jar:3.2.1:provided
> [INFO] | +-
> org.apache.maven.shared:maven-artifact-transfer:jar:0.10.0:provided
> [INFO] | +- org.jdom:jdom2:jar:2.0.6:provided
> ..
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.1:compile
> [INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18063) Remove unused import AbstractJavaKeyStoreProvider in Shell class
[ https://issues.apache.org/jira/browse/HADOOP-18063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18063: Component/s: command Hadoop Flags: Reviewed Target Version/s: 3.2.4, 3.3.2, 3.4.0 > Remove unused import AbstractJavaKeyStoreProvider in Shell class > > > Key: HADOOP-18063 > URL: https://issues.apache.org/jira/browse/HADOOP-18063 > Project: Hadoop Common > Issue Type: Bug > Components: command >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Attachments: image-2022-01-01-22-40-50-604.png > > Time Spent: 1h > Remaining Estimate: 0h > > In Shell, there are some invalid imports. > For example: > !image-2022-01-01-22-40-50-604.png! > Among them, AbstractJavaKeyStoreProvider does not seem to be referenced > anywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18057) Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets
[
https://issues.apache.org/jira/browse/HADOOP-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18057:
Component/s: fs/s3
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets
> ---
>
> Key: HADOOP-18057
> URL: https://issues.apache.org/jira/browse/HADOOP-18057
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Fixing typo validateEncrytionSecrets to validateEncryptionSecrets in
> {code:java}
> hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractTestS3AEncryption.java{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17995) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HADOOP-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17995: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HADOOP-17995 > URL: https://issues.apache.org/jira/browse/HADOOP-17995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > As [HADOOP-16947|https://issues.apache.org/jira/browse/HADOOP-16947] problem > with description, > Stale SumAndCount also should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson. > Ensure the DataNode JMX get SendPacketDownstreamAvgInfo Metrics is accurate -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17999: Component/s: viewfs Target Version/s: 3.2.4, 3.3.2, 2.10.2, 3.4.0 Affects Version/s: 3.2.4 3.3.2 2.10.2 3.4.0 > No-op implementation of setWriteChecksum and setVerifyChecksum in > ViewFileSystem > > > Key: HADOOP-17999 > URL: https://issues.apache.org/jira/browse/HADOOP-17999 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.4.0, 2.10.2, 3.3.2, 3.2.4 >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently setVerifyChecksum and setWriteChecksum initializes all target file > systems which causes delay in hadoop shell copy commands such as get, put, > copyFromLocal etc. > This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17931) Fix typos in usage message in winutils.exe
[ https://issues.apache.org/jira/browse/HADOOP-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17931: Component/s: winutils Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix typos in usage message in winutils.exe > -- > > Key: HADOOP-17931 > URL: https://issues.apache.org/jira/browse/HADOOP-17931 > Project: Hadoop Common > Issue Type: Bug > Components: winutils >Affects Versions: 3.4.0 >Reporter: Íñigo Goiri >Assignee: Gautham Banasandra >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The usage message for task creation in winutils.exe has a few typos: > * OPTOINS > * cup rate -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17940) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/HADOOP-17940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17940: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Upgrade Kafka to 2.8.1 > -- > > Key: HADOOP-17940 > URL: https://issues.apache.org/jira/browse/HADOOP-17940 > Project: Hadoop Common > Issue Type: Bug > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Kafka 2.4.0 has the following vulnerability. > [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38153] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17891:
Component/s: build
Target Version/s: 3.3.2, 3.4.0
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HADOOP-17891-Addendum-01.patch
>
> Time Spent: 17h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17904) Test Result Not Working In Jenkins Result
[ https://issues.apache.org/jira/browse/HADOOP-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17904: Component/s: build test Target Version/s: 3.4.0 > Test Result Not Working In Jenkins Result > - > > Key: HADOOP-17904 > URL: https://issues.apache.org/jira/browse/HADOOP-17904 > Project: Hadoop Common > Issue Type: Bug > Components: build, test >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The jenkins used to show the Test Results, now the list is broken and No PR > has that working. > eg: > [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/] > > For Daily Build No Test Result as well: > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17877) BuiltInGzipCompressor header and trailer should not be static variables
[ https://issues.apache.org/jira/browse/HADOOP-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17877: Component/s: compress io Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > BuiltInGzipCompressor header and trailer should not be static variables > --- > > Key: HADOOP-17877 > URL: https://issues.apache.org/jira/browse/HADOOP-17877 > Project: Hadoop Common > Issue Type: Bug > Components: compress, io >Affects Versions: 3.4.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > In the newly added BuiltInGzipCompressor, we should not let header and > trailer as static variables as they are for different instances. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17886) Upgrade ant to 1.10.11
[ https://issues.apache.org/jira/browse/HADOOP-17886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17886: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.2.3, 2.10.2, 3.4.0 > Upgrade ant to 1.10.11 > -- > > Key: HADOOP-17886 > URL: https://issues.apache.org/jira/browse/HADOOP-17886 > Project: Hadoop Common > Issue Type: Bug > Components: build, common >Affects Versions: 3.3.0, 3.2.2, 3.4.0, 2.10.2 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Vulnerabilities reported in org.apache.ant:ant:1.10.9 > * [CVE-2021-36374|https://nvd.nist.gov/vuln/detail/CVE-2021-36374] moderate > severity > * [CVE-2021-36373|https://nvd.nist.gov/vuln/detail/CVE-2021-36373] moderate > severity > suggested: org.apache.ant:ant ~> 1.10.11 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17844) Upgrade JSON smart to 2.4.7
[ https://issues.apache.org/jira/browse/HADOOP-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17844: Component/s: build common Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Upgrade JSON smart to 2.4.7 > --- > > Key: HADOOP-17844 > URL: https://issues.apache.org/jira/browse/HADOOP-17844 > Project: Hadoop Common > Issue Type: Bug > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently we are using JSON Smart 2.4.2 version which is vulnerable to - > CVE-2021-31684. > We can upgrade the version to 2.4.7 (2.4.5 or later). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17796) Upgrade jetty version to 9.4.43
[ https://issues.apache.org/jira/browse/HADOOP-17796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17796: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.2.4, 3.3.2, 3.4.0 > Upgrade jetty version to 9.4.43 > --- > > Key: HADOOP-17796 > URL: https://issues.apache.org/jira/browse/HADOOP-17796 > Project: Hadoop Common > Issue Type: Bug > Components: build, common >Affects Versions: 3.2.2, 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Renukaprasad C >Priority: Major > Labels: dependency, pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > > https://github.com/eclipse/jetty.project/security/advisories/GHSA-m6cp-vxjx-65j6 > https://github.com/eclipse/jetty.project/security/advisories/GHSA-gwcr-j4wh-j3cq -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17793) Better token validation
[ https://issues.apache.org/jira/browse/HADOOP-17793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17793: Component/s: security Target Version/s: 3.3.2, 3.2.3, 2.10.2, 3.4.0 Affects Version/s: 3.3.2 3.2.3 2.10.2 3.4.0 > Better token validation > --- > > Key: HADOOP-17793 > URL: https://issues.apache.org/jira/browse/HADOOP-17793 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.4.0, 2.10.2, 3.2.3, 3.3.2 >Reporter: Artem Smotrakov >Assignee: Artem Smotrakov >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Attachments: token.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > `MessageDigest.isEqual()` should be used for checking tokens. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17769) Upgrade JUnit to 4.13.2
[
https://issues.apache.org/jira/browse/HADOOP-17769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17769:
Component/s: build
test
Target Version/s: 3.3.2, 3.2.3, 2.10.2, 3.4.0
> Upgrade JUnit to 4.13.2
> ---
>
> Key: HADOOP-17769
> URL: https://issues.apache.org/jira/browse/HADOOP-17769
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.3.1, 3.4.0, 2.10.2, 3.2.3
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> JUnit 4.13.1 has a bug that is reported in Junit
> [issue-1652|https://github.com/junit-team/junit4/issues/1652] _Timeout
> ThreadGroups should not be destroyed_
> After upgrading Junit to 4.13.1 in HADOOP-17602, {{TestBlockRecovery}}
> started to fail regularly in branch-3.x and branch-2.10.
> While investigating the failure in branch-2.10 HDFS-16072, I found out that
> the bug is the main reason {{TestBlockRecovery}} started to fail because the
> timeout of the Junit would try to close a ThreadGroup that has been already
> closed which throws the {{java.lang.IllegalThreadStateException}}.
> The bug has been fixed in Junit-4.13.2
> For branch-3.x, HDFS-15940 did not address the root cause of the problem.
> Eventually, Splitting the {{TestBlockRecovery}} hid the bug, but the upgrade
> needs to be done so that the problem does not show up in another unit test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17645) Fix test failures in org.apache.hadoop.fs.azure.ITestOutputStreamSemantics
[ https://issues.apache.org/jira/browse/HADOOP-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17645: Component/s: fs/azure Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 > Fix test failures in org.apache.hadoop.fs.azure.ITestOutputStreamSemantics > -- > > Key: HADOOP-17645 > URL: https://issues.apache.org/jira/browse/HADOOP-17645 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.1, 3.4.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Failures after HADOOP-13327 > PageBlob and Compacting BlockBlob having only hflush and hsync capability. > The test wrongly assert capability DROPBEHIND, READAHEAD, UNBUFFER -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17750) Fix asf license errors in newly added files by HADOOP-17727
[ https://issues.apache.org/jira/browse/HADOOP-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17750: Component/s: build Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix asf license errors in newly added files by HADOOP-17727 > --- > > Key: HADOOP-17750 > URL: https://issues.apache.org/jira/browse/HADOOP-17750 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17703) checkcompatibility.py errors out when specifying annotations
[
https://issues.apache.org/jira/browse/HADOOP-17703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17703:
Component/s: scripts
Hadoop Flags: Reviewed
Target Version/s: 3.2.3, 3.3.1, 3.4.0
> checkcompatibility.py errors out when specifying annotations
>
>
> Key: HADOOP-17703
> URL: https://issues.apache.org/jira/browse/HADOOP-17703
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> [https://github.com/apache/hadoop/blob/trunk/dev-support/bin/checkcompatibility.py#L178]
> {code:java}
> with file(annotations_path, "w") as f: {code}
> is not a valid Pythonic code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17657) SequenceFile.Writer should implement StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17657: Component/s: io Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > SequenceFile.Writer should implement StreamCapabilities > --- > > Key: HADOOP-17657 > URL: https://issues.apache.org/jira/browse/HADOOP-17657 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 3.3.1, 3.4.0 >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on > S3 from Tez, which internally calls > org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS > DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not > implemented and throws java.lang.UnsupportedOperationException. > bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" > level="ERROR" thread="HistoryEventHandlingThread"] Thread > Thread[HistoryEventHandlingThread, 5,main] threw an > Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not > Syncable^M at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M > at org.apache.hadoop.fs.FS > DataOutputStream.hflush(FSDataOutputStream.java:136)^M at > org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at > org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr > iter.java:64)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M > at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han > dleEvent(ProtoHistoryLoggingService.java:198)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M > at java.lang.Thread.run(Thread.java:748)^M > In order to fix this issue we should implement StreamCapabilities in > SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not > supported. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17505) public interface GroupMappingServiceProvider needs default impl for getGroupsSet()
[ https://issues.apache.org/jira/browse/HADOOP-17505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17505: Component/s: security Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > public interface GroupMappingServiceProvider needs default impl for > getGroupsSet() > --- > > Key: HADOOP-17505 > URL: https://issues.apache.org/jira/browse/HADOOP-17505 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.4.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > HADOOP-17079 added "GroupMappingServiceProvider#getGroupsSet()" interface. > But since this is a public interface, it will break compilation of existing > implementations in downstreams. > Consider adding a default implementation in the interface to avoid such > failures. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17617) Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file
[ https://issues.apache.org/jira/browse/HADOOP-17617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17617: Component/s: documentation Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm > file > > > Key: HADOOP-17617 > URL: https://issues.apache.org/jira/browse/HADOOP-17617 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1, 3.4.0 >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HADOOP-17617.001.patch, HADOOP-17617.002.patch, > HADOOP-17617.003.patch, KMS_Key_Version_Response_01.png, > KMS_Key_Version_Response_02.png > > > Format of RESPONSE of Get Key Versions in KMS index.md.vm is incorrect > [https://hadoop.apache.org/docs/r3.1.1/hadoop-kms/index.html#Get_Key_Versions] > Attached the outputs of the commands for reference -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17588) CryptoInputStream#close() should be synchronized
[ https://issues.apache.org/jira/browse/HADOOP-17588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17588: Component/s: crypto Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.4.0 Affects Version/s: 3.3.5 3.4.0 > CryptoInputStream#close() should be synchronized > > > Key: HADOOP-17588 > URL: https://issues.apache.org/jira/browse/HADOOP-17588 > Project: Hadoop Common > Issue Type: Bug > Components: crypto >Affects Versions: 3.4.0, 3.3.5 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 3.4.0, 3.3.5 > > Attachments: HADOOP-17588.001.patch, image-2021-03-13-23-56-18-865.png > > > org.apache.hadoop.crypto.CryptoInputStream.close() - when 2 threads try to > close the stream second thread, fails with error. > This operation should be synchronized to avoid multiple threads to perform > the close operation concurrently. > !image-2021-03-13-23-56-18-865.png|thumbnail! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17619) Fix DelegationTokenRenewer#updateRenewalTime java doc error.
[ https://issues.apache.org/jira/browse/HADOOP-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17619: Component/s: documentation Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix DelegationTokenRenewer#updateRenewalTime java doc error. > > > Key: HADOOP-17619 > URL: https://issues.apache.org/jira/browse/HADOOP-17619 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The param of updateRenewalTime should be the renew cycle, not the new time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
