[GitHub] [hadoop] davidarcher commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.

2020-02-04 Thread GitBox
davidarcher commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation. URL: https://github.com/apache/hadoop/pull/1823#issuecomment-581898688 Hi @mukund-thakur and thank you for offering up this PR. I've run into this bug and while removing t

[GitHub] [hadoop] davidarcher edited a comment on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.

2020-02-04 Thread GitBox
davidarcher edited a comment on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation. URL: https://github.com/apache/hadoop/pull/1823#issuecomment-581898688 Hi @mukund-thakur and thank you for offering up this PR. I've run into this bug and while rem

[jira] [Created] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Nitish (Jira)
Nitish created HADOOP-16838: --- Summary: Support for `fs.s3a.endpoint.region` Key: HADOOP-16838 URL: https://issues.apache.org/jira/browse/HADOOP-16838 Project: Hadoop Common Issue Type: New Feature

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374651071 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374652628 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PositionedReadable.java

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374656257 ## File path: hadoop-common-project/benchmark/pom.xml ## @@ -0,0 +1,79 @@ + + +http://maven.apache.

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374650250 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374654994 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestAsyncReaderUti

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374652156 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PositionedReadable.java

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374653875 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/AsyncReaderUtils.j

[GitHub] [hadoop] steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
steveloughran commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374655352 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestAsyncReaderUti

[jira] [Commented] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029834#comment-17029834 ] Steve Loughran commented on HADOOP-16838: - it's called 'fs.s3.endpoint > Suppo

[jira] [Commented] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029835#comment-17029835 ] Steve Loughran commented on HADOOP-16838: - It sounds like you're using spark *

[jira] [Resolved] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16838. - Resolution: Invalid > Support for `fs.s3a.endpoint.region` > --

[jira] [Commented] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029837#comment-17029837 ] Steve Loughran commented on HADOOP-16838: - Actually, it does look like its in 2

[jira] [Reopened] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-16838: - > Support for `fs.s3a.endpoint.region` > > >

[jira] [Resolved] (HADOOP-16838) Support for `fs.s3a.endpoint.region`

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16838. - Resolution: Works for Me > Support for `fs.s3a.endpoint.region` > -

[jira] [Commented] (HADOOP-15643) Review/implement ABFS support for the extra fs ops which some apps (HBase) expects

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029839#comment-17029839 ] Steve Loughran commented on HADOOP-15643: - nice. Impala will use ByteBufferedPo

[GitHub] [hadoop] steveloughran commented on issue #1812: HADOOP-13223: remove need for winutils.exe

2020-02-04 Thread GitBox
steveloughran commented on issue #1812: HADOOP-13223: remove need for winutils.exe URL: https://github.com/apache/hadoop/pull/1812#issuecomment-581967126 Reminder: We do want a patch for this, but your code has to be on hadoop trunk, and the PR created against it. Thanks,. ---

[jira] [Commented] (HADOOP-16837) Spark-SQL test running on Windows: hadoop chgrp warnings

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029941#comment-17029941 ] Steve Loughran commented on HADOOP-16837: - Well, looks like the code you are p

[jira] [Updated] (HADOOP-16837) Spark-SQL test running on Windows: hadoop chgrp warnings

2020-02-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16837: Labels: (was: filesystem unit-test win10) > Spark-SQL test running on Windows: hadoop c

[jira] [Created] (HADOOP-16839) SparkLauncher does not read SPARK_CONF_DIR/spark-defaults.conf

2020-02-04 Thread Koji Noguchi (Jira)
Koji Noguchi created HADOOP-16839: - Summary: SparkLauncher does not read SPARK_CONF_DIR/spark-defaults.conf Key: HADOOP-16839 URL: https://issues.apache.org/jira/browse/HADOOP-16839 Project: Hadoop C

[jira] [Commented] (HADOOP-16711) Add way to skip "verifyBuckets" check in S3A fs init()

2020-02-04 Thread Mukund Thakur (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030010#comment-17030010 ] Mukund Thakur commented on HADOOP-16711: Just to make sure my understanding is

[GitHub] [hadoop] avijayanhwx commented on issue #1819: HDFS-14989. Add a 'swapBlockList' operation to Namenode.

2020-02-04 Thread GitBox
avijayanhwx commented on issue #1819: HDFS-14989. Add a 'swapBlockList' operation to Namenode. URL: https://github.com/apache/hadoop/pull/1819#issuecomment-582071905 Thank you for the review @ayushtkn. This is an automated me

[GitHub] [hadoop] avijayanhwx merged pull request #1819: HDFS-14989. Add a 'swapBlockList' operation to Namenode.

2020-02-04 Thread GitBox
avijayanhwx merged pull request #1819: HDFS-14989. Add a 'swapBlockList' operation to Namenode. URL: https://github.com/apache/hadoop/pull/1819 This is an automated message from the Apache Git Service. To respond to the mess

[jira] [Commented] (HADOOP-11610) hadoop-common build fails on JDK9

2020-02-04 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030140#comment-17030140 ] Kihwal Lee commented on HADOOP-11610: - JDK11 build is being fixed in HADOOP-16795

[jira] [Resolved] (HADOOP-11610) hadoop-common build fails on JDK9

2020-02-04 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HADOOP-11610. - Resolution: Won't Fix > hadoop-common build fails on JDK9 > - >

[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.

2020-02-04 Thread GitBox
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client. URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582124713 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|-

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374938059 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374939040 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PositionedReadable.java

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374939150 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PositionedReadable.java

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374940578 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374941406 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestAsyncReaderUtils.jav

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374942678 ## File path: hadoop-common-project/benchmark/pom.xml ## @@ -0,0 +1,79 @@ + + +http://maven.apache.org/PO

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374942944 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestAsyncReaderUtils.jav

[GitHub] [hadoop] omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on a change in pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#discussion_r374944370 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/AsyncReaderUtils.java #

[GitHub] [hadoop] omalley commented on issue #1830: HADOOP-11867: Add gather API to file system.

2020-02-04 Thread GitBox
omalley commented on issue #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830#issuecomment-582139394 The benchmark numbers are posted on the jira. You'll need to help with the spec that you've developed in fsdatainputstream.md. Fundame

[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time

2020-02-04 Thread Arpit Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030201#comment-17030201 ] Arpit Agarwal commented on HADOOP-16833: Thanks for this improvement and the of

[jira] [Created] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
wujinhu created HADOOP-16840: Summary: AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket Key: HADOOP-16840 URL: https://issues.apache.org/jira/browse/HADOOP-16840 Project: Hadoop

[jira] [Updated] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-16840: - Description: When hadoop lists object in versioning bucket with many delete marker in it, OSS will ret

[jira] [Updated] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-16840: - Description: When hadoop lists object in versioning bucket with many delete marker in it, OSS will ret

[jira] [Updated] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-16840: - Description: When hadoop lists object in versioning bucket with many delete marker in it, OSS will ret

[jira] [Created] (HADOOP-16841) The description of hadoop.http.authentication.signature.secret.file contains outdated information

2020-02-04 Thread Akira Ajisaka (Jira)
Akira Ajisaka created HADOOP-16841: -- Summary: The description of hadoop.http.authentication.signature.secret.file contains outdated information Key: HADOOP-16841 URL: https://issues.apache.org/jira/browse/HADOOP-

[jira] [Work started] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16840 started by wujinhu. > AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket > ---

[jira] [Updated] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-16840: - Attachment: HADOOP-16840.001.patch Status: Patch Available (was: In Progress) > AliyunOSS: get

[jira] [Commented] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread Hadoop QA (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030393#comment-17030393 ] Hadoop QA commented on HADOOP-16840: | (/) *{color:green}+1 overall{color}* | \\ \\

[jira] [Commented] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030398#comment-17030398 ] wujinhu commented on HADOOP-16840: -- Upload HADOOP-16840.002.patch(fix code style) > A

[jira] [Updated] (HADOOP-16840) AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

2020-02-04 Thread wujinhu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-16840: - Attachment: HADOOP-16840.002.patch > AliyunOSS: getFileStatus throws FileNotFoundException in versionin