[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-576998857 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--

[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-576941328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--

[GitHub] [hadoop] steveloughran commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table

2020-01-23 Thread GitBox
steveloughran commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-577677964 thanks; +1 from me Looking at the aws docs, the AWS key is free and unthrottled, private ones have a hit, but as i

[GitHub] [hadoop] bgaborg merged pull request #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table

2020-01-23 Thread GitBox
bgaborg merged pull request #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752 This is an automated message from the Apache Git Service. To respond to the message

[jira] [Commented] (HADOOP-16732) S3Guard to support encrypted DynamoDB table

2020-01-23 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022085#comment-17022085 ] Gabor Bota commented on HADOOP-16732: - +1 on PR #1752. Committing. > S3Guard to su

[GitHub] [hadoop] steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode

2020-01-23 Thread GitBox
steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#issuecomment-577686585 thanks, I'll fix the nits and split the two modes out. As noted, I was starting to think that way --

[jira] [Commented] (HADOOP-16732) S3Guard to support encrypted DynamoDB table

2020-01-23 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022113#comment-17022113 ] Hudson commented on HADOOP-16732: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Co

[GitHub] [hadoop] liuml07 commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table

2020-01-23 Thread GitBox
liuml07 commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-577765966 Thank you both @steveloughran andd @bgaborg for reviewing and committing. -

[GitHub] [hadoop] steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode

2020-01-23 Thread GitBox
steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#issuecomment-577828928 Move from to intermingled dirListingUnion algorithms in the same method, the auth and nonauth merge/update operations have been split int

[GitHub] [hadoop] steveloughran commented on a change in pull request #1810: HADOOP-16746 mkdirs and s3guard auth mode

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#discussion_r370300215 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java ##

[GitHub] [hadoop] steveloughran edited a comment on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode

2020-01-23 Thread GitBox
steveloughran edited a comment on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#issuecomment-577828928 Move from to intermingled dirListingUnion algorithms in the same method, the auth and nonauth merge/update operations have been sp

[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#issuecomment-577028589 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment |

[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#issuecomment-576989997 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |

[GitHub] [hadoop] steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#discussion_r370307601 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3gua

[GitHub] [hadoop] steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#discussion_r370305003 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3gua

[GitHub] [hadoop] steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#discussion_r370307065 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3gua

[GitHub] [hadoop] steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#discussion_r370306157 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3gua

[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576972274 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--

[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576965145 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--

[jira] [Assigned] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread Da Zhou (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou reassigned HADOOP-16826: Assignee: Karthick Narendran > ABFS: update abfs.md to include config keys for identity transfor

[GitHub] [hadoop] steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#discussion_r370319123 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md ## @

[GitHub] [hadoop] steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#discussion_r370317165 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java ###

[GitHub] [hadoop] steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#discussion_r370315573 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java ###

[GitHub] [hadoop] steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#discussion_r370319123 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md ## @

[GitHub] [hadoop] hadoop-yetus commented on issue #1635: HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency

2020-01-23 Thread GitBox
hadoop-yetus commented on issue #1635: HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency URL: https://github.com/apache/hadoop/pull/1635#issuecomment-577852404 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comm

[GitHub] [hadoop] hadoop-yetus commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode

2020-01-23 Thread GitBox
hadoop-yetus commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#issuecomment-577853532 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:---

[jira] [Commented] (HADOOP-16711) With S3Guard + authmode, consider skipping "verifyBuckets" check in S3A fs init()

2020-01-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022448#comment-17022448 ] Steve Loughran commented on HADOOP-16711: - Which s3 endpoint did you test again

[GitHub] [hadoop] mustafaiman commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
mustafaiman commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#issuecomment-577875161 Sorry, I had mentioned the testing in the jira ticket, forgot to add here. `tested with s3 us-east-1. -scale -

[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577902804 @steveloughran ``` we have the InconsistentS3AClient so simulate failures to S3; used for S3Guard. I've been wonderi

[GitHub] [hadoop] mustafaiman commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
mustafaiman commented on a change in pull request #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#discussion_r370410576 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard

[GitHub] [hadoop] mustafaiman commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
mustafaiman commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#issuecomment-577927953 I re-ran only the two tests as I only made changes to them in the subsequent commit. I used s3 us-west-2. ---

[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable

2020-01-23 Thread GitBox
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577943862 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|--

[GitHub] [hadoop] hadoop-yetus commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are…

2020-01-23 Thread GitBox
hadoop-yetus commented on issue #1815: HADOOP-16801: S3Guard-listFiles will not query s3 if all listings are… URL: https://github.com/apache/hadoop/pull/1815#issuecomment-577947692 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|

[GitHub] [hadoop] LeonGao91 closed pull request #1636: HDFS-14894. Add balancer parameter to balance top used nodes.

2020-01-23 Thread GitBox
LeonGao91 closed pull request #1636: HDFS-14894. Add balancer parameter to balance top used nodes. URL: https://github.com/apache/hadoop/pull/1636 This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [hadoop] LeonGao91 opened a new pull request #1816: HDFS-10648. Expose Balancer metrics through Metrics2.

2020-01-23 Thread GitBox
LeonGao91 opened a new pull request #1816: HDFS-10648. Expose Balancer metrics through Metrics2. URL: https://github.com/apache/hadoop/pull/1816 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which star

[GitHub] [hadoop] hadoop-yetus commented on issue #1816: HDFS-10648. Expose Balancer metrics through Metrics2.

2020-01-23 Thread GitBox
hadoop-yetus commented on issue #1816: HDFS-10648. Expose Balancer metrics through Metrics2. URL: https://github.com/apache/hadoop/pull/1816#issuecomment-577971161 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|--

[GitHub] [hadoop] DadanielZ closed pull request #1785: HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread GitBox
DadanielZ closed pull request #1785: HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation URL: https://github.com/apache/hadoop/pull/1785 This is an automated message from the Apache Git Serv

[GitHub] [hadoop] DadanielZ commented on issue #1785: HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread GitBox
DadanielZ commented on issue #1785: HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation URL: https://github.com/apache/hadoop/pull/1785#issuecomment-577988205 committed, thank you @karthick-rn --

[jira] [Commented] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread Da Zhou (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022681#comment-17022681 ] Da Zhou commented on HADOOP-16826: -- committed to trunk, thanks! > ABFS: update abfs.m

[jira] [Resolved] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread Da Zhou (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16826. -- Resolution: Fixed > ABFS: update abfs.md to include config keys for identity transformation > ---

[jira] [Commented] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation

2020-01-23 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022685#comment-17022685 ] Hudson commented on HADOOP-16826: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Co

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370473503 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -143,6 +143,15 @@

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370474451 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -194,7 +203,8 @@

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370478260 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebf

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370478672 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebf

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370476831 ## File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebf

[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob

2020-01-23 Thread GitBox
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob URL: https://github.com/apache/hadoop/pull/1790#discussion_r370474339 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -194,7 +203,8 @@