[GitHub] [hadoop] hadoop-yetus commented on pull request #3273: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845)
hadoop-yetus commented on pull request #3273:
URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894063057
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 13s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 1s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 59s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 72m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3273 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 610b7e95d601 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / af4adc2cbc66412de4fe602fd71dcb724f557f44 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/testReport/ |
| Max. process+thread count | 718 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[
https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=634976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634976
]
ASF GitHub Bot logged work on HADOOP-17618:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 07:28
Start Date: 06/Aug/21 07:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3273:
URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894063057
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 13s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 1s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 59s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 72m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3273 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 610b7e95d601 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / af4adc2cbc66412de4fe602fd71dcb724f557f44 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/testReport/ |
| Max. process+thread count | 718 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3273/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634976)
Time Spent: 9h 50m (was: 9h 40m)
> ABFS: Partially obfuscate SAS object IDs in Logs
>
>
> Key: HADOOP-17618
> URL: https://issues.apache.org/jira/browse/HADOOP-17618
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
>
[GitHub] [hadoop] sumangala-patki commented on pull request #3273: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845)
sumangala-patki commented on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894066970 ``` AppendBlob HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 76 HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 52 HNS-SharedKey [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 102, Failures: 1, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 NonHNS-SharedKey [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 ``` JIRAs to track failures: [TestAbfsClientThrottlingAnalyzer](https://issues.apache.org/jira/browse/HADOOP-17826), Appendblob [streamOps](https://issues.apache.org/jira/browse/HADOOP-17716), [Appendblob lease](https://issues.apache.org/jira/browse/HADOOP-17781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=634980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634980 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 07:35 Start Date: 06/Aug/21 07:35 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894066970 ``` AppendBlob HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 76 HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 52 HNS-SharedKey [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 102, Failures: 1, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 NonHNS-SharedKey [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 ``` JIRAs to track failures: [TestAbfsClientThrottlingAnalyzer](https://issues.apache.org/jira/browse/HADOOP-17826), Appendblob [streamOps](https://issues.apache.org/jira/browse/HADOOP-17716), [Appendblob lease](https://issues.apache.org/jira/browse/HADOOP-17781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634980) Time Spent: 10h (was: 9h 50m) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10h > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=634981&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634981 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 07:35 Start Date: 06/Aug/21 07:35 Worklog Time Spent: 10m Work Description: sumangala-patki edited a comment on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894066970 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` AppendBlob HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 76 HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 52 HNS-SharedKey [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 102, Failures: 1, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 NonHNS-SharedKey [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 ``` JIRAs to track failures: [TestAbfsClientThrottlingAnalyzer](https://issues.apache.org/jira/browse/HADOOP-17826), Appendblob [streamOps](https://issues.apache.org/jira/browse/HADOOP-17716), [Appendblob lease](https://issues.apache.org/jira/browse/HADOOP-17781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634981) Time Spent: 10h 10m (was: 10h) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10h 10m > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki edited a comment on pull request #3273: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845)
sumangala-patki edited a comment on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894066970 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` AppendBlob HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 76 HNS-OAuth [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 52 HNS-SharedKey [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 102, Failures: 1, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 NonHNS-SharedKey [INFO] Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276 [WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 40 ``` JIRAs to track failures: [TestAbfsClientThrottlingAnalyzer](https://issues.apache.org/jira/browse/HADOOP-17826), Appendblob [streamOps](https://issues.apache.org/jira/browse/HADOOP-17716), [Appendblob lease](https://issues.apache.org/jira/browse/HADOOP-17781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
aajisaka commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r684018904
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
+try {
+ processTokenAddOrUpdate(childData.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CREATED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forChanges((oldNode, node) -> {
+try {
+ processTokenAddOrUpdate(node.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CHANGED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forDeletes(childData -> {
+try {
+ processTokenRemoved(childData);
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tok
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634984&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634984
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 07:40
Start Date: 06/Aug/21 07:40
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r684018904
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
+try {
+ processTokenAddOrUpdate(childData.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CREATED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forChanges((oldNode, node) -> {
+try {
+ processTokenAddOrUpdate(node.getData());
+} catch (IOException e) {
+
[GitHub] [hadoop] hadoop-yetus commented on pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
hadoop-yetus commented on pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#issuecomment-894082260
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 247m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 332m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3269 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dca979d5987b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 456822bf8be0d09a5c3538ef0b280199f21b1a2b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/3/testReport/ |
| Max. process+thread count | 3789 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about thi
[jira] [Assigned] (HADOOP-17370) Upgrade commons-compress to 1.21
[ https://issues.apache.org/jira/browse/HADOOP-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HADOOP-17370: -- Assignee: Akira Ajisaka > Upgrade commons-compress to 1.21 > > > Key: HADOOP-17370 > URL: https://issues.apache.org/jira/browse/HADOOP-17370 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0, 3.2.1 >Reporter: Dongjoon Hyun >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684046420
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
did you meant to check for `!endpoint.isEmpty()`?
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/AbstractS3ACostTest.java
##
@@ -125,6 +126,13 @@ public AbstractS3ACostTest(
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
String bucketName = getTestBucketName(conf);
+// If AccessPoint ARN is set guarded tests are skipped
+String apArn = conf.get(ACCESS_POINT_ARN, "");
+if (isGuarded() && !apArn.isEmpty()) {
+ LOG.warn("Skipping test since AccessPoint ARN is set and is incompatible
with S3Guard.");
+ Assume.assumeFalse(true);
Review comment:
+1, also better to use `ContractUtils.skip(SKIP_MESSAGE)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel merged pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
sodonnel merged pull request #3269: URL: https://github.com/apache/hadoop/pull/3269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=634997&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634997
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 08:27
Start Date: 06/Aug/21 08:27
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684046420
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
did you meant to check for `!endpoint.isEmpty()`?
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/AbstractS3ACostTest.java
##
@@ -125,6 +126,13 @@ public AbstractS3ACostTest(
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
String bucketName = getTestBucketName(conf);
+// If AccessPoint ARN is set guarded tests are skipped
+String apArn = conf.get(ACCESS_POINT_ARN, "");
+if (isGuarded() && !apArn.isEmpty()) {
+ LOG.warn("Skipping test since AccessPoint ARN is set and is incompatible
with S3Guard.");
+ Assume.assumeFalse(true);
Review comment:
+1, also better to use `ContractUtils.skip(SKIP_MESSAGE)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634997)
Time Spent: 2h 40m (was: 2.5h)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #3274: HADOOP-17370. Upgrade commons-compress to 1.21
aajisaka opened a new pull request #3274: URL: https://github.com/apache/hadoop/pull/3274 JIRA: HADOOP-17370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17370) Upgrade commons-compress to 1.21
[ https://issues.apache.org/jira/browse/HADOOP-17370?focusedWorklogId=634999&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634999 ] ASF GitHub Bot logged work on HADOOP-17370: --- Author: ASF GitHub Bot Created on: 06/Aug/21 08:34 Start Date: 06/Aug/21 08:34 Worklog Time Spent: 10m Work Description: aajisaka opened a new pull request #3274: URL: https://github.com/apache/hadoop/pull/3274 JIRA: HADOOP-17370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634999) Time Spent: 2.5h (was: 2h 20m) > Upgrade commons-compress to 1.21 > > > Key: HADOOP-17370 > URL: https://issues.apache.org/jira/browse/HADOOP-17370 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0, 3.2.1 >Reporter: Dongjoon Hyun >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #3272: HADOOP-17837: Add unresolved endpoint value to UnknownHostException
jojochuang merged pull request #3272: URL: https://github.com/apache/hadoop/pull/3272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?focusedWorklogId=635003&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635003 ] ASF GitHub Bot logged work on HADOOP-17837: --- Author: ASF GitHub Bot Created on: 06/Aug/21 09:00 Start Date: 06/Aug/21 09:00 Worklog Time Spent: 10m Work Description: jojochuang merged pull request #3272: URL: https://github.com/apache/hadoop/pull/3272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635003) Time Spent: 0.5h (was: 20m) > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-17837: Assignee: Bryan Beaudreault > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-17837. -- Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~bbeaudreault]. The PR is merged. > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-17837: - Fix Version/s: 3.3.2 > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
tomscut commented on a change in pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#discussion_r684097589
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java
##
@@ -585,8 +585,10 @@ private void
selectRpcInputStreams(Collection streams,
int maxAllowedTxns = !onlyDurableTxns ? highestTxnCount :
responseCounts.get(responseCounts.size() - loggers.getMajoritySize());
if (maxAllowedTxns == 0) {
- LOG.debug("No new edits available in logs; requested starting from " +
- "ID " + fromTxnId);
+ if (LOG.isDebugEnabled()) {
+LOG.debug("No new edits available in logs; requested starting from " +
+"ID " + fromTxnId);
+ }
Review comment:
> If would be better to remove the `if (LOG.isDebugEnabled())` and
change the log statement to:
>
> ```
> LOG.debug("No new edits available in logs; requested starting from ID {}",
fromTxnId);
> ```
>
> This avoids any evaluation and inside the debug method, it checks if debug
is enabled or not before evaluating the string.
This is a good solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3274: HADOOP-17370. Upgrade commons-compress to 1.21
hadoop-yetus commented on pull request #3274:
URL: https://github.com/apache/hadoop/pull/3274#issuecomment-894138952
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 45m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 14m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 18s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 63m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3274 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux c2bc27f833f6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2bcd8b9d4089b62e97185b5a19f4f5799a818e2b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/testReport/ |
| Max. process+thread count | 727 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h
[jira] [Work logged] (HADOOP-17370) Upgrade commons-compress to 1.21
[
https://issues.apache.org/jira/browse/HADOOP-17370?focusedWorklogId=635020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635020
]
ASF GitHub Bot logged work on HADOOP-17370:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 09:40
Start Date: 06/Aug/21 09:40
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3274:
URL: https://github.com/apache/hadoop/pull/3274#issuecomment-894138952
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 45m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 14m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 18s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 63m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3274 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux c2bc27f833f6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2bcd8b9d4089b62e97185b5a19f4f5799a818e2b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/testReport/ |
| Max. process+thread count | 727 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3274/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the m
[jira] [Created] (HADOOP-17838) Update link of PoweredBy wiki page
Yi-Sheng Lien created HADOOP-17838: -- Summary: Update link of PoweredBy wiki page Key: HADOOP-17838 URL: https://issues.apache.org/jira/browse/HADOOP-17838 Project: Hadoop Common Issue Type: Improvement Reporter: Yi-Sheng Lien Assignee: Yi-Sheng Lien The [PoweredBy wiki page|https://cwiki.apache.org/confluence/display/hadoop/PoweredBy] on main page is not found. IMHO update it to [here|https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17838) Update link of PoweredBy wiki page
[ https://issues.apache.org/jira/browse/HADOOP-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi-Sheng Lien updated HADOOP-17838: --- Description: The [PoweredBy wiki page|https://cwiki.apache.org/confluence/display/hadoop/PoweredBy] on [main page|https://hadoop.apache.org/] is not found. IMHO update it to [here|https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy] was: The [PoweredBy wiki page|https://cwiki.apache.org/confluence/display/hadoop/PoweredBy] on main page is not found. IMHO update it to [here|https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy] > Update link of PoweredBy wiki page > -- > > Key: HADOOP-17838 > URL: https://issues.apache.org/jira/browse/HADOOP-17838 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yi-Sheng Lien >Assignee: Yi-Sheng Lien >Priority: Trivial > > The [PoweredBy wiki > page|https://cwiki.apache.org/confluence/display/hadoop/PoweredBy] on [main > page|https://hadoop.apache.org/] is not found. > IMHO update it to > [here|https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684101247
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
##
@@ -214,6 +220,31 @@ public void sign(SignableRequest request,
AWSCredentials credentials) {
}
}
+private String parseBucketFromHost(String host) {
+ // host: {bucket || accesspoint}.{s3 ||
s3-accesspoint}.{region}.amazonaws.com
+ String[] hostBits = host.split("\\.");
+ String bucketName = hostBits[0];
+ String service = hostBits[1];
+
+ if (service.contains("s3-accesspoint") ||
service.contains("s3-outposts") ||
+ service.contains("s3-object-lambda")) {
+// If AccessPoint then bucketName is of format `accessPoint-accountId`;
+String[] accessPointBits = hostBits[0].split("\\-");
Review comment:
what if AccessPoint Name have a "-" in it?
for eg: AP name = "mmt-ap", then `bucketName = "mmt-ap-ACCOUNT_ID"`, with
this split, we would end up with "mmt" as `accessPointName` and "ap" as
`accountID`.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -41,6 +41,7 @@
import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.ACCESS_POINT_ARN;
Review comment:
nit: unused import.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -25,6 +25,8 @@
import static
org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM;
Review comment:
nit: after refactoring, these would be becoming unused imports.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -39,12 +41,10 @@ protected Configuration createConfiguration() {
// get the KMS key for this test.
Configuration c = new Configuration();
String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
-if (StringUtils.isBlank(kmsKey) || !c.get(SERVER_SIDE_ENCRYPTION_ALGORITHM)
-.equals(S3AEncryptionMethods.CSE_KMS.name())) {
- skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
- SSE_KMS.getMethod() + " or CSE-KMS algorithm is used instead of "
- + "SSE-KMS");
-}
+
+skipIfKmsKeyIdIsNotSet(c);
+skipIfCSEIsNotEnabled(c);
Review comment:
Seems like my code had a bug in it. This should be skipping if CSE is
enabled or if KMS key is not set.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContractGetFileStatusV1List.java
##
@@ -18,11 +18,17 @@
package org.apache.hadoop.fs.s3a;
+import java.io.IOException;
+
+import org.junit.Assume;
+import org.junit.Test;
Review comment:
nit: unused imports.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/ArnResource.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import javax.annotation.Nonnull;
+
+import com.amazonaws.arn.Arn;
+import com.amazonaws.regions.RegionUtils;
+
+/**
+ * Represents an Arn Resource, this can be an accesspoint or bucket.
+ */
+public class ArnResource {
+
+ /**
+ * Resource name.
+ */
+ private final String name;
+
+ /**
+ * Resource owner account id.
+ */
+ private final String ownerAccountId;
+
+ /**
+ * Resource region.
+ */
+ private final String region;
+
+ /**
+ * Full Arn for the resource.
+ */
+ private final String fullArn;
+
+ /**
+ * Partition for the resource. Allowed partitions: aws, aws-cn, aws-us-gov
+ */
+ private final String partition;
+
+ /**
+ * Because of the different ways an endpoint can be constructed depending on
partition we're
+ * relying on the AWS SDK to produce the endpoint. In this case we need a
region key of the form
+ * {@code String.format("accesspoint-%s", awsRegion)}
+ */
+ private fina
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635033&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635033
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 10:18
Start Date: 06/Aug/21 10:18
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684101247
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
##
@@ -214,6 +220,31 @@ public void sign(SignableRequest request,
AWSCredentials credentials) {
}
}
+private String parseBucketFromHost(String host) {
+ // host: {bucket || accesspoint}.{s3 ||
s3-accesspoint}.{region}.amazonaws.com
+ String[] hostBits = host.split("\\.");
+ String bucketName = hostBits[0];
+ String service = hostBits[1];
+
+ if (service.contains("s3-accesspoint") ||
service.contains("s3-outposts") ||
+ service.contains("s3-object-lambda")) {
+// If AccessPoint then bucketName is of format `accessPoint-accountId`;
+String[] accessPointBits = hostBits[0].split("\\-");
Review comment:
what if AccessPoint Name have a "-" in it?
for eg: AP name = "mmt-ap", then `bucketName = "mmt-ap-ACCOUNT_ID"`, with
this split, we would end up with "mmt" as `accessPointName` and "ap" as
`accountID`.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -41,6 +41,7 @@
import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.ACCESS_POINT_ARN;
Review comment:
nit: unused import.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -25,6 +25,8 @@
import static
org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM;
Review comment:
nit: after refactoring, these would be becoming unused imports.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -39,12 +41,10 @@ protected Configuration createConfiguration() {
// get the KMS key for this test.
Configuration c = new Configuration();
String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
-if (StringUtils.isBlank(kmsKey) || !c.get(SERVER_SIDE_ENCRYPTION_ALGORITHM)
-.equals(S3AEncryptionMethods.CSE_KMS.name())) {
- skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
- SSE_KMS.getMethod() + " or CSE-KMS algorithm is used instead of "
- + "SSE-KMS");
-}
+
+skipIfKmsKeyIdIsNotSet(c);
+skipIfCSEIsNotEnabled(c);
Review comment:
Seems like my code had a bug in it. This should be skipping if CSE is
enabled or if KMS key is not set.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContractGetFileStatusV1List.java
##
@@ -18,11 +18,17 @@
package org.apache.hadoop.fs.s3a;
+import java.io.IOException;
+
+import org.junit.Assume;
+import org.junit.Test;
Review comment:
nit: unused imports.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/ArnResource.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import javax.annotation.Nonnull;
+
+import com.amazonaws.arn.Arn;
+import com.amazonaws.regions.RegionUtils;
+
+/**
+ * Represents an Arn Resource, this can be an accesspoint or bucket.
+ */
+public class ArnResource {
+
+ /**
+ * Resource name.
+ */
+ private final String name;
+
+ /**
+ * Resource owner account id.
+ */
+ private final String ownerAccountId;
+
+ /**
+ * Resource region.
+ */
+ private final String region;
+
+ /**
+ * Full Arn for the resource.
+ */
+ private fina
[GitHub] [hadoop] jojochuang merged pull request #3270: HDFS-16154. TestMiniJournalCluster failing intermittently because of …
jojochuang merged pull request #3270: URL: https://github.com/apache/hadoop/pull/3270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3272: HADOOP-17837: Add unresolved endpoint value to UnknownHostException
steveloughran commented on a change in pull request #3272:
URL: https://github.com/apache/hadoop/pull/3272#discussion_r684137107
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java
##
@@ -111,6 +111,7 @@ public void testInvalidAddress() throws Throwable {
fail("Should not have connected");
} catch (UnknownHostException uhe) {
LOG.info("Got exception: ", uhe);
+ assertEquals("invalid-test-host:0", uhe.getMessage());
Review comment:
Use `GenericTestUtils.assertExceptionContains()`; this rethrows the
exception if there's no match, and allows for extra text in the messge
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##
@@ -589,7 +589,7 @@ public static void connect(Socket socket,
} catch (SocketTimeoutException ste) {
throw new ConnectTimeoutException(ste.getMessage());
} catch (UnresolvedAddressException uae) {
- throw new UnknownHostException(uae.getMessage());
+ throw new UnknownHostException(endpoint.toString());
Review comment:
we still need that message from the original string so build it from
`uhe.getMessage() + " " + endpoint`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17833) Improve Magic Committer Performane
[ https://issues.apache.org/jira/browse/HADOOP-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17833: Summary: Improve Magic Committer Performane (was: createFile() under a magic path to skip all probes for file/dir at end of path) > Improve Magic Committer Performane > -- > > Key: HADOOP-17833 > URL: https://issues.apache.org/jira/browse/HADOOP-17833 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Magic committer tasks can be slow because every file created with > overwrite=false triggers a HEAD (verify there's no file) and a LIST (that > there's no dir). And because of delayed manifestations, it may not behave as > expected. > ParquetOutputFormat is one example of a library which does this. > we could fix parquet to use overwrite=true, but (a) there may be surprises in > other uses (b) it'd still leave the list and (c) do nothing for other formats > call > Proposed: createFile() under a magic path to skip all probes for file/dir at > end of path > Only a single task attempt Will be writing to that directory and it should > know what it is doing. If there is conflicting file names and parts across > tasks that won't even get picked up at this point. Oh and none of the > committers ever check for this: you'll get the last file manifested (s3a) or > renamed (file) > If we skip the checks we will save 2 HTTP requests/file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[
https://issues.apache.org/jira/browse/HADOOP-17837?focusedWorklogId=635044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635044
]
ASF GitHub Bot logged work on HADOOP-17837:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 10:42
Start Date: 06/Aug/21 10:42
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3272:
URL: https://github.com/apache/hadoop/pull/3272#discussion_r684137107
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java
##
@@ -111,6 +111,7 @@ public void testInvalidAddress() throws Throwable {
fail("Should not have connected");
} catch (UnknownHostException uhe) {
LOG.info("Got exception: ", uhe);
+ assertEquals("invalid-test-host:0", uhe.getMessage());
Review comment:
Use `GenericTestUtils.assertExceptionContains()`; this rethrows the
exception if there's no match, and allows for extra text in the messge
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##
@@ -589,7 +589,7 @@ public static void connect(Socket socket,
} catch (SocketTimeoutException ste) {
throw new ConnectTimeoutException(ste.getMessage());
} catch (UnresolvedAddressException uae) {
- throw new UnknownHostException(uae.getMessage());
+ throw new UnknownHostException(endpoint.toString());
Review comment:
we still need that message from the original string so build it from
`uhe.getMessage() + " " + endpoint`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635044)
Time Spent: 40m (was: 0.5h)
> Make it easier to debug UnknownHostExceptions from NetUtils.connect
> ---
>
> Key: HADOOP-17837
> URL: https://issues.apache.org/jira/browse/HADOOP-17837
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Most UnknownHostExceptions thrown throughout hadoop include a useful message,
> either the hostname that was not found or some other descriptor of the
> problem. The UnknownHostException thrown from NetUtils.connect only includes
> the [message of the underlying
> UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592].
> If you take a look at the source for UnresolvedAddressException, [it only
> has a no-args
> constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html]
> (java11, but same is true in other versions). So it never has a message,
> meaning the UnknownHostException message is empty.
> We should include the endpoint.toString() in the UnknownHostException thrown
> by NetUtils.connect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17833) Improve Magic Committer Performane
[ https://issues.apache.org/jira/browse/HADOOP-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394697#comment-17394697 ] Steve Loughran commented on HADOOP-17833: - Thinking about other improvements * knowing that all committers must be on the same (marker aware) Hadoop release, we should enable marker retention on every magic path. Saves on DELETE requests * skip the mkdirs() in task setup; saves on scan up tree and PUT; will need to make sure task commit is OK with FNFE on list * fix s3a openFile().with(FileStatus) to accept file status not an instance of S3AFS (in the openFile() enhancements patch, but we only need this), and JsonSerDeser to pass it down when opening a file. Saves on HEAD request when going from dir list to opening a file in task and job commit * make sure job commit is optimised as it is the critical path for compute * maybe: collect task commit stats as the manifest committer will do. Might be best done first for measuring optimisation; * include those of input and output streams if we can enhance json ser deser to add ability (new methods) to return them The first thre are strightforward with minimal production code changes; tests not that difficult. > Improve Magic Committer Performane > -- > > Key: HADOOP-17833 > URL: https://issues.apache.org/jira/browse/HADOOP-17833 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Magic committer tasks can be slow because every file created with > overwrite=false triggers a HEAD (verify there's no file) and a LIST (that > there's no dir). And because of delayed manifestations, it may not behave as > expected. > ParquetOutputFormat is one example of a library which does this. > we could fix parquet to use overwrite=true, but (a) there may be surprises in > other uses (b) it'd still leave the list and (c) do nothing for other formats > call > Proposed: createFile() under a magic path to skip all probes for file/dir at > end of path > Only a single task attempt Will be writing to that directory and it should > know what it is doing. If there is conflicting file names and parts across > tasks that won't even get picked up at this point. Oh and none of the > committers ever check for this: you'll get the last file manifested (s3a) or > renamed (file) > If we skip the checks we will save 2 HTTP requests/file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2845: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs
hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-816469810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2845: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs
hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-822958936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2845: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs
hadoop-yetus removed a comment on pull request #2845:
URL: https://github.com/apache/hadoop/pull/2845#issuecomment-825696671
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 81m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2845 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 524ec6161dfd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6a755632b83621204acdd6e29d0a10042690f1fb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/testReport/ |
| Max. process+thread count | 600 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
--
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=635055&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635055 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 10:58 Start Date: 06/Aug/21 10:58 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-816469810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635055) Time Spent: 10h 20m (was: 10h 10m) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10h 20m > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=635056&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635056 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 10:58 Start Date: 06/Aug/21 10:58 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-822958936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635056) Time Spent: 10.5h (was: 10h 20m) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10.5h > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[
https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=635057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635057
]
ASF GitHub Bot logged work on HADOOP-17618:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 10:58
Start Date: 06/Aug/21 10:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2845:
URL: https://github.com/apache/hadoop/pull/2845#issuecomment-825696671
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 81m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2845 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 524ec6161dfd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6a755632b83621204acdd6e29d0a10042690f1fb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/testReport/ |
| Max. process+thread count | 600 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2845/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
T
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2845: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs
hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-892181247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3273: HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845)
steveloughran commented on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894180995 +1 pending yetus being happy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=635059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635059 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 11:00 Start Date: 06/Aug/21 11:00 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3273: URL: https://github.com/apache/hadoop/pull/3273#issuecomment-894180995 +1 pending yetus being happy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635059) Time Spent: 11h (was: 10h 50m) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 11h > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?focusedWorklogId=635058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635058 ] ASF GitHub Bot logged work on HADOOP-17618: --- Author: ASF GitHub Bot Created on: 06/Aug/21 10:59 Start Date: 06/Aug/21 10:59 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2845: URL: https://github.com/apache/hadoop/pull/2845#issuecomment-892181247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635058) Time Spent: 10h 50m (was: 10h 40m) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10h 50m > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147473
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
Review comment:
Yes! That would be the desired outcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147618
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -400,6 +410,14 @@ public void initialize(URI name, Configuration
originalConf)
LOG.debug("Initializing S3AFileSystem for {}", bucket);
// clone the configuration into one with propagated bucket options
Configuration conf = propagateBucketOptions(originalConf, bucket);
+
+ String apArn = conf.getTrimmed(ACCESS_POINT_ARN, "");
Review comment:
Yup, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635060
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:01
Start Date: 06/Aug/21 11:01
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147473
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
Review comment:
Yes! That would be the desired outcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635060)
Time Spent: 3h (was: 2h 50m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635061&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635061
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:01
Start Date: 06/Aug/21 11:01
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147618
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -400,6 +410,14 @@ public void initialize(URI name, Configuration
originalConf)
LOG.debug("Initializing S3AFileSystem for {}", bucket);
// clone the configuration into one with propagated bucket options
Configuration conf = propagateBucketOptions(originalConf, bucket);
+
+ String apArn = conf.getTrimmed(ACCESS_POINT_ARN, "");
Review comment:
Yup, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635061)
Time Spent: 3h 10m (was: 3h)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147891
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
+ arn:aws:s3:eu-west-1:123456789101:accesspoint/finance-accesspoint
+
+```
+
+While keeping the global `accesspoint.arn` property set to empty `" "` which
is the default.
Review comment:
Yeah, my fault for mixing it up. I thought the default was `" "` not
`""` for properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635062
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:02
Start Date: 06/Aug/21 11:02
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684147891
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
+ arn:aws:s3:eu-west-1:123456789101:accesspoint/finance-accesspoint
+
+```
+
+While keeping the global `accesspoint.arn` property set to empty `" "` which
is the default.
Review comment:
Yeah, my fault for mixing it up. I thought the default was `" "` not
`""` for properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635062)
Time Spent: 3h 20m (was: 3h 10m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17785) mvn test failed about hadoop@3.2.1
[
https://issues.apache.org/jira/browse/HADOOP-17785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17785.
-
Resolution: Works for Me
you are probably running as root, so all permissions tests fail.
> mvn test failed about hadoop@3.2.1
> --
>
> Key: HADOOP-17785
> URL: https://issues.apache.org/jira/browse/HADOOP-17785
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.2.1
>Reporter: shixijun
>Priority: Minor
>
> {panel:title=mvn test failed about hadoop@3.2.1}
> mvn test failed
> {panel}
> [root@localhost spack-src]# mvn -version
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home:
> /home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/maven-3.6.3-fpgpwvz7es5yiaz2tez2pnlilrcatuvg
> Java version: 1.8.0_191, vendor: AdoptOpenJdk, runtime:
> /home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/openjdk-1.8.0_191-b12-fidptihybskgklbjoo4lagkacm6n6lod/jre
> Default locale: en_US, platform encoding: ANSI_X3.4-1968
> OS name: "linux", version: "4.18.0-80.el8.aarch64", arch: "aarch64", family:
> "unix"
> [root@localhost spack-src]# java -version
> openjdk version "1.8.0_191"
> OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_191-b12)
> OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.191-b12, mixed mode)
> [root@localhost spack-src]# mvn test
> ……
> [INFO] Running org.apache.hadoop.tools.TestCommandShell
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111
> s - in org.apache.hadoop.tools.TestCommandShell
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]
> TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> [ERROR]
> TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288
> Should throw IOException
> [ERROR]
> TestFileUtil.testFailFullyDelete:446->validateAndSetWritablePermissions:422
> The directory xSubDir *should* not have been deleted. expected: but
> was:
> [ERROR]
> TestFileUtil.testFailFullyDeleteContents:525->validateAndSetWritablePermissions:422
> The directory xSubDir *should* not have been deleted. expected: but
> was:
> [ERROR] TestFileUtil.testGetDU:571
> [ERROR] TestFsShellCopy.testPutSrcDirNoPerm:627->shellRun:80 expected:<1>
> but was:<0>
> [ERROR] TestFsShellCopy.testPutSrcFileNoPerm:652->shellRun:80 expected:<1>
> but was:<0>
> [ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109
> Checking for build/test/temp/RELATIVE1 in
> build/test/temp/RELATIVE0/block995011826146306285.tmp - FAILED!
> [ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109
> Checking for
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block792666236482175348.tmp
> - FAILED!
> [ERROR] TestLocalDirAllocator.test0:141->validateTempDirCreation:109
> Checking for
> file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5124616846677903649.tmp
> - FAILED!
> [ERROR]
> TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109
> Checking for build/test/temp/RELATIVE2 in
> build/test/temp/RELATIVE1/block1176062344115776027.tmp - FAILED!
> [ERROR]
> TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
> Checking for
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block3514694215643608527.tmp
> - FAILED!
> [ERROR]
> TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
> Checking for
> file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/h
[jira] [Created] (HADOOP-17839) LocalFS to support ability to disable permission get/set; remove need for winutils
Steve Loughran created HADOOP-17839: --- Summary: LocalFS to support ability to disable permission get/set; remove need for winutils Key: HADOOP-17839 URL: https://issues.apache.org/jira/browse/HADOOP-17839 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.3.1 Reporter: Steve Loughran Setting FS permissions on windows has always been a paint point for people running spark standalone, creating files through localFS required WINUTILS.EX To do this I'm seeing some other demand for disabling setting permissions, primarily because if you mount storage (including azure storage) to a VM, you cant call setPermissions; things fail. Proposed: 1. we add an option to disable permission setting 2. we add a path capability, which for hdfs, abfs is always true (not sure about wasb as it varies on file vs dir) 3. we add the option to turn off permission get/set for RawLocalFileSystem and downgrade to a no-op on write, ug+rw on read (or even make configurable?) (Distcp should maybe probe the path and downgrade -p options if the dest store doesn't do perms properly, which argues for wasb to return true, so file perms are set) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop-site] cxorm commented on pull request #26: HADOOP-17821. Move Ozone to related projects section
cxorm commented on pull request #26: URL: https://github.com/apache/hadoop-site/pull/26#issuecomment-894189153 Thanks @jojochuang for the review, could you be so kind as to take a look on it again : ) New patch is updated and description is refer from [Ozone site](https://ozone.apache.org/), please correct me if I miss somthing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17821) Move Ozone to related projects section
[ https://issues.apache.org/jira/browse/HADOOP-17821?focusedWorklogId=635067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635067 ] ASF GitHub Bot logged work on HADOOP-17821: --- Author: ASF GitHub Bot Created on: 06/Aug/21 11:14 Start Date: 06/Aug/21 11:14 Worklog Time Spent: 10m Work Description: cxorm commented on pull request #26: URL: https://github.com/apache/hadoop-site/pull/26#issuecomment-894189153 Thanks @jojochuang for the review, could you be so kind as to take a look on it again : ) New patch is updated and description is refer from [Ozone site](https://ozone.apache.org/), please correct me if I miss somthing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635067) Time Spent: 40m (was: 0.5h) > Move Ozone to related projects section > -- > > Key: HADOOP-17821 > URL: https://issues.apache.org/jira/browse/HADOOP-17821 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yi-Sheng Lien >Assignee: Yi-Sheng Lien >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Hi all, as Ozone was spun to TLP, it has individual web site. > Now on Modules part of Hadoop [website|https://hadoop.apache.org/], the link > of Ozone website is old page. > IMHO there are two ways to fix it : > 1. update it to new page. > 2. move Ozone to Related projects part on Hadoop website > Please feel free to give me some feedback, thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault commented on a change in pull request #3272: HADOOP-17837: Add unresolved endpoint value to UnknownHostException
bbeaudreault commented on a change in pull request #3272:
URL: https://github.com/apache/hadoop/pull/3272#discussion_r684155669
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##
@@ -589,7 +589,7 @@ public static void connect(Socket socket,
} catch (SocketTimeoutException ste) {
throw new ConnectTimeoutException(ste.getMessage());
} catch (UnresolvedAddressException uae) {
- throw new UnknownHostException(uae.getMessage());
+ throw new UnknownHostException(endpoint.toString());
Review comment:
Thank you for taking a look. I considered this, but a UHE cannot have a
message. It only as has no args constructor so it cannot be created with a
message. There's also no setMessage method as far as I can tell, so can't be
added after.
https://docs.oracle.com/javase/8/docs/api/java/net/UnknownHostException.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[
https://issues.apache.org/jira/browse/HADOOP-17837?focusedWorklogId=635068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635068
]
ASF GitHub Bot logged work on HADOOP-17837:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:17
Start Date: 06/Aug/21 11:17
Worklog Time Spent: 10m
Work Description: bbeaudreault commented on a change in pull request
#3272:
URL: https://github.com/apache/hadoop/pull/3272#discussion_r684155669
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##
@@ -589,7 +589,7 @@ public static void connect(Socket socket,
} catch (SocketTimeoutException ste) {
throw new ConnectTimeoutException(ste.getMessage());
} catch (UnresolvedAddressException uae) {
- throw new UnknownHostException(uae.getMessage());
+ throw new UnknownHostException(endpoint.toString());
Review comment:
Thank you for taking a look. I considered this, but a UHE cannot have a
message. It only as has no args constructor so it cannot be created with a
message. There's also no setMessage method as far as I can tell, so can't be
added after.
https://docs.oracle.com/javase/8/docs/api/java/net/UnknownHostException.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635068)
Time Spent: 50m (was: 40m)
> Make it easier to debug UnknownHostExceptions from NetUtils.connect
> ---
>
> Key: HADOOP-17837
> URL: https://issues.apache.org/jira/browse/HADOOP-17837
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Most UnknownHostExceptions thrown throughout hadoop include a useful message,
> either the hostname that was not found or some other descriptor of the
> problem. The UnknownHostException thrown from NetUtils.connect only includes
> the [message of the underlying
> UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592].
> If you take a look at the source for UnresolvedAddressException, [it only
> has a no-args
> constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html]
> (java11, but same is true in other versions). So it never has a message,
> meaning the UnknownHostException message is empty.
> We should include the endpoint.toString() in the UnknownHostException thrown
> by NetUtils.connect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684160558
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
No, what I initially intended is to say "if you're not setting the
endpoint then I'll provide a default Access Point endpoint". This is because I
don't know what endpoint the user wants to point it to.
This is also why your tests are failing when you set the endpoint to
`ap-south-1`. I'm open to adding another `fs.s3a.accesspoint-endpoint`
configuration if it's better to provide an option to override only the access
point endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635070
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:26
Start Date: 06/Aug/21 11:26
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684160558
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
No, what I initially intended is to say "if you're not setting the
endpoint then I'll provide a default Access Point endpoint". This is because I
don't know what endpoint the user wants to point it to.
This is also why your tests are failing when you set the endpoint to
`ap-south-1`. I'm open to adding another `fs.s3a.accesspoint-endpoint`
configuration if it's better to provide an option to override only the access
point endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635070)
Time Spent: 3.5h (was: 3h 20m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
Bryan Beaudreault created HADOOP-17840: -- Summary: Backport HADOOP-17837 to branch-3.2 Key: HADOOP-17840 URL: https://issues.apache.org/jira/browse/HADOOP-17840 Project: Hadoop Common Issue Type: Sub-task Reporter: Bryan Beaudreault Assignee: Bryan Beaudreault -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684164823
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
##
@@ -214,6 +220,31 @@ public void sign(SignableRequest request,
AWSCredentials credentials) {
}
}
+private String parseBucketFromHost(String host) {
+ // host: {bucket || accesspoint}.{s3 ||
s3-accesspoint}.{region}.amazonaws.com
+ String[] hostBits = host.split("\\.");
+ String bucketName = hostBits[0];
+ String service = hostBits[1];
+
+ if (service.contains("s3-accesspoint") ||
service.contains("s3-outposts") ||
+ service.contains("s3-object-lambda")) {
+// If AccessPoint then bucketName is of format `accessPoint-accountId`;
+String[] accessPointBits = hostBits[0].split("\\-");
Review comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635075
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:34
Start Date: 06/Aug/21 11:34
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684164823
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
##
@@ -214,6 +220,31 @@ public void sign(SignableRequest request,
AWSCredentials credentials) {
}
}
+private String parseBucketFromHost(String host) {
+ // host: {bucket || accesspoint}.{s3 ||
s3-accesspoint}.{region}.amazonaws.com
+ String[] hostBits = host.split("\\.");
+ String bucketName = hostBits[0];
+ String service = hostBits[1];
+
+ if (service.contains("s3-accesspoint") ||
service.contains("s3-outposts") ||
+ service.contains("s3-object-lambda")) {
+// If AccessPoint then bucketName is of format `accessPoint-accountId`;
+String[] accessPointBits = hostBits[0].split("\\-");
Review comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635075)
Time Spent: 3h 40m (was: 3.5h)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684165520
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -39,12 +41,10 @@ protected Configuration createConfiguration() {
// get the KMS key for this test.
Configuration c = new Configuration();
String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
-if (StringUtils.isBlank(kmsKey) || !c.get(SERVER_SIDE_ENCRYPTION_ALGORITHM)
-.equals(S3AEncryptionMethods.CSE_KMS.name())) {
- skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
- SSE_KMS.getMethod() + " or CSE-KMS algorithm is used instead of "
- + "SSE-KMS");
-}
+
+skipIfKmsKeyIdIsNotSet(c);
+skipIfCSEIsNotEnabled(c);
Review comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635077
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:35
Start Date: 06/Aug/21 11:35
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684165520
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
##
@@ -39,12 +41,10 @@ protected Configuration createConfiguration() {
// get the KMS key for this test.
Configuration c = new Configuration();
String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
-if (StringUtils.isBlank(kmsKey) || !c.get(SERVER_SIDE_ENCRYPTION_ALGORITHM)
-.equals(S3AEncryptionMethods.CSE_KMS.name())) {
- skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
- SSE_KMS.getMethod() + " or CSE-KMS algorithm is used instead of "
- + "SSE-KMS");
-}
+
+skipIfKmsKeyIdIsNotSet(c);
+skipIfCSEIsNotEnabled(c);
Review comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635077)
Time Spent: 3h 50m (was: 3h 40m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault opened a new pull request #3275: HADOOP-17840: Backport HADOOP-17837 to branch-3.2
bbeaudreault opened a new pull request #3275: URL: https://github.com/apache/hadoop/pull/3275 https://issues.apache.org/jira/browse/HADOOP-17840 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684165971
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/LoggingHttpResponseEncoder.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapred;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+import io.netty.handler.codec.http.HttpHeaders;
+import io.netty.handler.codec.http.HttpResponse;
+import io.netty.handler.codec.http.HttpResponseEncoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+class LoggingHttpResponseEncoder extends HttpResponseEncoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(LoggingHttpResponseEncoder.class);
+ private final boolean logStacktraceOfEncodingMethods;
+
+ public LoggingHttpResponseEncoder(boolean logStacktraceOfEncodingMethods) {
+this.logStacktraceOfEncodingMethods = logStacktraceOfEncodingMethods;
+ }
+
+ @Override
+ public boolean acceptOutboundMessage(Object msg) throws Exception {
+printExecutingMethod();
+LOG.info("OUTBOUND MESSAGE: " + msg);
+return super.acceptOutboundMessage(msg);
+ }
+
+ @Override
+ protected void encodeInitialLine(ByteBuf buf, HttpResponse response) throws
Exception {
+LOG.debug("Executing method: {}, response: {}",
+getExecutingMethodName(), response);
+logStacktraceIfRequired();
+super.encodeInitialLine(buf, response);
+ }
+
+ @Override
+ protected void encode(ChannelHandlerContext ctx, Object msg,
+ List out) throws Exception {
+LOG.debug("Encoding to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+logStacktraceIfRequired();
+super.encode(ctx, msg, out);
+ }
+
+ @Override
+ protected void encodeHeaders(HttpHeaders headers, ByteBuf buf) {
+printExecutingMethod();
+super.encodeHeaders(headers, buf);
+ }
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise
+ promise) throws Exception {
+LOG.debug("Writing to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+super.write(ctx, msg, promise);
+ }
+
+ private void logStacktraceIfRequired() {
+if (logStacktraceOfEncodingMethods) {
+ LOG.debug("Stacktrace: ", new Throwable());
+}
+ }
+
+ private void printExecutingMethod() {
+String methodName = getExecutingMethodName();
+LOG.debug("Executing method: {}", methodName);
+ }
+
+ private String getExecutingMethodName() {
+try {
+ StackTraceElement[] stackTrace = Thread.currentThread()
+ .getStackTrace();
+ // Array items (indices):
+ // 0: java.lang.Thread.getStackTrace(...)
+ // 1: TestShuffleHandler$LoggingHttpResponseEncoder
+ // .getExecutingMethodName(...)
+ String methodName = stackTrace[2].getMethodName();
+ //If this method was called from printExecutingMethod,
+ // we have yet another stack frame
+ if (methodName.endsWith("printExecutingMethod")) {
Review comment:
Makes sense, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17840: Labels: pull-request-available (was: ) > Backport HADOOP-17837 to branch-3.2 > --- > > Key: HADOOP-17840 > URL: https://issues.apache.org/jira/browse/HADOOP-17840 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-17840?focusedWorklogId=635078&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635078 ] ASF GitHub Bot logged work on HADOOP-17840: --- Author: ASF GitHub Bot Created on: 06/Aug/21 11:36 Start Date: 06/Aug/21 11:36 Worklog Time Spent: 10m Work Description: bbeaudreault opened a new pull request #3275: URL: https://github.com/apache/hadoop/pull/3275 https://issues.apache.org/jira/browse/HADOOP-17840 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635078) Remaining Estimate: 0h Time Spent: 10m > Backport HADOOP-17837 to branch-3.2 > --- > > Key: HADOOP-17840 > URL: https://issues.apache.org/jira/browse/HADOOP-17840 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635079
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:36
Start Date: 06/Aug/21 11:36
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684165971
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/LoggingHttpResponseEncoder.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapred;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+import io.netty.handler.codec.http.HttpHeaders;
+import io.netty.handler.codec.http.HttpResponse;
+import io.netty.handler.codec.http.HttpResponseEncoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+class LoggingHttpResponseEncoder extends HttpResponseEncoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(LoggingHttpResponseEncoder.class);
+ private final boolean logStacktraceOfEncodingMethods;
+
+ public LoggingHttpResponseEncoder(boolean logStacktraceOfEncodingMethods) {
+this.logStacktraceOfEncodingMethods = logStacktraceOfEncodingMethods;
+ }
+
+ @Override
+ public boolean acceptOutboundMessage(Object msg) throws Exception {
+printExecutingMethod();
+LOG.info("OUTBOUND MESSAGE: " + msg);
+return super.acceptOutboundMessage(msg);
+ }
+
+ @Override
+ protected void encodeInitialLine(ByteBuf buf, HttpResponse response) throws
Exception {
+LOG.debug("Executing method: {}, response: {}",
+getExecutingMethodName(), response);
+logStacktraceIfRequired();
+super.encodeInitialLine(buf, response);
+ }
+
+ @Override
+ protected void encode(ChannelHandlerContext ctx, Object msg,
+ List out) throws Exception {
+LOG.debug("Encoding to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+logStacktraceIfRequired();
+super.encode(ctx, msg, out);
+ }
+
+ @Override
+ protected void encodeHeaders(HttpHeaders headers, ByteBuf buf) {
+printExecutingMethod();
+super.encodeHeaders(headers, buf);
+ }
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise
+ promise) throws Exception {
+LOG.debug("Writing to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+super.write(ctx, msg, promise);
+ }
+
+ private void logStacktraceIfRequired() {
+if (logStacktraceOfEncodingMethods) {
+ LOG.debug("Stacktrace: ", new Throwable());
+}
+ }
+
+ private void printExecutingMethod() {
+String methodName = getExecutingMethodName();
+LOG.debug("Executing method: {}", methodName);
+ }
+
+ private String getExecutingMethodName() {
+try {
+ StackTraceElement[] stackTrace = Thread.currentThread()
+ .getStackTrace();
+ // Array items (indices):
+ // 0: java.lang.Thread.getStackTrace(...)
+ // 1: TestShuffleHandler$LoggingHttpResponseEncoder
+ // .getExecutingMethodName(...)
+ String methodName = stackTrace[2].getMethodName();
+ //If this method was called from printExecutingMethod,
+ // we have yet another stack frame
+ if (methodName.endsWith("printExecutingMethod")) {
Review comment:
Makes sense, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635079)
Time Spent: 1h (was: 50m)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
>
[jira] [Commented] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394717#comment-17394717 ] Bryan Beaudreault commented on HADOOP-17837: Thank you very much [~weichiu]. If you don't mind, I submitted HADOOP-17840 and [https://github.com/apache/hadoop/pull/3275] to backport to branch-3.2 > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 50m > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684168185
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
##
@@ -1507,4 +1507,13 @@ public static void skipIfKmsKeyIdIsNotSet(Configuration
configuration) {
}
}
+ /**
+ * Skip if a test doesn't use CSE.
+ */
+ public static void skipIfCSEIsNotEnabled(Configuration configuration) {
+String encryption =
configuration.get(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "");
+if (!encryption.equals(S3AEncryptionMethods.CSE_KMS.getMethod())) {
Review comment:
Sorry, I don't understand. Why would I change this method to skip if CSE
is enabled? And the places where it's called (even though I added it you have
the better context, so trying to understand it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635080
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:40
Start Date: 06/Aug/21 11:40
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684168185
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
##
@@ -1507,4 +1507,13 @@ public static void skipIfKmsKeyIdIsNotSet(Configuration
configuration) {
}
}
+ /**
+ * Skip if a test doesn't use CSE.
+ */
+ public static void skipIfCSEIsNotEnabled(Configuration configuration) {
+String encryption =
configuration.get(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "");
+if (!encryption.equals(S3AEncryptionMethods.CSE_KMS.getMethod())) {
Review comment:
Sorry, I don't understand. Why would I change this method to skip if CSE
is enabled? And the places where it's called (even though I added it you have
the better context, so trying to understand it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635080)
Time Spent: 4h (was: 3h 50m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684169065
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/LoggingHttpResponseEncoder.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapred;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+import io.netty.handler.codec.http.HttpHeaders;
+import io.netty.handler.codec.http.HttpResponse;
+import io.netty.handler.codec.http.HttpResponseEncoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+class LoggingHttpResponseEncoder extends HttpResponseEncoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(LoggingHttpResponseEncoder.class);
+ private final boolean logStacktraceOfEncodingMethods;
+
+ public LoggingHttpResponseEncoder(boolean logStacktraceOfEncodingMethods) {
+this.logStacktraceOfEncodingMethods = logStacktraceOfEncodingMethods;
+ }
+
+ @Override
+ public boolean acceptOutboundMessage(Object msg) throws Exception {
+printExecutingMethod();
+LOG.info("OUTBOUND MESSAGE: " + msg);
+return super.acceptOutboundMessage(msg);
+ }
+
+ @Override
+ protected void encodeInitialLine(ByteBuf buf, HttpResponse response) throws
Exception {
+LOG.debug("Executing method: {}, response: {}",
+getExecutingMethodName(), response);
+logStacktraceIfRequired();
+super.encodeInitialLine(buf, response);
+ }
+
+ @Override
+ protected void encode(ChannelHandlerContext ctx, Object msg,
+ List out) throws Exception {
+LOG.debug("Encoding to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+logStacktraceIfRequired();
+super.encode(ctx, msg, out);
+ }
+
+ @Override
+ protected void encodeHeaders(HttpHeaders headers, ByteBuf buf) {
+printExecutingMethod();
+super.encodeHeaders(headers, buf);
+ }
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise
+ promise) throws Exception {
+LOG.debug("Writing to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+super.write(ctx, msg, promise);
+ }
+
+ private void logStacktraceIfRequired() {
+if (logStacktraceOfEncodingMethods) {
+ LOG.debug("Stacktrace: ", new Throwable());
+}
+ }
+
+ private void printExecutingMethod() {
+String methodName = getExecutingMethodName();
+LOG.debug("Executing method: {}", methodName);
+ }
+
+ private String getExecutingMethodName() {
+try {
+ StackTraceElement[] stackTrace = Thread.currentThread()
+ .getStackTrace();
+ // Array items (indices):
+ // 0: java.lang.Thread.getStackTrace(...)
+ // 1: TestShuffleHandler$LoggingHttpResponseEncoder
+ // .getExecutingMethodName(...)
+ String methodName = stackTrace[2].getMethodName();
+ //If this method was called from printExecutingMethod,
+ // we have yet another stack frame
+ if (methodName.endsWith("printExecutingMethod")) {
+methodName = stackTrace[3].getMethodName();
+ }
+ String className = this.getClass().getSimpleName();
+ return className + "#" + methodName;
+} catch (Throwable t) {
+ LOG.error("Error while getting execution method name", t);
+ return null;
Review comment:
This makes sense as well, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635082&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635082
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:42
Start Date: 06/Aug/21 11:42
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684169065
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/LoggingHttpResponseEncoder.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapred;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+import io.netty.handler.codec.http.HttpHeaders;
+import io.netty.handler.codec.http.HttpResponse;
+import io.netty.handler.codec.http.HttpResponseEncoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+class LoggingHttpResponseEncoder extends HttpResponseEncoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(LoggingHttpResponseEncoder.class);
+ private final boolean logStacktraceOfEncodingMethods;
+
+ public LoggingHttpResponseEncoder(boolean logStacktraceOfEncodingMethods) {
+this.logStacktraceOfEncodingMethods = logStacktraceOfEncodingMethods;
+ }
+
+ @Override
+ public boolean acceptOutboundMessage(Object msg) throws Exception {
+printExecutingMethod();
+LOG.info("OUTBOUND MESSAGE: " + msg);
+return super.acceptOutboundMessage(msg);
+ }
+
+ @Override
+ protected void encodeInitialLine(ByteBuf buf, HttpResponse response) throws
Exception {
+LOG.debug("Executing method: {}, response: {}",
+getExecutingMethodName(), response);
+logStacktraceIfRequired();
+super.encodeInitialLine(buf, response);
+ }
+
+ @Override
+ protected void encode(ChannelHandlerContext ctx, Object msg,
+ List out) throws Exception {
+LOG.debug("Encoding to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+logStacktraceIfRequired();
+super.encode(ctx, msg, out);
+ }
+
+ @Override
+ protected void encodeHeaders(HttpHeaders headers, ByteBuf buf) {
+printExecutingMethod();
+super.encodeHeaders(headers, buf);
+ }
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise
+ promise) throws Exception {
+LOG.debug("Writing to channel {}: {}", ctx.channel(), msg);
+printExecutingMethod();
+super.write(ctx, msg, promise);
+ }
+
+ private void logStacktraceIfRequired() {
+if (logStacktraceOfEncodingMethods) {
+ LOG.debug("Stacktrace: ", new Throwable());
+}
+ }
+
+ private void printExecutingMethod() {
+String methodName = getExecutingMethodName();
+LOG.debug("Executing method: {}", methodName);
+ }
+
+ private String getExecutingMethodName() {
+try {
+ StackTraceElement[] stackTrace = Thread.currentThread()
+ .getStackTrace();
+ // Array items (indices):
+ // 0: java.lang.Thread.getStackTrace(...)
+ // 1: TestShuffleHandler$LoggingHttpResponseEncoder
+ // .getExecutingMethodName(...)
+ String methodName = stackTrace[2].getMethodName();
+ //If this method was called from printExecutingMethod,
+ // we have yet another stack frame
+ if (methodName.endsWith("printExecutingMethod")) {
+methodName = stackTrace[3].getMethodName();
+ }
+ String className = this.getClass().getSimpleName();
+ return className + "#" + methodName;
+} catch (Throwable t) {
+ LOG.error("Error while getting execution method name", t);
+ return null;
Review comment:
This makes sense as well, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
F
[GitHub] [hadoop] bogthe commented on pull request #3260: HADOOP-17198 Support S3 AccessPoint
bogthe commented on pull request #3260: URL: https://github.com/apache/hadoop/pull/3260#issuecomment-894202912 @mehakmeet APs are not supposed to work with S3 Guard. I'll update the documentation to point that out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[ https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635083&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635083 ] ASF GitHub Bot logged work on HADOOP-17198: --- Author: ASF GitHub Bot Created on: 06/Aug/21 11:43 Start Date: 06/Aug/21 11:43 Worklog Time Spent: 10m Work Description: bogthe commented on pull request #3260: URL: https://github.com/apache/hadoop/pull/3260#issuecomment-894202912 @mehakmeet APs are not supposed to work with S3 Guard. I'll update the documentation to point that out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635083) Time Spent: 4h 10m (was: 4h) > Support S3 Access Points > > > Key: HADOOP-17198 > URL: https://issues.apache.org/jira/browse/HADOOP-17198 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bogdan Stolojan >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > Improve VPC integration by supporting access points for buckets > https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html > Not sure how to do this *at all*; -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault commented on a change in pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
bbeaudreault commented on a change in pull request #3271:
URL: https://github.com/apache/hadoop/pull/3271#discussion_r684170071
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
##
@@ -994,4 +983,93 @@ public String confAsString() {
+ domainSocketDisableIntervalSeconds;
}
}
+
+ /**
+ * Handles calculating the wait time when BlockMissingException is caught.
+ */
+ public static class FetchBlockLocationsRetryer {
+private final int maxBlockAcquireFailures;
+private final int timeWindowBase;
+private final int timeWindowMultiplier;
+private final int timeWindowMax;
+private final boolean enableRandom;
+
+public FetchBlockLocationsRetryer(Configuration conf) {
+ this(conf, true);
+}
+
+/**
+ * It helps for testing to be able to disable the random factor. It should
remain
+ * enabled for non-test use
+ */
+@VisibleForTesting
+FetchBlockLocationsRetryer(Configuration conf, boolean enableRandom) {
+ maxBlockAcquireFailures = conf.getInt(
+ DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_KEY,
+ DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT);
+ timeWindowBase = conf.getInt(
+ Retry.WINDOW_BASE_KEY,
+ Retry.WINDOW_BASE_DEFAULT);
+ timeWindowMultiplier = conf.getInt(
+ Retry.WINDOW_MULTIPLIER_KEY,
+ Retry.WINDOW_MULTIPLIER_DEFAULT);
+ timeWindowMax = conf.getInt(
+ Retry.WINDOW_MAXIMUM_KEY,
+ Retry.WINDOW_MAXIMUM_DEFAULT
+ );
+ this.enableRandom = enableRandom;
+}
+
+/**
+ * For tests, exposes the maximum allowed failures
+ */
+@VisibleForTesting
+public int getMaxBlockAcquireFailures() {
+ return maxBlockAcquireFailures;
+}
+
+/**
+ * Returns whether the passed number of failures is greater or equal to
the maximum
+ * allowed failures.
+ */
+public boolean isMaxFailuresExceeded(int numFailures) {
+ return numFailures >= maxBlockAcquireFailures;
+}
+
+/**
+ * The wait time is calculated using a grace period, a time window, and a
random factor
+ * applied to that time window. With each subsequent failure, the grace
period expands
+ * to the maximum value of the previous time window, and the time window
upper limit expands
+ * by a constant exponential multiplier. The first retry has a grace
period of 0ms.
+ *
+ * With default settings, the first failure will result in a wait time of
a random number
+ * between 0 and 3000ms. The second failure will have a grace period of
3000ms, and an
+ * additional wait time of a random number between 0 and 6000ms.
Subsequent failures will
+ * expand to 6000ms grace period and 0 - 9000ms, then 9000ms grace and 0 -
12000ms, etc.
+ *
+ * This behavior can be made more and less aggressive by configuring the
base value (default 3000ms)
+ * and constant exponential multiplier (default 1). For example, a base of
10 and multiplier 5 could
+ * result in one very fast retry that quickly backs off in case of
multiple failures. This may be useful
+ * for low latency applications. One downside with high multipliers is how
quickly the backoff can get
+ * to very high numbers. One can further customize this by setting a
maximum window size to cap
+ */
+public double getWaitTime(int numFailures) {
+ double gracePeriod = backoff(numFailures);
+ double waitTimeWithRandomFactor = backoff(numFailures + 1) *
getRandomFactor();
+
+ return gracePeriod + waitTimeWithRandomFactor;
+}
+
+private double backoff(int failures) {
+ return Math.min(timeWindowBase * Math.pow(timeWindowMultiplier,
failures) * failures, timeWindowMax);
Review comment:
If i were adding this feature as brand new, I probably wouldn't include
the `* failures` here -- the base and exponential are good enough IMO. But I
needed this to maintain 100% parity with the existing backoff strategy
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
##
@@ -994,4 +983,93 @@ public String confAsString() {
+ domainSocketDisableIntervalSeconds;
}
}
+
+ /**
+ * Handles calculating the wait time when BlockMissingException is caught.
+ */
+ public static class FetchBlockLocationsRetryer {
+private final int maxBlockAcquireFailures;
+private final int timeWindowBase;
+private final int timeWindowMultiplier;
+private final int timeWindowMax;
+private final boolean enableRandom;
+
+public FetchBlockLocationsRetryer(Configuration conf) {
+ this(conf, true);
+}
+
+/**
+ * It helps for testing to be able to disable the random factor. It should
remain
+ * enabled for non-test use
+ */
+@VisibleForTesting
+FetchBlockLocationsRetryer(Configuration c
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684170128
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -290,6 +302,36 @@ public void operationComplete(ChannelFuture future) throws
Exception {
shuffleConnections.decr();
}
}
+
+ static class NettyChannelHelper {
+static ChannelFuture writeToChannel(Channel ch, Object obj) {
+ LOG.debug("Writing {} to channel: {}", obj.getClass().getSimpleName(),
ch.id());
+ return ch.writeAndFlush(obj);
+}
+
+static void writeToChannelAndClose(Channel ch, Object obj) {
Review comment:
Makes sense, also added to writeToChannelAndClose so all of the method
return types are uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635085
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:44
Start Date: 06/Aug/21 11:44
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684170128
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -290,6 +302,36 @@ public void operationComplete(ChannelFuture future) throws
Exception {
shuffleConnections.decr();
}
}
+
+ static class NettyChannelHelper {
+static ChannelFuture writeToChannel(Channel ch, Object obj) {
+ LOG.debug("Writing {} to channel: {}", obj.getClass().getSimpleName(),
ch.id());
+ return ch.writeAndFlush(obj);
+}
+
+static void writeToChannelAndClose(Channel ch, Object obj) {
Review comment:
Makes sense, also added to writeToChannelAndClose so all of the method
return types are uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635085)
Time Spent: 1h 20m (was: 1h 10m)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
> Key: HADOOP-15327
> URL: https://issues.apache.org/jira/browse/HADOOP-15327
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch,
> HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch,
> HADOOP-15327.005.patch,
> getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log,
> testfailure-testMapFileAccess-emptyresponse.zip,
> testfailure-testReduceFromPartialMem.zip
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> This way, we can remove the dependencies on the netty3 (jboss.netty)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684173504
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
##
@@ -1507,4 +1507,13 @@ public static void skipIfKmsKeyIdIsNotSet(Configuration
configuration) {
}
}
+ /**
+ * Skip if a test doesn't use CSE.
+ */
+ public static void skipIfCSEIsNotEnabled(Configuration configuration) {
+String encryption =
configuration.get(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "");
+if (!encryption.equals(S3AEncryptionMethods.CSE_KMS.getMethod())) {
Review comment:
The tests that require this are SSE tests which shouldn't skip if
CSE-KMS is the encryption method used. It should've been skipped if CSE-KMS is
enabled, to tell the user that SSE-KMS is not the method used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635086
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:50
Start Date: 06/Aug/21 11:50
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684173504
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
##
@@ -1507,4 +1507,13 @@ public static void skipIfKmsKeyIdIsNotSet(Configuration
configuration) {
}
}
+ /**
+ * Skip if a test doesn't use CSE.
+ */
+ public static void skipIfCSEIsNotEnabled(Configuration configuration) {
+String encryption =
configuration.get(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "");
+if (!encryption.equals(S3AEncryptionMethods.CSE_KMS.getMethod())) {
Review comment:
The tests that require this are SSE tests which shouldn't skip if
CSE-KMS is the encryption method used. It should've been skipped if CSE-KMS is
enabled, to tell the user that SSE-KMS is not the method used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635086)
Time Spent: 4h 20m (was: 4h 10m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684173898
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -828,27 +893,44 @@ public void destroy() {
}
}
-@Override
-public ChannelPipeline getPipeline() throws Exception {
- ChannelPipeline pipeline = Channels.pipeline();
+@Override protected void initChannel(SocketChannel ch) throws Exception {
+ ChannelPipeline pipeline = ch.pipeline();
if (sslFactory != null) {
pipeline.addLast("ssl", new SslHandler(sslFactory.createSSLEngine()));
}
pipeline.addLast("decoder", new HttpRequestDecoder());
- pipeline.addLast("aggregator", new HttpChunkAggregator(1 << 16));
- pipeline.addLast("encoder", new HttpResponseEncoder());
+ pipeline.addLast("aggregator", new HttpObjectAggregator(1 << 16));
Review comment:
Thanks, this is also valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635087
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:51
Start Date: 06/Aug/21 11:51
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684173898
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -828,27 +893,44 @@ public void destroy() {
}
}
-@Override
-public ChannelPipeline getPipeline() throws Exception {
- ChannelPipeline pipeline = Channels.pipeline();
+@Override protected void initChannel(SocketChannel ch) throws Exception {
+ ChannelPipeline pipeline = ch.pipeline();
if (sslFactory != null) {
pipeline.addLast("ssl", new SslHandler(sslFactory.createSSLEngine()));
}
pipeline.addLast("decoder", new HttpRequestDecoder());
- pipeline.addLast("aggregator", new HttpChunkAggregator(1 << 16));
- pipeline.addLast("encoder", new HttpResponseEncoder());
+ pipeline.addLast("aggregator", new HttpObjectAggregator(1 << 16));
Review comment:
Thanks, this is also valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635087)
Time Spent: 1.5h (was: 1h 20m)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
> Key: HADOOP-15327
> URL: https://issues.apache.org/jira/browse/HADOOP-15327
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch,
> HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch,
> HADOOP-15327.005.patch,
> getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log,
> testfailure-testMapFileAccess-emptyresponse.zip,
> testfailure-testReduceFromPartialMem.zip
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This way, we can remove the dependencies on the netty3 (jboss.netty)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on pull request #3260: URL: https://github.com/apache/hadoop/pull/3260#issuecomment-894207049 >APs are not supposed to work with S3 Guard. Shouldn't we just throw an exception during initialization if AccessPoint is set and S3Guard is enabled to tell the users that AccessPoints are incompatible with S3Guard? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684174702
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -785,37 +833,54 @@ private void removeJobShuffleInfo(JobID jobId) throws
IOException {
}
}
- static class TimeoutHandler extends IdleStateAwareChannelHandler {
+ @VisibleForTesting
+ public void setUseOutboundExceptionHandler(boolean useHandler) {
+this.useOutboundExceptionHandler = useHandler;
+ }
+ static class TimeoutHandler extends IdleStateHandler {
+private final int connectionKeepAliveTimeOut;
private boolean enabledTimeout;
+public TimeoutHandler(int connectionKeepAliveTimeOut) {
+ //disable reader timeout
+ //set writer timeout to configured timeout value
+ //disable all idle timeout
+ super(0, connectionKeepAliveTimeOut, 0);
Review comment:
Good catch, modified to:
super(0, connectionKeepAliveTimeOut, 0, TimeUnit.SECONDS);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[ https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635089&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635089 ] ASF GitHub Bot logged work on HADOOP-17198: --- Author: ASF GitHub Bot Created on: 06/Aug/21 11:52 Start Date: 06/Aug/21 11:52 Worklog Time Spent: 10m Work Description: mehakmeet commented on pull request #3260: URL: https://github.com/apache/hadoop/pull/3260#issuecomment-894207049 >APs are not supposed to work with S3 Guard. Shouldn't we just throw an exception during initialization if AccessPoint is set and S3Guard is enabled to tell the users that AccessPoints are incompatible with S3Guard? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 635089) Time Spent: 4.5h (was: 4h 20m) > Support S3 Access Points > > > Key: HADOOP-17198 > URL: https://issues.apache.org/jira/browse/HADOOP-17198 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bogdan Stolojan >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Improve VPC integration by supporting access points for buckets > https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html > Not sure how to do this *at all*; -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635090
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:52
Start Date: 06/Aug/21 11:52
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684174702
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -785,37 +833,54 @@ private void removeJobShuffleInfo(JobID jobId) throws
IOException {
}
}
- static class TimeoutHandler extends IdleStateAwareChannelHandler {
+ @VisibleForTesting
+ public void setUseOutboundExceptionHandler(boolean useHandler) {
+this.useOutboundExceptionHandler = useHandler;
+ }
+ static class TimeoutHandler extends IdleStateHandler {
+private final int connectionKeepAliveTimeOut;
private boolean enabledTimeout;
+public TimeoutHandler(int connectionKeepAliveTimeOut) {
+ //disable reader timeout
+ //set writer timeout to configured timeout value
+ //disable all idle timeout
+ super(0, connectionKeepAliveTimeOut, 0);
Review comment:
Good catch, modified to:
super(0, connectionKeepAliveTimeOut, 0, TimeUnit.SECONDS);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635090)
Time Spent: 1h 40m (was: 1.5h)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
> Key: HADOOP-15327
> URL: https://issues.apache.org/jira/browse/HADOOP-15327
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch,
> HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch,
> HADOOP-15327.005.patch,
> getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log,
> testfailure-testMapFileAccess-emptyresponse.zip,
> testfailure-testReduceFromPartialMem.zip
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This way, we can remove the dependencies on the netty3 (jboss.netty)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684177660
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -828,27 +893,44 @@ public void destroy() {
}
}
-@Override
-public ChannelPipeline getPipeline() throws Exception {
- ChannelPipeline pipeline = Channels.pipeline();
+@Override protected void initChannel(SocketChannel ch) throws Exception {
+ ChannelPipeline pipeline = ch.pipeline();
if (sslFactory != null) {
pipeline.addLast("ssl", new SslHandler(sslFactory.createSSLEngine()));
}
pipeline.addLast("decoder", new HttpRequestDecoder());
- pipeline.addLast("aggregator", new HttpChunkAggregator(1 << 16));
- pipeline.addLast("encoder", new HttpResponseEncoder());
+ pipeline.addLast("aggregator", new HttpObjectAggregator(1 << 16));
+ pipeline.addLast(ENCODER_HANDLER_NAME, new HttpResponseEncoder());
pipeline.addLast("chunking", new ChunkedWriteHandler());
pipeline.addLast("shuffle", SHUFFLE);
- pipeline.addLast("idle", idleStateHandler);
- pipeline.addLast(TIMEOUT_HANDLER, new TimeoutHandler());
- return pipeline;
+ addOutboundHandlersIfRequired(pipeline);
+ pipeline.addLast(TIMEOUT_HANDLER, new
TimeoutHandler(connectionKeepAliveTimeOut));
// TODO factor security manager into pipeline
// TODO factor out encode/decode to permit binary shuffle
// TODO factor out decode of index to permit alt. models
}
+
+private void addOutboundHandlersIfRequired(ChannelPipeline pipeline) {
+ if (useOutboundExceptionHandler) {
+
//https://stackoverflow.com/questions/50612403/catch-all-exception-handling-for-outbound-channelhandler
+pipeline.addLast("outboundExceptionHandler", new
ChannelOutboundHandlerAdapter() {
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg,
ChannelPromise promise) throws Exception {
+
promise.addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
+super.write(ctx, msg, promise);
+ }
+});
+ }
+ if (useOutboundLogger) {
+//Replace HttpResponseEncoder with LoggingHttpResponseEncoder
+//Need to use the same name as before, otherwise we would have 2
encoders
+pipeline.replace(ENCODER_HANDLER_NAME, ENCODER_HANDLER_NAME, new
LoggingHttpResponseEncoder(false));
Review comment:
Makes sense, inlined the method and got rid of the pipeline.replace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635091
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 11:57
Start Date: 06/Aug/21 11:57
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684177660
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -828,27 +893,44 @@ public void destroy() {
}
}
-@Override
-public ChannelPipeline getPipeline() throws Exception {
- ChannelPipeline pipeline = Channels.pipeline();
+@Override protected void initChannel(SocketChannel ch) throws Exception {
+ ChannelPipeline pipeline = ch.pipeline();
if (sslFactory != null) {
pipeline.addLast("ssl", new SslHandler(sslFactory.createSSLEngine()));
}
pipeline.addLast("decoder", new HttpRequestDecoder());
- pipeline.addLast("aggregator", new HttpChunkAggregator(1 << 16));
- pipeline.addLast("encoder", new HttpResponseEncoder());
+ pipeline.addLast("aggregator", new HttpObjectAggregator(1 << 16));
+ pipeline.addLast(ENCODER_HANDLER_NAME, new HttpResponseEncoder());
pipeline.addLast("chunking", new ChunkedWriteHandler());
pipeline.addLast("shuffle", SHUFFLE);
- pipeline.addLast("idle", idleStateHandler);
- pipeline.addLast(TIMEOUT_HANDLER, new TimeoutHandler());
- return pipeline;
+ addOutboundHandlersIfRequired(pipeline);
+ pipeline.addLast(TIMEOUT_HANDLER, new
TimeoutHandler(connectionKeepAliveTimeOut));
// TODO factor security manager into pipeline
// TODO factor out encode/decode to permit binary shuffle
// TODO factor out decode of index to permit alt. models
}
+
+private void addOutboundHandlersIfRequired(ChannelPipeline pipeline) {
+ if (useOutboundExceptionHandler) {
+
//https://stackoverflow.com/questions/50612403/catch-all-exception-handling-for-outbound-channelhandler
+pipeline.addLast("outboundExceptionHandler", new
ChannelOutboundHandlerAdapter() {
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg,
ChannelPromise promise) throws Exception {
+
promise.addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
+super.write(ctx, msg, promise);
+ }
+});
+ }
+ if (useOutboundLogger) {
+//Replace HttpResponseEncoder with LoggingHttpResponseEncoder
+//Need to use the same name as before, otherwise we would have 2
encoders
+pipeline.replace(ENCODER_HANDLER_NAME, ENCODER_HANDLER_NAME, new
LoggingHttpResponseEncoder(false));
Review comment:
Makes sense, inlined the method and got rid of the pipeline.replace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635091)
Time Spent: 1h 50m (was: 1h 40m)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
> Key: HADOOP-15327
> URL: https://issues.apache.org/jira/browse/HADOOP-15327
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch,
> HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch,
> HADOOP-15327.005.patch,
> getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log,
> testfailure-testMapFileAccess-emptyresponse.zip,
> testfailure-testReduceFromPartialMem.zip
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This way, we can remove the dependencies on the netty3 (jboss.netty)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684183200
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, if the endpoint isn't set/empty, then we set it to the Accesspoint
endpoint, but if the endpoint is set, we set it to empty?
Maybe set endpoint to Accesspoint endpoint if Accesspoint is set in both
cases?
Maybe I got it totally wrong, please, correct me, if this isn't what we're
doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635096
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 12:07
Start Date: 06/Aug/21 12:07
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684183200
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, if the endpoint isn't set/empty, then we set it to the Accesspoint
endpoint, but if the endpoint is set, we set it to empty?
Maybe set endpoint to Accesspoint endpoint if Accesspoint is set in both
cases?
Maybe I got it totally wrong, please, correct me, if this isn't what we're
doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635096)
Time Spent: 4h 40m (was: 4.5h)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635099
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 12:13
Start Date: 06/Aug/21 12:13
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635099)
Time Spent: 4h 50m (was: 4h 40m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well? or we would have to change something like
`s3.ap-south-1.amazonaws.com` to `s3-accesspoint.ap-south-1.amazonaws.com`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635100
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 12:16
Start Date: 06/Aug/21 12:16
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well? or we would have to change something like
`s3.ap-south-1.amazonaws.com` to `s3-accesspoint.ap-south-1.amazonaws.com`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635100)
Time Spent: 5h (was: 4h 50m)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on pull request #3209: HDFS-16129. Fixing the signature secret file misusage in HttpFS.
brumi1024 commented on pull request #3209: URL: https://github.com/apache/hadoop/pull/3209#issuecomment-894222863 @tomicooler thanks for the updates, it looks good to me, +1 (non-binding). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well? or we would have to change something like
`s3.ap-south-1.amazonaws.com` to `s3-accesspoint.ap-south-1.amazonaws.com`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394732#comment-17394732 ] Brahma Reddy Battula commented on HADOOP-17840: --- [~bbeaudreault] thanks reporting, I thought of cherry-pick this but looks [~ste...@apache.org] has some comments previous PR.. Can you please address those..? > Backport HADOOP-17837 to branch-3.2 > --- > > Key: HADOOP-17840 > URL: https://issues.apache.org/jira/browse/HADOOP-17840 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=635105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635105
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 12:22
Start Date: 06/Aug/21 12:22
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r684186895
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -822,10 +854,21 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
+// If there's no endpoint set, then use the default for bucket or
AccessPoint. Overriding is
+// useful when using FIPS or DualStack S3 endpoints.
+String endpoint = conf.getTrimmed(ENDPOINT, "");
+if (endpoint.isEmpty()) {
Review comment:
So, basically, if we have an Accesspoint set, we can't have an endpoint
set as well? or we would have to change something like
`s3.ap-south-1.amazonaws.com` to `s3-accesspoint.ap-south-1.amazonaws.com`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635105)
Time Spent: 5h 10m (was: 5h)
> Support S3 Access Points
>
>
> Key: HADOOP-17198
> URL: https://issues.apache.org/jira/browse/HADOOP-17198
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Bogdan Stolojan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Improve VPC integration by supporting access points for buckets
> https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html
> Not sure how to do this *at all*;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684193981
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -904,11 +986,11 @@ public void setPort(int port) {
}
@Override
-public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent evt)
+public void channelActive(ChannelHandlerContext ctx)
throws Exception {
- super.channelOpen(ctx, evt);
-
- if ((maxShuffleConnections > 0) && (accepted.size() >=
maxShuffleConnections)) {
+ LOG.debug("channelActive");
Review comment:
Very much making sense. Fixed a lot of logging messages in the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394734#comment-17394734 ] Bryan Beaudreault commented on HADOOP-17840: Thanks for the comment [~brahmareddy]. I am not sure what the best practice is in Hadoop. Do you want me to address those comments in this backport PR and leave it alone in trunk? Or submit a new PR for trunk? One of Steve's comments does not apply – only the the change to use a test util. I would be happy to make the test util change if it's deemed worth it. > Backport HADOOP-17837 to branch-3.2 > --- > > Key: HADOOP-17840 > URL: https://issues.apache.org/jira/browse/HADOOP-17840 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=635106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-635106
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 12:25
Start Date: 06/Aug/21 12:25
Worklog Time Spent: 10m
Work Description: szilard-nemeth commented on a change in pull request
#3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684193981
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -904,11 +986,11 @@ public void setPort(int port) {
}
@Override
-public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent evt)
+public void channelActive(ChannelHandlerContext ctx)
throws Exception {
- super.channelOpen(ctx, evt);
-
- if ((maxShuffleConnections > 0) && (accepted.size() >=
maxShuffleConnections)) {
+ LOG.debug("channelActive");
Review comment:
Very much making sense. Fixed a lot of logging messages in the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 635106)
Time Spent: 2h (was: 1h 50m)
> Upgrade MR ShuffleHandler to use Netty4
> ---
>
> Key: HADOOP-15327
> URL: https://issues.apache.org/jira/browse/HADOOP-15327
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch,
> HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch,
> HADOOP-15327.005.patch,
> getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log,
> testfailure-testMapFileAccess-emptyresponse.zip,
> testfailure-testReduceFromPartialMem.zip
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This way, we can remove the dependencies on the netty3 (jboss.netty)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
szilard-nemeth commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r684194897
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
##
@@ -920,31 +1002,50 @@ public void channelOpen(ChannelHandlerContext ctx,
ChannelStateEvent evt)
// fetch failure.
headers.put(RETRY_AFTER_HEADER, String.valueOf(FETCH_RETRY_DELAY));
sendError(ctx, "", TOO_MANY_REQ_STATUS, headers);
-return;
+ } else {
+super.channelActive(ctx);
+accepted.add(ctx.channel());
+LOG.debug("Added channel: {}. Accepted number of connections={}",
+ctx.channel(), acceptedConnections.get());
}
- accepted.add(evt.getChannel());
}
@Override
-public void messageReceived(ChannelHandlerContext ctx, MessageEvent evt)
+public void channelInactive(ChannelHandlerContext ctx) throws Exception {
+ LOG.trace("Executing channelInactive");
+ super.channelInactive(ctx);
+ acceptedConnections.decrementAndGet();
+ LOG.debug("New value of Accepted number of connections={}",
+ acceptedConnections.get());
Review comment:
Thanks for catching this. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
