[GitHub] [hadoop] hadoop-yetus commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config
hadoop-yetus commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config URL: https://github.com/apache/hadoop/pull/1585#issuecomment-538280533 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 38 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | yamllint | 0 | yamllint was not available. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 33 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 36 | hadoop-ozone in trunk failed. | | -1 | compile | 24 | hadoop-hdds in trunk failed. | | -1 | compile | 18 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 794 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 27 | hadoop-hdds in trunk failed. | | -1 | javadoc | 25 | hadoop-ozone in trunk failed. | | -0 | patch | 893 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 40 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 42 | hadoop-ozone in the patch failed. | | -1 | compile | 29 | hadoop-hdds in the patch failed. | | -1 | compile | 24 | hadoop-ozone in the patch failed. | | -1 | javac | 29 | hadoop-hdds in the patch failed. | | -1 | javac | 24 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 716 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 26 | hadoop-hdds in the patch failed. | | -1 | javadoc | 24 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 33 | hadoop-hdds in the patch failed. | | -1 | unit | 30 | hadoop-ozone in the patch failed. | | +1 | asflicense | 38 | The patch does not generate ASF License warnings. | | | | 2189 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1585 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs | | uname | Linux d730fd7431e9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ec8f691 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1585/2/artifact/out/patch-java
[GitHub] [hadoop] adoroszlai opened a new pull request #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
adoroszlai opened a new pull request #1590: HDDS-2238. Container Data Scrubber
spams log in empty cluster
URL: https://github.com/apache/hadoop/pull/1590
## What changes were proposed in this pull request?
1. Add configurable interval for container data scan iterations
(`hdds.containerscrub.data.scan.interval`). A small delay will not affect
scanning if there is enough I/O work to do. However, it prevents the scanner
threads from keeping CPU busy when there are no or few healthy closed
containers to be scanned.
2. Only count iterations where at least one container was scanned.
3. Some code cleanup:
* get rid of raw Container` usage (add generic parameter ``)
* fix some javadoc (add `@param` descriptions, remove `@throws` without
description)
https://issues.apache.org/jira/browse/HDDS-2238
## How was this patch tested?
Test script:
```
cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
echo 'OZONE-SITE.XML_hdds.containerscrub.enabled=true' >> docker-config
echo 'OZONE-SITE.XML_hdds.containerscrub.data.scan.interval=1' >>
docker-config
KEEP_RUNNING=true ./test.sh
for i in 1 2; do
docker-compose exec scm ozone scmcli container close ${i}
sleep 10
done
docker-compose logs datanode | grep \
-e 'Completed an iteration of container data scrubber' \
-e 'Container .* is closed'
```
Result:
```
datanode_1 | 2019-10-04 07:01:00 INFO Container:335 - Container 1 is
closed with bcsId 0.
datanode_1 | 2019-10-04 07:01:00 INFO ContainerDataScanner:115 - Completed
an iteration of container data scrubber in 0 minutes. Number of iterations
(since the data-node restart) : 1, Number of containers scanned in this
iteration : 1, Number of unhealthy containers found in this iteration : 1
datanode_1 | 2019-10-04 07:01:10 INFO Container:335 - Container 2 is
closed with bcsId 0.
datanode_1 | 2019-10-04 07:01:10 INFO ContainerDataScanner:115 - Completed
an iteration of container data scrubber in 0 minutes. Number of iterations
(since the data-node restart) : 2, Number of containers scanned in this
iteration : 1, Number of unhealthy containers found in this iteration : 1
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-538282622 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1585: HDDS-2230. Invalid entries in ozonesecure-mr config
adoroszlai commented on a change in pull request #1585: HDDS-2230. Invalid entries in ozonesecure-mr config URL: https://github.com/apache/hadoop/pull/1585#discussion_r331335642 ## File path: hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml ## @@ -23,17 +23,23 @@ services: args: buildno: 1 hostname: kdc +networks: + - ozone Review comment: Default network does not work, since its name is `ozonesecure-mr_default`, which triggers `URISyntaxException` due to `_`. Adding an explicit name avoids the `_default` suffix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r331375792
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AuthConfigurations.java
##
@@ -0,0 +1,40 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Responsible to keep all the Azure Blob File System auth related
+ * configurations.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public final class AuthConfigurations {
+
+ public static final String DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT =
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r331376185
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
##
@@ -109,17 +112,15 @@ public static AzureADToken
getTokenUsingClientCreds(String authEndpoint,
* @return {@link AzureADToken} obtained using the creds
* @throws IOException throws IOException if there is a failure in obtaining
the token
*/
- public static AzureADToken getTokenFromMsi(String tenantGuid, String
clientId,
- boolean bypassCache) throws
IOException {
-String authEndpoint =
"http://169.254.169.254/metadata/identity/oauth2/token";;
-
+ public static AzureADToken getTokenFromMsi(final String authEndpoint,
+ final String tenantGuid, final String clientId, String authority,
+ boolean bypassCache) throws IOException {
QueryParams qp = new QueryParams();
qp.add("api-version", "2018-02-01");
qp.add("resource", RESOURCE_NAME);
-
if (tenantGuid != null && tenantGuid.length() > 0) {
- String authority = "https://login.microsoftonline.com/"; + tenantGuid;
+ authority = authority + tenantGuid;
Review comment:
Done.
Will go ahead with printing a debug level log.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rbalamohan opened a new pull request #1591: HADOOP-16629: support copyFile in s3afilesystem
rbalamohan opened a new pull request #1591: HADOOP-16629: support copyFile in s3afilesystem URL: https://github.com/apache/hadoop/pull/1591 Changes: This is subtask of HADOOP-16604 which aims to provide copy functionality for cloud native applications. Intent of this PR is to provide `copyFile(URI src, URI dst)` functionality for S3AFileSystem (HADOOP-16629). Testing was done in `region=us-west-2` on my local laptop. ``` [ERROR] Tests run: 1090, Failures: 5, Errors: 22, Skipped: 318 ``` I observed good number of tests timing out and few of them throwing NPE. e.g ``` [ERROR] testPrefixVsDirectory(org.apache.hadoop.fs.s3a.ITestAuthoritativePath) Time elapsed: 13.164 s <<< ERROR! java.lang.NullPointerException at org.apache.hadoop.fs.s3a.ITestAuthoritativePath.teardown(ITestAuthoritativePath.java:94) ``` I will check if i can run few more runs to reduce the error count. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
hadoop-yetus commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-538296550 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 16 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 867 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 969 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 35 | hadoop-hdds in trunk failed. | | -1 | findbugs | 20 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 28 | hadoop-hdds: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | -0 | checkstyle | 30 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 709 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 21 | hadoop-ozone in the patch failed. | | -1 | findbugs | 31 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 27 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2459 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1590 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f2e75c4de5e8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ec8f691 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/1/artifact/out/pa
[jira] [Created] (HADOOP-16630) CLONE - ABFS: Config to enable/disable flush operation
Sneha Vijayarajan created HADOOP-16630: -- Summary: CLONE - ABFS: Config to enable/disable flush operation Key: HADOOP-16630 URL: https://issues.apache.org/jira/browse/HADOOP-16630 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan Fix For: 3.3.0 Make flush operation enabled/disabled through configuration. This is part of performance improvements for ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16630) Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-16630: --- Summary: Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2 (was: CLONE - ABFS: Config to enable/disable flush operation) > Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to > branch-2 > > > Key: HADOOP-16630 > URL: https://issues.apache.org/jira/browse/HADOOP-16630 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Minor > Fix For: 3.3.0 > > > Make flush operation enabled/disabled through configuration. This is part of > performance improvements for ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
hadoop-yetus commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-538313910 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 50 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 827 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 927 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 24 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 24 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 58 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 764 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 31 | hadoop-hdds in the patch failed. | | -1 | findbugs | 19 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 27 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2424 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1590 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fdfdfd8aa431 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b23bdaf | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1590/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://b
[GitHub] [hadoop] steveloughran commented on issue #1587: HADOOP-16626. S3A ITestRestrictedReadAccess fails
steveloughran commented on issue #1587: HADOOP-16626. S3A ITestRestrictedReadAccess fails URL: https://github.com/apache/hadoop/pull/1587#issuecomment-538314567 * we are unsetting any bucket-specific choices of s3guard, so that even when people (me) have it enabled you explicitly * the problem I've tried to fix was the discovery that Filesyste.get() triggered the load of things like HdfsConfiguration, it's adding of hdfs-default.xml &c, which then reinstated the unset options * If any XML resource is added to any config created with default resources, all options from core-default, core-site which had been unset are reinstated* Which of course is entirely unexpected. Having a look at why things are failing for you, -Ds3guard -Dlocal had been fine for me. I had problems with this when changing the test because you need to share the same local store instance across the real fs and the restricted one -otherwise the restricted one's cache is empty, so it falls back to s3 checks. I may just change the test so that it requires s3guard + ddb set on the maven command line to run those guarded tests, and not worry about the local one at all. It's just complicating things too much and *especially given the local store cannot be used in production* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1587: HADOOP-16626. S3A ITestRestrictedReadAccess fails
steveloughran commented on issue #1587: HADOOP-16626. S3A ITestRestrictedReadAccess fails URL: https://github.com/apache/hadoop/pull/1587#issuecomment-538315035 oh, and it does work for me: -Dit.test=ITestRestrictedReadAccess -Ds3guard I am going to go to only running the guarded tests if -Ds3guard -Ddynamo is set This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
hadoop-yetus commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows URL: https://github.com/apache/hadoop/pull/1481#issuecomment-538317090 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 75 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1244 | trunk passed | | +1 | compile | 28 | trunk passed | | +1 | checkstyle | 20 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | trunk passed | | 0 | spotbugs | 50 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 48 | trunk passed | | -0 | patch | 71 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 27 | the patch passed | | +1 | compile | 23 | the patch passed | | +1 | javac | 23 | the patch passed | | +1 | checkstyle | 16 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 851 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 21 | the patch passed | | +1 | findbugs | 56 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 68 | hadoop-azure in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 3542 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1481 | | JIRA Issue | HADOOP-16587 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3e88e1847028 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b23bdaf | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/5/testReport/ | | Max. process+thread count | 308 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[
https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944338#comment-16944338
]
Hadoop QA commented on HADOOP-16587:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
50s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:orange}-0{color} | {color:orange} patch {color} | {color:orange} 1m
11s{color} | {color:orange} Used diff version of patch file. Binary files and
potentially other changes not applied. Please rebase and squash commits if
necessary. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1481 |
| JIRA Issue | HADOOP-16587 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3e88e1847028 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b23bdaf |
[GitHub] [hadoop] snvijaya opened a new pull request #1592: Hadoop 16630 : Backport of Hadoop-16548 : Disable Flush() over config
snvijaya opened a new pull request #1592: Hadoop 16630 : Backport of Hadoop-16548 : Disable Flush() over config URL: https://github.com/apache/hadoop/pull/1592 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16631) CLONE - ABFS: fileSystemExists() should not call container level apis
Sneha Vijayarajan created HADOOP-16631: -- Summary: CLONE - ABFS: fileSystemExists() should not call container level apis Key: HADOOP-16631 URL: https://issues.apache.org/jira/browse/HADOOP-16631 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan Fix For: 3.3.0 ABFS driver should not use container level api "Get Container Properties" as there is no concept of container in HDFS, and this caused some RBAC check issue. Fix: use getFileStatus() to check if the container exists. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16631) Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-16631: --- Summary: Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2 (was: CLONE - ABFS: fileSystemExists() should not call container level apis) > Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container > level apis" to Branch-2 > --- > > Key: HADOOP-16631 > URL: https://issues.apache.org/jira/browse/HADOOP-16631 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > ABFS driver should not use container level api "Get Container Properties" as > there is no concept of container in HDFS, and this caused some RBAC check > issue. > Fix: use getFileStatus() to check if the container exists. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16629) support copyFile in s3afilesystem
[
https://issues.apache.org/jira/browse/HADOOP-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944354#comment-16944354
]
Hadoop QA commented on HADOOP-16629:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
35s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
56s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
25s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 25s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 52s{color} | {color:orange} root: The patch generated 22 new + 106 unchanged
- 0 fixed = 128 total (was 106) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 32s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.TestFilterFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoo
[GitHub] [hadoop] hadoop-yetus commented on issue #1591: HADOOP-16629: support copyFile in s3afilesystem
hadoop-yetus commented on issue #1591: HADOOP-16629: support copyFile in s3afilesystem URL: https://github.com/apache/hadoop/pull/1591#issuecomment-538325634 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 51 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 69 | Maven dependency ordering for branch | | +1 | mvninstall | 1415 | trunk passed | | -1 | compile | 236 | root in trunk failed. | | +1 | checkstyle | 158 | trunk passed | | +1 | mvnsite | 115 | trunk passed | | +1 | shadedclient | 1120 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 119 | trunk passed | | 0 | spotbugs | 68 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 201 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 91 | the patch passed | | -1 | compile | 205 | root in the patch failed. | | -1 | javac | 205 | root in the patch failed. | | -0 | checkstyle | 172 | root: The patch generated 22 new + 106 unchanged - 0 fixed = 128 total (was 106) | | +1 | mvnsite | 112 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 813 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 106 | the patch passed | | +1 | findbugs | 179 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 512 | hadoop-common in the patch failed. | | +1 | unit | 76 | hadoop-aws in the patch passed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 5816 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.TestHarFileSystem | | | hadoop.fs.TestFilterFileSystem | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1591 | | JIRA Issue | HADOOP-16629 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 31f98978aff9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b23bdaf | | Default Java | 1.8.0_222 | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/branch-compile-root.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/patch-compile-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/testReport/ | | Max. process+thread count | 1514 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1591/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on issue #1578: HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo commented on issue #1578: HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1578#issuecomment-538330389 The "Switch case falls through" reported by findbugs is by design so that we won't fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on issue #1578: HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo commented on issue #1578: HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1578#issuecomment-538330912 The integration failures definitely are not related since the new code is not yet used in Ozone. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo merged pull request #1578: HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo merged pull request #1578: HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo opened a new pull request #1593: HDDS-2204. Avoid buffer coping in checksum verification.
szetszwo opened a new pull request #1593: HDDS-2204. Avoid buffer coping in checksum verification. URL: https://github.com/apache/hadoop/pull/1593 See https://issues.apache.org/jira/browse/HDDS-2204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1593: HDDS-2204. Avoid buffer coping in checksum verification.
hadoop-yetus commented on issue #1593: HDDS-2204. Avoid buffer coping in checksum verification. URL: https://github.com/apache/hadoop/pull/1593#issuecomment-538347183 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 37 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 37 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 859 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 19 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 973 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 47 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 722 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 31 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 29 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2404 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1593 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8b31ae32a557 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4cf0b36 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://bui
[GitHub] [hadoop] szetszwo opened a new pull request #1594: Revert "HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C"
szetszwo opened a new pull request #1594: Revert "HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C" URL: https://github.com/apache/hadoop/pull/1594 Reverts apache/hadoop#1578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on issue #1578: HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo commented on issue #1578: HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1578#issuecomment-538356707 > The "Switch case falls through" reported by findbugs is by design so that we won't fix it. Oops, I should have added it to findbugsExcludeFile.xml. Otherwise, the warning will keep coming up. Will revert the commit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo merged pull request #1594: Revert "HDDS-2222 Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C"
szetszwo merged pull request #1594: Revert "HDDS- Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C" URL: https://github.com/apache/hadoop/pull/1594 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config
elek commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config URL: https://github.com/apache/hadoop/pull/1585#issuecomment-538357881 BTW we use very weak containers where we have passwordless sudo, we can add additional step after config generation and before startup to make the etc directory read only... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo opened a new pull request #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo opened a new pull request #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595 See https://issues.apache.org/jira/browse/HDDS- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1542: HDDS-2140. Add robot test for GDPR feature
elek commented on issue #1542: HDDS-2140. Add robot test for GDPR feature URL: https://github.com/apache/hadoop/pull/1542#issuecomment-538364957 > Thereby, even if you get your hands on chunks, you will still read encrypted junk :) Yes, I understand. That's the question. How can we prove that we only have junk. I would like to prove that the junk is encrypted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek closed pull request #1542: HDDS-2140. Add robot test for GDPR feature
elek closed pull request #1542: HDDS-2140. Add robot test for GDPR feature URL: https://github.com/apache/hadoop/pull/1542 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek closed pull request #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
elek closed pull request #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host URL: https://github.com/apache/hadoop/pull/1551 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
elek commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host URL: https://github.com/apache/hadoop/pull/1551#issuecomment-538372245 +1 Thanks the updated @sodonnel Let's do the complex part in the follow-up jira. I am pushed it to the trunk... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
hadoop-yetus commented on issue #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595#issuecomment-538375230 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 76 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 41 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 39 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 58 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 941 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in trunk failed. | | -1 | javadoc | 17 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1030 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 31 | hadoop-hdds in trunk failed. | | -1 | findbugs | 18 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 31 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | compile | 21 | hadoop-hdds in the patch failed. | | -1 | compile | 15 | hadoop-ozone in the patch failed. | | -1 | javac | 21 | hadoop-hdds in the patch failed. | | -1 | javac | 15 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 24 | hadoop-hdds: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 796 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 28 | hadoop-hdds in the patch failed. | | -1 | findbugs | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 21 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 2488 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1595 | | Optional Tests | dupname asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c27d0ebfbb42 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / bffcd33 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/1/artifact/out/patch-compile-hadoop-hdds.txt
[jira] [Commented] (HADOOP-16466) Clean up the Assert usage in tests
[
https://issues.apache.org/jira/browse/HADOOP-16466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944465#comment-16944465
]
Hadoop QA commented on HADOOP-16466:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 36 unchanged - 3 fixed = 38 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | HADOOP-16466 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982187/HADOOP-16466.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7eb859504709 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2478cba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16572/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16572/artifact/out/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP
[GitHub] [hadoop] fapifta opened a new pull request #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used
fapifta opened a new pull request #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used URL: https://github.com/apache/hadoop/pull/1596 There are a couple of things in this pull request. ByteStringHelper was used in two general places, once in BlockOutputStream, and once in ChunkUtils. ChunkUtils was an easy one, as in that case the method was used directly from KeyValueHandler only, where the configuration is present. I moved the logic back to the KeyValueHandler class, and at initialization time, I created the proper converter based on the config. At this point I realized that we use byte[] type in readChunk like methods here and there, but at the end most of the usage outside is via a ByteBuffer after a ByteBuffer.wrap(byte[]) call, so I changed the readChunk logic to return a ByteBuffer instead, and used that one, with this I eliminated the need of a conversion from byte[] to ByteString. BlockOutputStream was a bit harder, as in that case, RPCClient initialized the old class with the config. The decision became the following: based on the Configuration, RPCClient already created an XCeiverClientManager that preserved a reference to the configuration, and which was already available in the BlockOutputStreamEntryPool class, that uses a BufferPool to create the buffers in BlockOutputStream, so that BlockOutputStream when does the conversion can reach it via its BufferPool. I have cleaned up one code path, that came into my way during implementation, and I removed two declared exception from ChunkUtils.readData, and with that eliminated the need for a try-catch block in the place where it is called. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] fapifta commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used
fapifta commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used URL: https://github.com/apache/hadoop/pull/1596#issuecomment-538379814 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
hadoop-yetus commented on issue #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595#issuecomment-538382301 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 54 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 37 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 42 | hadoop-ozone in trunk failed. | | -1 | compile | 23 | hadoop-hdds in trunk failed. | | -1 | compile | 14 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 960 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 19 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1062 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 34 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 40 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 30 | hadoop-hdds: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 713 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2503 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1595 | | Optional Tests | dupname asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4a3e8233f620 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d061c84 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1595/2/artifact/out/patch-compile-hadoop-hdds.txt |
[GitHub] [hadoop] adoroszlai opened a new pull request #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars
adoroszlai opened a new pull request #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars URL: https://github.com/apache/hadoop/pull/1597 ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/HDDS-2250 ## How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars
adoroszlai commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars URL: https://github.com/apache/hadoop/pull/1597#issuecomment-538382616 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek closed pull request #1585: HDDS-2230. Invalid entries in ozonesecure-mr config
elek closed pull request #1585: HDDS-2230. Invalid entries in ozonesecure-mr config URL: https://github.com/apache/hadoop/pull/1585 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1442: HADOOP-16570. S3A committers encounter scale issues
steveloughran commented on a change in pull request #1442: HADOOP-16570. S3A
committers encounter scale issues
URL: https://github.com/apache/hadoop/pull/1442#discussion_r331485629
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##
@@ -833,4 +833,10 @@ private Constants() {
public static final String AWS_SERVICE_IDENTIFIER_S3 = "S3";
public static final String AWS_SERVICE_IDENTIFIER_DDB = "DDB";
public static final String AWS_SERVICE_IDENTIFIER_STS = "STS";
+
+ /**
+ * How long to wait for the thread pool to terminate when cleaning up.
+ * Value: {@value} seconds.
+ */
+ public static final int THREAD_POOL_SHUTDOWN_DELAY = 30;
Review comment:
will do. thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek closed pull request #1570: HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in co…
elek closed pull request #1570: HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in co… URL: https://github.com/apache/hadoop/pull/1570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1115: HADOOP-16207 testMR failures
steveloughran closed pull request #1115: HADOOP-16207 testMR failures URL: https://github.com/apache/hadoop/pull/1115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1115: HADOOP-16207 testMR failures
steveloughran commented on issue #1115: HADOOP-16207 testMR failures URL: https://github.com/apache/hadoop/pull/1115#issuecomment-538391305 thanks -merged in! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16207) Improved S3A MR tests
[
https://issues.apache.org/jira/browse/HADOOP-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16207:
Summary: Improved S3A MR tests (was: Fix
ITestDirectoryCommitMRJob.testMRJob)
> Improved S3A MR tests
> -
>
> Key: HADOOP-16207
> URL: https://issues.apache.org/jira/browse/HADOOP-16207
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Reported failure of {{ITestDirectoryCommitMRJob}} in validation runs of
> HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run:
> Path "is recorded as deleted by S3Guard"
> {code}
> waitForConsistency();
> assertIsDirectory(outputPath) /* here */
> {code}
> The file is there but there's a tombstone. Possibilities
> * some race condition with another test
> * tombstones aren't timing out
> * committers aren't creating that base dir in a way which cleans up S3Guard's
> tombstones.
> Remember: we do have to delete that dest dir before the committer runs unless
> overwrite==true, so at the start of the run there will be a tombstone. It
> should be overwritten by a success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on issue #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo commented on issue #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595#issuecomment-538392068 There were no findbugs warnings. Since there are no code changes except findbugsExcludeFile.xml compared to #1578 , will merge the commit soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo merged pull request #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
szetszwo merged pull request #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16207) Improved S3A MR tests
[
https://issues.apache.org/jira/browse/HADOOP-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16207.
-
Fix Version/s: 3.3.0
Resolution: Fixed
> Improved S3A MR tests
> -
>
> Key: HADOOP-16207
> URL: https://issues.apache.org/jira/browse/HADOOP-16207
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.3.0
>
>
> Reported failure of {{ITestDirectoryCommitMRJob}} in validation runs of
> HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run:
> Path "is recorded as deleted by S3Guard"
> {code}
> waitForConsistency();
> assertIsDirectory(outputPath) /* here */
> {code}
> The file is there but there's a tombstone. Possibilities
> * some race condition with another test
> * tombstones aren't timing out
> * committers aren't creating that base dir in a way which cleans up S3Guard's
> tombstones.
> Remember: we do have to delete that dest dir before the committer runs unless
> overwrite==true, so at the start of the run there will be a tombstone. It
> should be overwritten by a success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16207) Improved S3A MR tests
[
https://issues.apache.org/jira/browse/HADOOP-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944488#comment-16944488
]
Steve Loughran commented on HADOOP-16207:
-
Final patch replaces the committer-specific terasort and MR test jobs with
parameterization
of the (now single tests) and use of file:// over hdfs:// as the cluster FS.
The parameterization ensures that only one of the specific committer tests
run at a time -overloads of the test machines are less likely, and so the
suites can be pulled back into the parallel phase.
There's also more detailed validation of the stage outputs of the terasorting;
if one test fails the rest are all skipped. This and the fact that job
output is stored under target/yarn-${timestamp} means failures should
be more debuggable.
We also have the s3guard operations log enabled; on guarded runs this tracks
all PUT/DELETE/TOMBSTONE calls made of a store, so acts as the log of what
changes were made there. If we see intermittent issues here again (And after
the HADOOP-16570 changes) then we are better positioned to understand the
failures
> Improved S3A MR tests
> -
>
> Key: HADOOP-16207
> URL: https://issues.apache.org/jira/browse/HADOOP-16207
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Reported failure of {{ITestDirectoryCommitMRJob}} in validation runs of
> HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run:
> Path "is recorded as deleted by S3Guard"
> {code}
> waitForConsistency();
> assertIsDirectory(outputPath) /* here */
> {code}
> The file is there but there's a tombstone. Possibilities
> * some race condition with another test
> * tombstones aren't timing out
> * committers aren't creating that base dir in a way which cleans up S3Guard's
> tombstones.
> Remember: we do have to delete that dest dir before the committer runs unless
> overwrite==true, so at the start of the run there will be a tombstone. It
> should be overwritten by a success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16207) Improved S3A MR tests
[
https://issues.apache.org/jira/browse/HADOOP-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944495#comment-16944495
]
Hudson commented on HADOOP-16207:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17474 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17474/])
HADOOP-16207 Improved S3A MR tests. (stevel: rev
f44abc3e11676579bdea94fce045d081ae38e6c3)
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitMRJob.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITestPartitionCommitMRJob.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITestStagingCommitMRJobBadDest.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/terasort/ITestTerasortOnS3A.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/terasort/ITestTerasortMagicCommitter.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractYarnClusterITest.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/terasort/ITestTerasortDirectoryCommitter.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/magic/ITestMagicCommitMRJob.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITestDirectoryCommitMRJob.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractCommitITest.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitterConstants.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/integration/ITestS3ACommitterMRJob.java
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/terasort/AbstractCommitTerasortIT.java
* (edit) hadoop-tools/hadoop-aws/pom.xml
* (edit) hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
* (delete)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITestStagingCommitMRJob.java
> Improved S3A MR tests
> -
>
> Key: HADOOP-16207
> URL: https://issues.apache.org/jira/browse/HADOOP-16207
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.3.0
>
>
> Reported failure of {{ITestDirectoryCommitMRJob}} in validation runs of
> HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run:
> Path "is recorded as deleted by S3Guard"
> {code}
> waitForConsistency();
> assertIsDirectory(outputPath) /* here */
> {code}
> The file is there but there's a tombstone. Possibilities
> * some race condition with another test
> * tombstones aren't timing out
> * committers aren't creating that base dir in a way which cleans up S3Guard's
> tombstones.
> Remember: we do have to delete that dest dir before the committer runs unless
> overwrite==true, so at the start of the run there will be a tombstone. It
> should be overwritten by a success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars
hadoop-yetus commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars URL: https://github.com/apache/hadoop/pull/1597#issuecomment-538396361 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 82 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 30 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 32 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 969 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 20 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 40 | hadoop-ozone in the patch failed. | | -1 | compile | 22 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 22 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 811 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 21 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 2272 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1597 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 03569846e5a9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6171a41 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https
[GitHub] [hadoop] hadoop-yetus commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used
hadoop-yetus commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used URL: https://github.com/apache/hadoop/pull/1596#issuecomment-538396987 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 65 | Maven dependency ordering for branch | | -1 | mvninstall | 40 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 39 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 59 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 854 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 20 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 951 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 23 | hadoop-hdds in the patch failed. | | -1 | compile | 17 | hadoop-ozone in the patch failed. | | -1 | javac | 23 | hadoop-hdds in the patch failed. | | -1 | javac | 17 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 701 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in the patch failed. | | -1 | javadoc | 19 | hadoop-ozone in the patch failed. | | -1 | findbugs | 33 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 27 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2437 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1596 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 954300498c97 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6171a41 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1596/1/artifact/out/patch-compile-hadoop-hdds.txt |
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[
https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944506#comment-16944506
]
Hadoop QA commented on HADOOP-16598:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
40s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
0s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
37s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
59s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
33s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
8s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
40s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
33s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
45s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 54s{color}
| {color:red} bkjournal in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
48s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}250m 9s{color} |
{color
[GitHub] [hadoop] hadoop-yetus commented on issue #1593: HDDS-2204. Avoid buffer coping in checksum verification.
hadoop-yetus commented on issue #1593: HDDS-2204. Avoid buffer coping in checksum verification. URL: https://github.com/apache/hadoop/pull/1593#issuecomment-538409261 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 17 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 845 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in trunk failed. | | -1 | javadoc | 21 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 946 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 31 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 721 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in the patch failed. | | -1 | javadoc | 19 | hadoop-ozone in the patch failed. | | -1 | findbugs | 33 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 27 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2351 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1593 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e58ac87a6281 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 531cc93 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1593/2/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://bui
[GitHub] [hadoop] dineshchitlangia commented on issue #1542: HDDS-2140. Add robot test for GDPR feature
dineshchitlangia commented on issue #1542: HDDS-2140. Add robot test for GDPR feature URL: https://github.com/apache/hadoop/pull/1542#issuecomment-538416545 > > Thereby, even if you get your hands on chunks, you will still read encrypted junk :) > > Yes, I understand. That's the question. How can we prove that we only have junk. I would like to prove that the junk is encrypted. @elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1542: HDDS-2140. Add robot test for GDPR feature
elek commented on issue #1542: HDDS-2140. Add robot test for GDPR feature URL: https://github.com/apache/hadoop/pull/1542#issuecomment-538425539 > @elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR Wow, and with perfect java description. Yes, it's exactly that. It's my shame that I didn't know. You deserve a photo with this source code: http://www.peteradamsphoto.com/apache-20th-anniversary/ ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-538427465 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 76 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 11 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1202 | trunk passed | | +1 | compile | 32 | trunk passed | | +1 | checkstyle | 24 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 26 | trunk passed | | 0 | spotbugs | 56 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 55 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 32 | the patch passed | | +1 | compile | 28 | the patch passed | | +1 | javac | 28 | the patch passed | | -0 | checkstyle | 19 | hadoop-tools/hadoop-aws: The patch generated 6 new + 43 unchanged - 0 fixed = 49 total (was 43) | | +1 | mvnsite | 31 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 857 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 62 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 79 | hadoop-aws in the patch passed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3561 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/12/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1442 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 75b04da25d99 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 531cc93 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/12/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/12/testReport/ | | Max. process+thread count | 421 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/12/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-538430235 rebasing against trunk HADOOP-16207 I'm getting some failures which I'm blaming on test setup code, such as the fact that temp dirs on forked runs are coming in wrong: ``` fs.s3a.buffer.dirtarget/build/test:fork-0001 ``` I'll address here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early
hadoop-yetus commented on issue #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early URL: https://github.com/apache/hadoop/pull/1584#issuecomment-538438155 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 30 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 48 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 871 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 969 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 19 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 56 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 719 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2367 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1584 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 482bb984fa76 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 531cc93 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1584/2/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | h
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944587#comment-16944587 ] Wei-Chiu Chuang commented on HADOOP-16579: -- +1 will commit soon. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-538246119 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 71 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for branch | | -1 | mvninstall | 29 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 54 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 942 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 20 | hadoop-hdds in trunk failed. | | -1 | javadoc | 17 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1030 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 30 | hadoop-hdds in trunk failed. | | -1 | findbugs | 16 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 15 | Maven dependency ordering for patch | | -1 | mvninstall | 31 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 35 | hadoop-ozone in the patch failed. | | -1 | compile | 21 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 21 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 28 | hadoop-ozone: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 801 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 20 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 28 | hadoop-hdds in the patch failed. | | -1 | findbugs | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 23 | hadoop-ozone in the patch failed. | | +1 | asflicense | 30 | The patch does not generate ASF License warnings. | | | | 2522 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1589 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fb4933b9414e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 844b766 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-538247315 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 921 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1009 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 17 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 15 | Maven dependency ordering for patch | | -1 | mvninstall | 31 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | compile | 21 | hadoop-hdds in the patch failed. | | -1 | compile | 15 | hadoop-ozone in the patch failed. | | -1 | javac | 21 | hadoop-hdds in the patch failed. | | -1 | javac | 15 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 26 | hadoop-ozone: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 774 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 28 | hadoop-hdds in the patch failed. | | -1 | findbugs | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 23 | hadoop-hdds in the patch failed. | | -1 | unit | 23 | hadoop-ozone in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 2431 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1589 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 74a8dbbb1c89 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 844b766 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/P
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
hadoop-yetus removed a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-538260268 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 308 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 1 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 58 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 868 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 973 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 35 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 18 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 24 | hadoop-hdds in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 24 | hadoop-hdds in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 30 | hadoop-ozone: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 757 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 27 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2736 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1589 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d373ef588be1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c99a121 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1589/3/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR
[jira] [Created] (HADOOP-16632) Partitioned S3A magic committers can leaving pending files, maybe upload data
Steve Loughran created HADOOP-16632:
---
Summary: Partitioned S3A magic committers can leaving pending
files, maybe upload data
Key: HADOOP-16632
URL: https://issues.apache.org/jira/browse/HADOOP-16632
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.3, 3.2.1
Reporter: Steve Loughran
Assignee: Steve Loughran
Partitioned S3A magic committers can leaving pending files, maybe upload data
This surfaced in an assertion failure on a parallel test run.
I thought it was actually a test failure, but with HADOOP-16207 all the docs
are preserved in the local FS and I can understand what happened.
h3. Junit process
{code}
[INFO]
[ERROR] Failures:
[ERROR]
ITestS3ACommitterMRJob.test_200_execute:344->customPostExecutionValidation:356
Expected a java.io.FileNotFoundException to be thrown, but got the result: :
"Found magic dir which should have been deleted at
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic;
isDirectory=true; modification_time=0; access_time=0; owner=stevel;
group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null
versionId=null
[s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8
s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8.pending
{code}
Full details to follow in the comment as they are, well, detailed.
Key point: AM-side job and task cleanup can happen before the worker task
finishes its writes. This will result in files under __magic. It may result in
pending uploads too -but only if the write began after the AM job cleanup did a
list + abort of all pending uploads under the destination directory
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16632) Partitioned S3A magic committers can leave pending files under __magic, maybe uploads
[
https://issues.apache.org/jira/browse/HADOOP-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16632:
Summary: Partitioned S3A magic committers can leave pending files under
__magic, maybe uploads (was: Partitioned S3A magic committers can leaving
pending files, maybe upload data)
> Partitioned S3A magic committers can leave pending files under __magic, maybe
> uploads
> -
>
> Key: HADOOP-16632
> URL: https://issues.apache.org/jira/browse/HADOOP-16632
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1, 3.1.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> Partitioned S3A magic committers can leaving pending files, maybe upload data
> This surfaced in an assertion failure on a parallel test run.
> I thought it was actually a test failure, but with HADOOP-16207 all the docs
> are preserved in the local FS and I can understand what happened.
> h3. Junit process
> {code}
> [INFO]
> [ERROR] Failures:
> [ERROR]
> ITestS3ACommitterMRJob.test_200_execute:344->customPostExecutionValidation:356
> Expected a java.io.FileNotFoundException to be thrown, but got the result: :
> "Found magic dir which should have been deleted at
> S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic;
> isDirectory=true; modification_time=0; access_time=0; owner=stevel;
> group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
> isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null
> versionId=null
> [s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8
> s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8.pending
> {code}
> Full details to follow in the comment as they are, well, detailed.
>
> Key point: AM-side job and task cleanup can happen before the worker task
> finishes its writes. This will result in files under __magic. It may result
> in pending uploads too -but only if the write began after the AM job cleanup
> did a list + abort of all pending uploads under the destination directory
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16632) Partitioned S3A magic committers can leave pending files under __magic, maybe uploads
[
https://issues.apache.org/jira/browse/HADOOP-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944624#comment-16944624
]
Steve Loughran commented on HADOOP-16632:
-
This test is failing because the successful job should have deleted the __magic
directory, but the assertions finds the output of an attempt. Which is
potentially the sign of something going very wrong,
As it is -all is good, except for the cleanup, which is beyond our control.
For some reason this task was speculatively executed, and the second attempt
`_0003_m_08_1` was not committed before the job completed and the AM
terminated. When the worker requested permission to commit, it couldn't connect
over the (broken) RPC channel, retried a bit and gave up.
h3. task attempt_1570197469968_0003_m_08_1
{code:java}
2019-10-04 15:02:34,237 INFO [main]
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: File
s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8
is written as magic file to path
fork-0001/test/ITestS3ACommitterMRJob-execute-magic/part-m-8
2019-10-04 15:02:35,717 INFO [main]
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: Uncommitted data
pending to file
s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8;
commit metadata for 1 parts in
fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8.pending.
sixe: 4690 byte(s)
2019-10-04 15:02:36,877 INFO [main]
org.apache.hadoop.fs.s3a.S3ABlockOutputStream: File
fork-0001/test/ITestS3ACommitterMRJob-execute-magic/part-m-8 will be
visible when the job is committed
2019-10-04 15:02:36,888 INFO [main] org.apache.hadoop.mapred.Task:
Task:attempt_1570197469968_0003_m_08_1 is done. And is in the process of
committing
2019-10-04 15:02:36,889 INFO [main]
org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Starting:
needsTaskCommit task attempt_1570197469968_0003_m_08_1
2019-10-04 15:02:39,118 INFO [main]
org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: needsTaskCommit
task attempt_1570197469968_0003_m_08_1: duration 0:02.230s
2019-10-04 15:02:39,124 WARN [main] org.apache.hadoop.mapred.Task: Failure
sending commit pending: java.io.EOFException: End of File Exception between
local host is: "HW13176-2.local/192.168.1.6"; destination host is:
"localhost":57166; : java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:837)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1499)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at com.sun.proxy.$Proxy8.commitPending(Unknown Source)
at org.apache.hadoop.mapred.Task.done(Task.java:1253)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:351)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1872)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1182)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1078)
{code}
h3. task attempt_1570197469968_0003_m_08_1
And it looks like MR task failure is a no-warning System.exit(-1) call -the
workers do not get a chance to do any cleanup themselves.
{code:java}
2019-10-04 15:03:39,315 WARN [communication thread]
org.apache.hadoop.mapred.Task: Last retry, killing
attempt_1570197469968_0003_m_08_1
{code}
An another attempt at the same task had already been committed. Thus: the
output of this attempt was not needed
[GitHub] [hadoop] steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-538461283 Also filed: https://issues.apache.org/jira/browse/HADOOP-16632 The failed assertion was caused by a speculative task writing its .pending output file to its attempt directory after the job had completed. This is my first full trace what happens during a partition and I am pleased the actual output of the job was correct. We just can't prevent partitioned MR tasks from writing to the attempt directories after the job completes -and as there is a risk that pending uploads may be outstanding, document the need to have a life cycle rule to clean these up. Which people should have anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars
hadoop-yetus commented on issue #1597: HDDS-2250. Generated configs missing from ozone-filesystem-lib jars URL: https://github.com/apache/hadoop/pull/1597#issuecomment-538466122 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 38 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 36 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 974 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 31 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | compile | 22 | hadoop-hdds in the patch failed. | | -1 | compile | 14 | hadoop-ozone in the patch failed. | | -1 | javac | 22 | hadoop-hdds in the patch failed. | | -1 | javac | 14 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 810 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 29 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 2227 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1597 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux d86e32ee678d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / f826420 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1597/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | http
[GitHub] [hadoop] cxorm commented on issue #1570: HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in co…
cxorm commented on issue #1570: HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in co… URL: https://github.com/apache/hadoop/pull/1570#issuecomment-538468666 Thanks @elek and @adoroszlai for the reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 merged pull request #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early
bharatviswa504 merged pull request #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early URL: https://github.com/apache/hadoop/pull/1584 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early
bharatviswa504 commented on issue #1584: HDDS-2237. KeyDeletingService throws NPE if it's started too early URL: https://github.com/apache/hadoop/pull/1584#issuecomment-538475911 Thank You @elek for the contribution and @arp7 for the review. I have committed this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1595: HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C
elek commented on issue #1595: HDDS-. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C URL: https://github.com/apache/hadoop/pull/1595#issuecomment-538477400 Checkstyle errors seem to be introduced: ``` https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds--6c7sc/checkstyle/summary.txt ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek opened a new pull request #1598: HDDS-2251. Add an option to customize unit.sh and integration.sh parameters
elek opened a new pull request #1598: HDDS-2251. Add an option to customize unit.sh and integration.sh parameters URL: https://github.com/apache/hadoop/pull/1598 hadoop-ozone/dev-support/checks/unit.sh (and same with integration) provides an easy entrypoint to execute all the unit/integration test. But in same cases it would be great to use the script but further specify the scope of the test. With this simple patch it will be possible to adjust the surefire parameters. See: https://issues.apache.org/jira/browse/HDDS-2251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
steveloughran commented on a change in pull request #1481: HADOOP-16587: Made
auth endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r331600662
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
##
@@ -109,17 +112,16 @@ public static AzureADToken
getTokenUsingClientCreds(String authEndpoint,
* @return {@link AzureADToken} obtained using the creds
* @throws IOException throws IOException if there is a failure in obtaining
the token
*/
- public static AzureADToken getTokenFromMsi(String tenantGuid, String
clientId,
- boolean bypassCache) throws
IOException {
-String authEndpoint =
"http://169.254.169.254/metadata/identity/oauth2/token";;
-
+ public static AzureADToken getTokenFromMsi(final String authEndpoint,
+ final String tenantGuid, final String clientId, String authority,
+ boolean bypassCache) throws IOException {
QueryParams qp = new QueryParams();
qp.add("api-version", "2018-02-01");
qp.add("resource", RESOURCE_NAME);
-
if (tenantGuid != null && tenantGuid.length() > 0) {
- String authority = "https://login.microsoftonline.com/"; + tenantGuid;
+ authority = authority + tenantGuid;
+ LOG.debug("MSI authority : " + authority);
Review comment:
Can you do a `LOG.debug("'MSI Authority :{}", authority)` . It'll save
building the string up when a debug level log isn't needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
steveloughran commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows URL: https://github.com/apache/hadoop/pull/1481#issuecomment-538483491 Ok, all LGTM. One final little nit about the debug string concatenation (SLF4J does it itself when needed; it's only commons-logging where you need to do it by hand) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils.
hadoop-yetus commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils. URL: https://github.com/apache/hadoop/pull/1486#issuecomment-538483912 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 10 | https://github.com/apache/hadoop/pull/1486 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1486 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/5/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16466) Clean up the Assert usage in tests
[ https://issues.apache.org/jira/browse/HADOOP-16466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944697#comment-16944697 ] Íñigo Goiri commented on HADOOP-16466: -- There is one of the checkstyles that we introduced. However, I think we can take the chance and fix the indentation in all the areas close to the changes. > Clean up the Assert usage in tests > -- > > Key: HADOOP-16466 > URL: https://issues.apache.org/jira/browse/HADOOP-16466 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HADOOP-16466.001.patch > > > This tickets started with https://issues.apache.org/jira/browse/HDFS-14449 > and we would like to clean up all of the Assert usage in tests to make the > repo cleaner. This mainly is to make use static imports for the Assert > functions and use function call without the *Assert.* explicitly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16632) Partitioned S3A magic committers can leave pending files under __magic, maybe uploads
[
https://issues.apache.org/jira/browse/HADOOP-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944698#comment-16944698
]
Steve Loughran commented on HADOOP-16632:
-
(FWIW this is surfacing because with the move of the committers back to the
parallel phase, 12 threads + scale overloads my laptop. I need to either get
hold of a better box or use smaller thread counts)
> Partitioned S3A magic committers can leave pending files under __magic, maybe
> uploads
> -
>
> Key: HADOOP-16632
> URL: https://issues.apache.org/jira/browse/HADOOP-16632
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1, 3.1.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> Partitioned S3A magic committers can leaving pending files, maybe upload data
> This surfaced in an assertion failure on a parallel test run.
> I thought it was actually a test failure, but with HADOOP-16207 all the docs
> are preserved in the local FS and I can understand what happened.
> h3. Junit process
> {code}
> [INFO]
> [ERROR] Failures:
> [ERROR]
> ITestS3ACommitterMRJob.test_200_execute:344->customPostExecutionValidation:356
> Expected a java.io.FileNotFoundException to be thrown, but got the result: :
> "Found magic dir which should have been deleted at
> S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic;
> isDirectory=true; modification_time=0; access_time=0; owner=stevel;
> group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
> isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null
> versionId=null
> [s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8
> s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8.pending
> {code}
> Full details to follow in the comment as they are, well, detailed.
>
> Key point: AM-side job and task cleanup can happen before the worker task
> finishes its writes. This will result in files under __magic. It may result
> in pending uploads too -but only if the write began after the AM job cleanup
> did a list + abort of all pending uploads under the destination directory
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-538491961 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 12 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1211 | trunk passed | | +1 | compile | 32 | trunk passed | | +1 | checkstyle | 25 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 836 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 59 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | -0 | checkstyle | 19 | hadoop-tools/hadoop-aws: The patch generated 7 new + 49 unchanged - 0 fixed = 56 total (was 49) | | +1 | mvnsite | 30 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 907 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 24 | the patch passed | | +1 | findbugs | 72 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 97 | hadoop-aws in the patch passed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 3594 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1442 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f37178571813 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4510970 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/13/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/13/testReport/ | | Max. process+thread count | 342 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/13/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config
adoroszlai commented on issue #1585: HDDS-2230. Invalid entries in ozonesecure-mr config URL: https://github.com/apache/hadoop/pull/1585#issuecomment-538492308 Thanks @xiaoyuyao and @elek for reviewing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hanishakoneru commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils.
hanishakoneru commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils. URL: https://github.com/apache/hadoop/pull/1486#issuecomment-538492516 Thanks @bharatviswa504 and @arp7 for the reviews. Checkstyle issues were related. Fixed that. Opened HDDS-2255 to for improving the messages. Will commit the patch after CI run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia commented on issue #1542: HDDS-2140. Add robot test for GDPR feature
dineshchitlangia commented on issue #1542: HDDS-2140. Add robot test for GDPR feature URL: https://github.com/apache/hadoop/pull/1542#issuecomment-538492692 > > @elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR > > Wow, and with perfect java description. Yes, it's exactly that. It's my shame that I didn't know. > > You deserve a photo with this source code: > http://www.peteradamsphoto.com/apache-20th-anniversary/ > > ;-) Thanks, because it was a complex test and I wanted to ensure that no one accidentally modifies the test without getting the context, I made it so descriptive. Glad it looks good and helps. Good Idea, I will get a pic like that 😄 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16633) Tunehadoop-aws parallel tests to use balanced runorder for parallel tests
Steve Loughran created HADOOP-16633: --- Summary: Tunehadoop-aws parallel tests to use balanced runorder for parallel tests Key: HADOOP-16633 URL: https://issues.apache.org/jira/browse/HADOOP-16633 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3, test Affects Versions: 3.2.1 Reporter: Steve Loughran Maven failsafe and surefire parallel runners support a runorder attribute which determines how parallel runs are chosen [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16633) Tune hadoop-aws parallel test surefire/failsafe settings
[ https://issues.apache.org/jira/browse/HADOOP-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16633: Summary: Tune hadoop-aws parallel test surefire/failsafe settings (was: Tunehadoop-aws parallel tests to use balanced runorder for parallel tests) > Tune hadoop-aws parallel test surefire/failsafe settings > > > Key: HADOOP-16633 > URL: https://issues.apache.org/jira/browse/HADOOP-16633 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3, test >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Priority: Major > > Maven failsafe and surefire parallel runners support a runorder attribute > which determines how parallel runs are chosen > [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1598: HDDS-2251. Add an option to customize unit.sh and integration.sh parameters
hadoop-yetus commented on issue #1598: HDDS-2251. Add an option to customize unit.sh and integration.sh parameters URL: https://github.com/apache/hadoop/pull/1598#issuecomment-538494612 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 84 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 32 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 895 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 38 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 43 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 1 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 846 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | -1 | unit | 26 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 2193 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1598 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs | | uname | Linux 2f550eed1242 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3f16651 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/testReport/ | | Max. process+thread count | 305 (vs. ulimit of 5500) | | modules | C: hadoop-ozone U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1598/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on issue #1517: HDDS-2169. Avoid buffer copies while submitting client requests in Ratis
bshashikant commented on issue #1517: HDDS-2169. Avoid buffer copies while
submitting client requests in Ratis
URL: https://github.com/apache/hadoop/pull/1517#issuecomment-538495734
Thanks @szetszwo for updating the patch. I tried to run the tests in
TestDataValidateWithUnsafeByteOperations and i see the following exception
being thrown:
`2019-10-04 23:13:25,556
[ce18dfb1-da4d-401f-9614-bec32477b5f3@group-0099BCD205B6-SegmentedRaftLogWorker]
INFO segmented.SegmentedRaftLogWorker
(SegmentedRaftLogWorker.java:execute(574)) -
ce18dfb1-da4d-401f-9614-bec32477b5f3@group-0099BCD205B6-SegmentedRaftLogWorker:
created new log segment
/Users/sbanerjee/github_hadoop/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-cd3ca672-68cd-49fd-bdb3-a7fc97d18c23/datanode-1/data/ratis/ace05abb-b740-47f7-95d4-0099bcd205b6/current/log_inprogress_0
2019-10-04 23:13:25,557
[ee7f2721-1de4-4264-8bf3-d340e83f8791@group-0099BCD205B6-SegmentedRaftLogWorker]
INFO segmented.SegmentedRaftLogWorker
(SegmentedRaftLogWorker.java:execute(574)) -
ee7f2721-1de4-4264-8bf3-d340e83f8791@group-0099BCD205B6-SegmentedRaftLogWorker:
created new log segment
/Users/sbanerjee/github_hadoop/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-cd3ca672-68cd-49fd-bdb3-a7fc97d18c23/datanode-2/data/ratis/ace05abb-b740-47f7-95d4-0099bcd205b6/current/log_inprogress_0
2019-10-04 23:13:25,874 [pool-56-thread-1] ERROR impl.ChunkManagerImpl
(ChunkUtils.java:writeData(89)) - data array does not match the length
specified. DataLen: 1048576 Byte Array: 1048749
2019-10-04 23:13:25,874 [pool-56-thread-1] INFO keyvalue.KeyValueHandler
(ContainerUtils.java:logAndReturnError(146)) - Operation: WriteChunk : Trace
ID: cab5af5eafbad5ed:6a87e816d7e0ce20:e3ff42a900c31035:0 : Message: data array
does not match the length specified. DataLen: 1048576 Byte Array: 1048749 :
Result: INVALID_WRITE_SIZE
2019-10-04 23:13:25,881
[EventQueue-IncrementalContainerReportForIncrementalContainerReportHandler]
WARN container.IncrementalContainerReportHandler
(AbstractContainerReportHandler.java:updateContainerState(143)) - Container #1
is in OPEN state, but the datanode eb79af53-823f-485d-8402-ff71443cc79f{ip:
192.168.0.64, host: 192.168.0.64, networkLocation: /default-rack, certSerialId:
null} reports an UNHEALTHY replica.
23:13:25.886 [pool-56-thread-1] ERROR DNAudit - user=null | ip=null |
op=WRITE_CHUNK {blockData=conID: 1 locID: 102905348118937600 bcsId: 0} |
ret=FAILURE
java.lang.Exception: data array does not match the length specified.
DataLen: 1048576 Byte Array: 1048749
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:330)
~[classes/:?]
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:150)
~[classes/:?]
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:411)
~[classes/:?]
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:419)
~[classes/:?]
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$1(ContainerStateMachine.java:454)
~[classes/:?]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2019-10-04 23:13:25,896 [pool-56-thread-1] ERROR ratis.ContainerStateMachine
(ContainerStateMachine.java:lambda$handleWriteChunk$2(474)) -
group-0099BCD205B6: writeChunk writeStateMachineData failed:
blockIdcontainerID: 1
localID: 102905348118937600
blockCommitSequenceId: 0
logIndex 1 chunkName 102905348118937600_chunk_1 Error message: data array
does not match the length specified. DataLen: 1048576 Byte Array: 1048749
Container Result: INVALID_WRITE_SIZE
`
Can you please check?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on issue #1528: HDDS-2181. Ozone Manager should send correct ACL type in ACL requests…
xiaoyuyao commented on issue #1528: HDDS-2181. Ozone Manager should send correct ACL type in ACL requests… URL: https://github.com/apache/hadoop/pull/1528#issuecomment-538495750 Thanks @vivekratnavel for the update. The latest change in OzoneNativeAuthorizer LGTM. Can you take look at the CI failures, some of them seem related to this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1536: HDDS-2164 : om.db.checkpoints is getting filling up fast.
anuengineer commented on issue #1536: HDDS-2164 : om.db.checkpoints is getting filling up fast. URL: https://github.com/apache/hadoop/pull/1536#issuecomment-538496672 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1536: HDDS-2164 : om.db.checkpoints is getting filling up fast.
avijayanhwx commented on issue #1536: HDDS-2164 : om.db.checkpoints is getting filling up fast. URL: https://github.com/apache/hadoop/pull/1536#issuecomment-538497245 > @avijayanhwx findbugs violations are related to this change, can you take a look at it? My latest patch should fix the findbugs issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1442: HADOOP-16570. S3A committers encounter scale issues
steveloughran closed pull request #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
steveloughran commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-538499572 ok, merged in. thank you for the reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #1531: HADOOP-16579 - Upgrade to Apache Curator 4.2.0 excluding ZK
jojochuang merged pull request #1531: HADOOP-16579 - Upgrade to Apache Curator 4.2.0 excluding ZK URL: https://github.com/apache/hadoop/pull/1531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16570) S3A committers leak threads/raises OOM on job/task commit at scale
[ https://issues.apache.org/jira/browse/HADOOP-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16570. - Fix Version/s: 3.3.0 Resolution: Fixed done. I do have a backlog of things to go to 3.2 and 3.1, this should be on it. Most people won't hit the job commit problem, but thread pool leaks -inevitably > S3A committers leak threads/raises OOM on job/task commit at scale > -- > > Key: HADOOP-16570 > URL: https://issues.apache.org/jira/browse/HADOOP-16570 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > > The fixed size ThreadPool created in AbstractS3ACommitter doesn't get cleaned > up at EOL; as a result you leak the no. of threads set in > "fs.s3a.committer.threads" > Not visible in MR/distcp jobs, but ultimately causes OOM on Spark -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16579: - Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~nkalmar]. The patch passes our internal basic tests. Merged the PR and we'll work on further integration tests. Will come back to this if something wrong happens. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > Fix For: 3.3.0 > > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16633) Tune hadoop-aws parallel test surefire/failsafe settings
[ https://issues.apache.org/jira/browse/HADOOP-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16633: Description: We can do more to improve our test runs by looking at the failsafe docs [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] * default value to be by core, eg. 2C * use the runorder attribute which determines how parallel runs are chosen; random for better nondeterminism We'd need to experiment first, which can be done by setting failsafe.runOrder was: Maven failsafe and surefire parallel runners support a runorder attribute which determines how parallel runs are chosen [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] > Tune hadoop-aws parallel test surefire/failsafe settings > > > Key: HADOOP-16633 > URL: https://issues.apache.org/jira/browse/HADOOP-16633 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3, test >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Priority: Major > > We can do more to improve our test runs by looking at the failsafe docs > [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] > > * default value to be by core, eg. 2C > * use the runorder attribute which determines how parallel runs are chosen; > random for better nondeterminism > We'd need to experiment first, which can be done by setting failsafe.runOrder -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16632) Speculating & Partitioned S3A magic committers can leave pending files under __magic
[
https://issues.apache.org/jira/browse/HADOOP-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16632:
Summary: Speculating & Partitioned S3A magic committers can leave pending
files under __magic (was: Partitioned S3A magic committers can leave pending
files under __magic, maybe uploads)
> Speculating & Partitioned S3A magic committers can leave pending files under
> __magic
>
>
> Key: HADOOP-16632
> URL: https://issues.apache.org/jira/browse/HADOOP-16632
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1, 3.1.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> Partitioned S3A magic committers can leaving pending files, maybe upload data
> This surfaced in an assertion failure on a parallel test run.
> I thought it was actually a test failure, but with HADOOP-16207 all the docs
> are preserved in the local FS and I can understand what happened.
> h3. Junit process
> {code}
> [INFO]
> [ERROR] Failures:
> [ERROR]
> ITestS3ACommitterMRJob.test_200_execute:344->customPostExecutionValidation:356
> Expected a java.io.FileNotFoundException to be thrown, but got the result: :
> "Found magic dir which should have been deleted at
> S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic;
> isDirectory=true; modification_time=0; access_time=0; owner=stevel;
> group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
> isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null
> versionId=null
> [s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8
> s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_08_1/__base/part-m-8.pending
> {code}
> Full details to follow in the comment as they are, well, detailed.
>
> Key point: AM-side job and task cleanup can happen before the worker task
> finishes its writes. This will result in files under __magic. It may result
> in pending uploads too -but only if the write began after the AM job cleanup
> did a list + abort of all pending uploads under the destination directory
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1599: HADOOP-16632. Speculating & Partitioned S3A magic committers can leave pending files under __magic
steveloughran opened a new pull request #1599: HADOOP-16632. Speculating & Partitioned S3A magic committers can leave pending files under __magic URL: https://github.com/apache/hadoop/pull/1599 Downgrade the checks for leftover __magic entries from fail to warn now the parallel test runs make speculation more likely. That or we disable specex. Change-Id: Ia4df2e90f82a06dbae69f3fdaadcbb0e0d713b38 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] fapifta commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used
fapifta commented on issue #1596: HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used URL: https://github.com/apache/hadoop/pull/1596#issuecomment-538504664 Acceptance test failures seems to be related to an internally examined similar failure in XCeiverClientGrpc, where the relevant stack trace is: Caused by: java.lang.NullPointerException at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:387) at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:285) at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:251) at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:234) at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:249) at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52) at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75) at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121) at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112) at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555) From the Integration test failures org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient failure might be related, but there in the output before the error we can see that the client wanted replication factor of 3, while the cluster had only 1 DN. So issues seem to be unrelated, however I can not run these tests locally as easy as they are timing out in my local machine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
