[GitHub] [hadoop] zhuxiangyi opened a new pull request #3155: HDFS-16095. Add lsQuotaList command and getQuotaListing api for hdfs …
zhuxiangyi opened a new pull request #3155: URL: https://github.com/apache/hadoop/pull/3155 …quota. ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #3136: HDFS-16086. Add volume information to datanode log for tracing
tomscut commented on a change in pull request #3136:
URL: https://github.com/apache/hadoop/pull/3136#discussion_r660262918
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
##
@@ -929,7 +929,7 @@ public void writeBlock(final ExtendedBlock block,
if (isDatanode ||
stage == BlockConstructionStage.PIPELINE_CLOSE_RECOVERY) {
datanode.closeBlock(block, null, storageUuid, isOnTransientStorage);
-LOG.info("Received {} src: {} dest: {} of size {}",
+LOG.info("Received {} src: {} dest: {} volume: {} of size {}",
Review comment:
Thanks for pointing this, I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #3136: HDFS-16086. Add volume information to datanode log for tracing
tomscut commented on a change in pull request #3136:
URL: https://github.com/apache/hadoop/pull/3136#discussion_r660251541
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##
@@ -1631,6 +1633,7 @@ public ReplicaHandler createRbw(
if (ref == null) {
ref = volumes.getNextVolume(storageType, storageId, b.getNumBytes());
}
+ LOG.info("Creating Rbw, block: {} on volume: {}", b, ref.getVolume());
Review comment:
> is this really necessary? IMO logging one message for every rbw is
just too much.
I will change this to DEBUG level, do you think it is OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #3136: HDFS-16086. Add volume information to datanode log for tracing
tomscut commented on a change in pull request #3136: URL: https://github.com/apache/hadoop/pull/3136#discussion_r660251192 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java ## @@ -587,7 +587,7 @@ public void readBlock(final ExtendedBlock block, final String clientTraceFmt = clientName.length() > 0 && ClientTraceLog.isInfoEnabled() ? String.format(DN_CLIENTTRACE_FORMAT, localAddress, remoteAddress, -"%d", "HDFS_READ", clientName, "%d", +"", "%d", "HDFS_READ", clientName, "%d", Review comment: Because volume has been added to DN_CLIENTTRACE_FORMAT, some adaptations have been made. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #3136: HDFS-16086. Add volume information to datanode log for tracing
tomscut commented on a change in pull request #3136:
URL: https://github.com/apache/hadoop/pull/3136#discussion_r660249250
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/Replica.java
##
@@ -19,49 +19,56 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState;
+import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi;
/**
* This represents block replicas which are stored in DataNode.
*/
@InterfaceAudience.Private
public interface Replica {
/** Get the block ID */
- public long getBlockId();
+ long getBlockId();
/** Get the generation stamp */
- public long getGenerationStamp();
+ long getGenerationStamp();
/**
* Get the replica state
* @return the replica state
*/
- public ReplicaState getState();
+ ReplicaState getState();
/**
* Get the number of bytes received
* @return the number of bytes that have been received
*/
- public long getNumBytes();
+ long getNumBytes();
/**
* Get the number of bytes that have written to disk
* @return the number of bytes that have written to disk
*/
- public long getBytesOnDisk();
+ long getBytesOnDisk();
/**
* Get the number of bytes that are visible to readers
* @return the number of bytes that are visible to readers
*/
- public long getVisibleLength();
+ long getVisibleLength();
Review comment:
Thanks @jojochuang for your review. This change is to fix checkstyle. I
will restore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3136: HDFS-16086. Add volume information to datanode log for tracing
jojochuang commented on a change in pull request #3136:
URL: https://github.com/apache/hadoop/pull/3136#discussion_r660246410
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/Replica.java
##
@@ -19,49 +19,56 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState;
+import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi;
/**
* This represents block replicas which are stored in DataNode.
*/
@InterfaceAudience.Private
public interface Replica {
/** Get the block ID */
- public long getBlockId();
+ long getBlockId();
/** Get the generation stamp */
- public long getGenerationStamp();
+ long getGenerationStamp();
/**
* Get the replica state
* @return the replica state
*/
- public ReplicaState getState();
+ ReplicaState getState();
/**
* Get the number of bytes received
* @return the number of bytes that have been received
*/
- public long getNumBytes();
+ long getNumBytes();
/**
* Get the number of bytes that have written to disk
* @return the number of bytes that have written to disk
*/
- public long getBytesOnDisk();
+ long getBytesOnDisk();
/**
* Get the number of bytes that are visible to readers
* @return the number of bytes that are visible to readers
*/
- public long getVisibleLength();
+ long getVisibleLength();
Review comment:
please do not change these interface methods. These changes are not
required and makes backport harder.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
##
@@ -587,7 +587,7 @@ public void readBlock(final ExtendedBlock block,
final String clientTraceFmt =
clientName.length() > 0 && ClientTraceLog.isInfoEnabled()
? String.format(DN_CLIENTTRACE_FORMAT, localAddress, remoteAddress,
-"%d", "HDFS_READ", clientName, "%d",
+"", "%d", "HDFS_READ", clientName, "%d",
Review comment:
looks like redundant change?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##
@@ -1631,6 +1633,7 @@ public ReplicaHandler createRbw(
if (ref == null) {
ref = volumes.getNextVolume(storageType, storageId, b.getNumBytes());
}
+ LOG.info("Creating Rbw, block: {} on volume: {}", b, ref.getVolume());
Review comment:
is this really necessary? IMO logging one message for every rbw is just
too much.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
##
@@ -929,7 +929,7 @@ public void writeBlock(final ExtendedBlock block,
if (isDatanode ||
stage == BlockConstructionStage.PIPELINE_CLOSE_RECOVERY) {
datanode.closeBlock(block, null, storageUuid, isOnTransientStorage);
-LOG.info("Received {} src: {} dest: {} of size {}",
+LOG.info("Received {} src: {} dest: {} volume: {} of size {}",
Review comment:
missing the parameter for volume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17777) Update clover-maven-plugin version from 3.3.0 to 4.4.1
[ https://issues.apache.org/jira/browse/HADOOP-1?focusedWorklogId=615964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615964 ] ASF GitHub Bot logged work on HADOOP-1: --- Author: ASF GitHub Bot Created on: 29/Jun/21 02:38 Start Date: 29/Jun/21 02:38 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3152: URL: https://github.com/apache/hadoop/pull/3152#issuecomment-870184654 LGTM. I suppose the test failures aren't related? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 615964) Time Spent: 0.5h (was: 20m) > Update clover-maven-plugin version from 3.3.0 to 4.4.1 > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Update clover-maven-plugin version to 4.4.1, some important changes are as > follows: > * the license key is no longer required to run Clover > * the Atlassian brand and logos were removed to avoid any trademark > violations > * the 'org.openclover.*' groupId is used for artifacts > The license key is no longer required is the key factor to do this update. So > the developer can run and get the code coverage easier. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3152: HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1
jojochuang commented on pull request #3152: URL: https://github.com/apache/hadoop/pull/3152#issuecomment-870184654 LGTM. I suppose the test failures aren't related? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17777) Update clover-maven-plugin version from 3.3.0 to 4.4.1
[
https://issues.apache.org/jira/browse/HADOOP-1?focusedWorklogId=615960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615960
]
ASF GitHub Bot logged work on HADOOP-1:
---
Author: ASF GitHub Bot
Created on: 29/Jun/21 02:29
Start Date: 29/Jun/21 02:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3152:
URL: https://github.com/apache/hadoop/pull/3152#issuecomment-870181348
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 2s | | trunk passed |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 24m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 125m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 19m 51s | | the patch passed |
| +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 47s | | the patch passed |
| +1 :green_heart: | compile | 18m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 11s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 39s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 29m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1069m 18s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3152/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 1308m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.tools.dynamometer.TestDynamometerInfra |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3152/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3152 |
| Optional Tests | dupname asflicense codespell compile javac javadoc
mvninstall mvnsite unit shadedclient xml |
| uname | Linux d5aadf635cb1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4db3b23ce4d6b9eb673bb6ceabbb3d9e0a8c8211 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3152: HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1
hadoop-yetus commented on pull request #3152:
URL: https://github.com/apache/hadoop/pull/3152#issuecomment-870181348
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 2s | | trunk passed |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 24m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 125m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 19m 51s | | the patch passed |
| +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 47s | | the patch passed |
| +1 :green_heart: | compile | 18m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 11s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 39s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 29m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1069m 18s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3152/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 1308m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.tools.dynamometer.TestDynamometerInfra |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3152/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3152 |
| Optional Tests | dupname asflicense codespell compile javac javadoc
mvninstall mvnsite unit shadedclient xml |
| uname | Linux d5aadf635cb1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4db3b23ce4d6b9eb673bb6ceabbb3d9e0a8c8211 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3152/1/testReport/ |
| Max. process+thread count | 2931 (vs. ulimit of 5500) |
| modules | C: hadoop-maven-plugins . U: . |
| Console output |
[jira] [Comment Edited] (HADOOP-17201) Spark job with s3acommitter stuck at the last stage
[
https://issues.apache.org/jira/browse/HADOOP-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370953#comment-17370953
]
Daniel Zhi edited comment on HADOOP-17201 at 6/28/21, 11:57 PM:
Our tasks using StagingCommitter could be "stuck" in
S3AFileSystem.deleteUnnecessaryFakeDirectories() for 300~500 seconds because
the code retries with exponential delay (not a problem by itself) and the final
exception being swallowed. The logic collects O(depth) parent dirs of an object
path and call s3.deleteObjects() on them. For example, putObjectDirect of
s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/file
would lead to batch delete of following 7 objects:
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/}}
# {{s3://bucket/05aee454/root/uuid/}}
# {{s3://bucket/05aee454/root/}}
# {{s3://bucket/05aee454/}}
Such batch delete happens in every putObjectDirect() and rename(), and the keys
spread across multiple levels. Either simply due to too many of such calls, and
likely the fact that keys are across levels hence contention across parallel
such requests, it triggers a large number of SlowDown errors from S3. The retry
with exponential delay (and finally swallowing the exception) makes the task
just appear to hang without progress for sometime O(500) seconds, and no
obvious failure in the end.
Most of the fake parent paths end with "/" actually do not exist in S3 (maybe
except the immediate one created by createFakeDirectory). It is not clear to me
what's the exact purpose of deleteUnnecessaryFakeDirectories() but the impact
is severe when use s3 as the defaultFS (for the simplicity to avoid managing
HDFS and S3 being strong consistent [https://aws.amazon.com/s3/consistency/]).
was (Author: danzhi):
Our tasks using StagingCommitter could be "stuck" in
S3AFileSystem.deleteUnnecessaryFakeDirectories() for 300~500 seconds because
the code retries with exponential delay (not a problem by itself) and the final
exception being swallowed. The logic collects O(depth) parent dirs of an object
path and call s3.deleteObjects() on them. For example, putObjectDirect of
s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/file
would lead to batch delete of following 7 objects:
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/}}
# {{s3://bucket/05aee454/root/uuid/}}
# {{s3://bucket/05aee454/root/}}
# {{s3://bucket/05aee454/}}
Such batch delete happens in every putObjectDirect() and rename(), and the keys
spread across multiple levels. Either simply due to too many of such calls, and
likely the fact that keys are across levels hence contention across parallel
such requests, it trigger large number of SlowDown error from S3. The retry
with exponential delay (and finally swallow the exception) make the task just
appear as hang without progress for sometime O(500) seconds, and no obvious
failure in the end.
Most of the fake parent paths end with "/" actually do not exist in S3 (maybe
except the immediate one created by createFakeDirectory). It is not clear to me
what's the exact purpose of deleteUnnecessaryFakeDirectories() but the impact
is severe when use s3 as the defaultFS (for the simplicity to avoid managing
HDFS and S3 being strong consistent [https://aws.amazon.com/s3/consistency/]).
> Spark job with s3acommitter stuck at the last stage
> ---
>
> Key: HADOOP-17201
> URL: https://issues.apache.org/jira/browse/HADOOP-17201
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: we are on spark 2.4.5/hadoop 3.2.1 with s3a committer.
> spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
> spark.hadoop.fs.s3a.committer.name: magic
>Reporter: Dyno
>Priority: Major
> Labels: pull-request-available
> Attachments: exec-120.log, exec-125.log, exec-25.log, exec-31.log,
> exec-36.log, exec-44.log, exec-5.log, exec-64.log, exec-7.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> usually our spark job took 1 hour or 2 to finish, occasionally it runs for
> more than 3 hour and then we know it's stuck and usually the executor has
> stack like this
> {{
> "Executor task launch worker for task 78620" #265 daemon prio=5 os_prio=0
> tid=0x7f73e0005000 nid=0x12d waiting on condition [0x7f74cb291000]
>
[jira] [Comment Edited] (HADOOP-17201) Spark job with s3acommitter stuck at the last stage
[
https://issues.apache.org/jira/browse/HADOOP-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370953#comment-17370953
]
Daniel Zhi edited comment on HADOOP-17201 at 6/28/21, 11:43 PM:
Our tasks using StagingCommitter could be "stuck" in
S3AFileSystem.deleteUnnecessaryFakeDirectories() for 300~500 seconds because
the code retries with exponential delay (not a problem by itself) and the final
exception being swallowed. The logic collects O(depth) parent dirs of an object
path and call s3.deleteObjects() on them. For example, putObjectDirect of
s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/file
would lead to batch delete of following 7 objects:
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/}}
# {{s3://bucket/05aee454/root/uuid/}}
# {{s3://bucket/05aee454/root/}}
# {{s3://bucket/05aee454/}}
Such batch delete happens in every putObjectDirect() and rename(), and the keys
spread across multiple levels. Either simply due to too many of such calls, and
likely the fact that keys are across levels hence contention across parallel
such requests, it trigger large number of SlowDown error from S3. The retry
with exponential delay (and finally swallow the exception) make the task just
appear as hang without progress for sometime O(500) seconds, and no obvious
failure in the end.
Most of the fake parent paths end with "/" actually do not exist in S3 (maybe
except the immediate one created by createFakeDirectory). It is not clear to me
what's the exact purpose of deleteUnnecessaryFakeDirectories() but the impact
is severe when use s3 as the defaultFS (for the simplicity to avoid managing
HDFS and S3 being strong consistent [https://aws.amazon.com/s3/consistency/]).
was (Author: danzhi):
I noticed the task could be "stuck" in
S3AFileSystem.deleteUnnecessaryFakeDirectories() for 300~500 seconds because
the code retries with exponential delay (not a problem by itself) and the final
exception being swallowed. The logic collects O(depth) parent dirs of an object
path and call s3.deleteObjects() on them. For example, putObjectDirect of
s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/file
would lead to following 7 objects:
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/}}
# {{s3://bucket/05aee454/root/uuid/}}
# {{s3://bucket/05aee454/root/}}
# {{s3://bucket/05aee454/}}
Such batch delete happens in every putObjectDirect() and rename(), and the keys
spread across multiple levels. Either simply due to too many of such calls,
and/or the fact is delete keys across levels with key contention across
parallel requests, it trigger SlowDown error from S3. The retry with
exponential delay (and finally swallow the exception) make the task just appear
as hang without progress for sometime O(500) seconds.
Most of the fake parent path end with "/" actually does not exist in S3 (maybe
except the immediate one created by createFakeDirectory). It is not clear to me
what's the exact purpose of deleteUnnecessaryFakeDirectories() but the impact
is severe for scenarios using s3 as the defaultFS (for simplicity to avoid
managing HDFS and S3's announcement of strong consistency of
https://aws.amazon.com/s3/consistency/).
> Spark job with s3acommitter stuck at the last stage
> ---
>
> Key: HADOOP-17201
> URL: https://issues.apache.org/jira/browse/HADOOP-17201
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: we are on spark 2.4.5/hadoop 3.2.1 with s3a committer.
> spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
> spark.hadoop.fs.s3a.committer.name: magic
>Reporter: Dyno
>Priority: Major
> Labels: pull-request-available
> Attachments: exec-120.log, exec-125.log, exec-25.log, exec-31.log,
> exec-36.log, exec-44.log, exec-5.log, exec-64.log, exec-7.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> usually our spark job took 1 hour or 2 to finish, occasionally it runs for
> more than 3 hour and then we know it's stuck and usually the executor has
> stack like this
> {{
> "Executor task launch worker for task 78620" #265 daemon prio=5 os_prio=0
> tid=0x7f73e0005000 nid=0x12d waiting on condition [0x7f74cb291000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at
[GitHub] [hadoop] hadoop-yetus commented on pull request #3150: HDFS-16092. Avoid creating LayoutFlags redundant objects
hadoop-yetus commented on pull request #3150:
URL: https://github.com/apache/hadoop/pull/3150#issuecomment-870118931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 432m 8s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 518m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.TestViewDistributedFileSystemContract |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeWithHdfsScheme |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3150 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8567bd79b47e 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
[jira] [Commented] (HADOOP-17201) Spark job with s3acommitter stuck at the last stage
[
https://issues.apache.org/jira/browse/HADOOP-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370953#comment-17370953
]
Daniel Zhi commented on HADOOP-17201:
-
I noticed the task could be "stuck" in
S3AFileSystem.deleteUnnecessaryFakeDirectories() for 300~500 seconds because
the code retries with exponential delay (not a problem by itself) and the final
exception being swallowed. The logic collects O(depth) parent dirs of an object
path and call s3.deleteObjects() on them. For example, putObjectDirect of
s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/file
would lead to following 7 objects:
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/0/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/_temporary/}}
# {{s3://bucket/05aee454/root/uuid/staging-uploads/}}
# {{s3://bucket/05aee454/root/uuid/}}
# {{s3://bucket/05aee454/root/}}
# {{s3://bucket/05aee454/}}
Such batch delete happens in every putObjectDirect() and rename(), and the keys
spread across multiple levels. Either simply due to too many of such calls,
and/or the fact is delete keys across levels with key contention across
parallel requests, it trigger SlowDown error from S3. The retry with
exponential delay (and finally swallow the exception) make the task just appear
as hang without progress for sometime O(500) seconds.
Most of the fake parent path end with "/" actually does not exist in S3 (maybe
except the immediate one created by createFakeDirectory). It is not clear to me
what's the exact purpose of deleteUnnecessaryFakeDirectories() but the impact
is severe for scenarios using s3 as the defaultFS (for simplicity to avoid
managing HDFS and S3's announcement of strong consistency of
https://aws.amazon.com/s3/consistency/).
> Spark job with s3acommitter stuck at the last stage
> ---
>
> Key: HADOOP-17201
> URL: https://issues.apache.org/jira/browse/HADOOP-17201
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: we are on spark 2.4.5/hadoop 3.2.1 with s3a committer.
> spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
> spark.hadoop.fs.s3a.committer.name: magic
>Reporter: Dyno
>Priority: Major
> Labels: pull-request-available
> Attachments: exec-120.log, exec-125.log, exec-25.log, exec-31.log,
> exec-36.log, exec-44.log, exec-5.log, exec-64.log, exec-7.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> usually our spark job took 1 hour or 2 to finish, occasionally it runs for
> more than 3 hour and then we know it's stuck and usually the executor has
> stack like this
> {{
> "Executor task launch worker for task 78620" #265 daemon prio=5 os_prio=0
> tid=0x7f73e0005000 nid=0x12d waiting on condition [0x7f74cb291000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:349)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1457)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1717)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2785)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2751)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$finalizeMultipartUpload$1(WriteOperationHelper.java:238)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper$$Lambda$210/1059071691.execute(Unknown
> Source)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at
> org.apache.hadoop.fs.s3a.Invoker$$Lambda$23/586859139.execute(Unknown Source)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:226)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:271)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.complete(S3ABlockOutputStream.java:660)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$200(S3ABlockOutputStream.java:521)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:385)
> at
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #2836: HDFS-15936.Solve BlockSender#sendPacket() does not record SocketTimeout exception.
hadoop-yetus commented on pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#issuecomment-870082325
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 15s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 36 unchanged - 1
fixed = 36 total (was 37) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 235m 47s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 320m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2836 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1741da014125 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ee238561d9cb3c1a5c14ab69610fd420ab376319 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/testReport/ |
| Max. process+thread count | 3155 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615838
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 21:30
Start Date: 28/Jun/21 21:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 37 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 43s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0
fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 79m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 31624da3802c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecee1809de89ed5b4fc76df79dd64895537aab99 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 37 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 43s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0
fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 79m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 31624da3802c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecee1809de89ed5b4fc76df79dd64895537aab99 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/testReport/ |
| Max. process+thread count | 678 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615802
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:11
Start Date: 28/Jun/21 20:11
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084530
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
Review comment:
primaryRequestId is applicable for any method call that triggers more
than one http request. For example, methods using continuation logic like
listStatus and rename
Have added comments explaining use of tracingContext and its members at the
beginning of file
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615803
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:11
Start Date: 28/Jun/21 20:11
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084742
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ *triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ *null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+this.fileSystemID = fileSystemID;
+this.hadoopOpName = hadoopOpName;
+this.clientCorrelationID =
validateClientCorrelationID(clientCorrelationID);
+streamID = EMPTY_STRING;
+retryCount = 0;
+primaryRequestID = EMPTY_STRING;
+format = tracingHeaderFormat;
+this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+listener);
+primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+}
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+this.fileSystemID = originalTracingContext.fileSystemID;
+this.streamID = originalTracingContext.streamID;
+this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+this.hadoopOpName =
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084742
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ *triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ *null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+this.fileSystemID = fileSystemID;
+this.hadoopOpName = hadoopOpName;
+this.clientCorrelationID =
validateClientCorrelationID(clientCorrelationID);
+streamID = EMPTY_STRING;
+retryCount = 0;
+primaryRequestID = EMPTY_STRING;
+format = tracingHeaderFormat;
+this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+listener);
+primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+}
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+this.fileSystemID = originalTracingContext.fileSystemID;
+this.streamID = originalTracingContext.streamID;
+this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+this.hadoopOpName = originalTracingContext.hadoopOpName;
+this.retryCount = 0;
+this.primaryRequestID = originalTracingContext.primaryRequestID;
+this.format = originalTracingContext.format;
+if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+}
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+if ((clientCorrelationID.length() >
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084530
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
Review comment:
primaryRequestId is applicable for any method call that triggers more
than one http request. For example, methods using continuation logic like
listStatus and rename
Have added comments explaining use of tracingContext and its members at the
beginning of file
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615801
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:10
Start Date: 28/Jun/21 20:10
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084305
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -221,17 +226,27 @@ private void completeExecute() throws
AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+tracingContext.generateClientRequestId();
+httpOperation.getConnection()
+.setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+tracingContext.constructHeader());
+ }
+
/**
* Executes a single HTTP operation to complete the REST operation. If it
* fails, there may be a retry. The retryCount is incremented with each
* attempt.
*/
- private boolean executeHttpOperation(final int retryCount) throws
AzureBlobFileSystemException {
+ private boolean executeHttpOperation(final int retryCount,
+TracingContext tracingContext) throws AzureBlobFileSystemException {
AbfsHttpOperation httpOperation = null;
try {
// initialize the HTTP request and open the connection
httpOperation = new AbfsHttpOperation(url, method, requestHeaders);
incrementCounter(AbfsStatistic.CONNECTIONS_MADE, 1);
+ updateClientRequestHeader(httpOperation, tracingContext);
Review comment:
done
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Listener.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+
+/**
+ * Interface for testing identifiers tracked via TracingContext
+ * Implemented in TracingHeaderValidator
+ */
+
+public interface Listener {
Review comment:
yes, it's an interface to trigger header tests through callback when
header is constructed. The tests are run only when listener is registered (not
null), which is done across existing tests for different methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615801)
Time Spent: 6h 50m (was: 6h 40m)
> ABFS: Add Identifiers to Client Request Header
> --
>
> Key: HADOOP-17290
> URL: https://issues.apache.org/jira/browse/HADOOP-17290
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Adding unique values to the client request header to assist in correlating
> requests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615798
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:10
Start Date: 28/Jun/21 20:10
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083974
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+this.outputStreamId = getOutputStreamId();
+this.tracingContext = new TracingContext(tracingContext);
+this.tracingContext.setStreamID(outputStreamId);
+this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
done
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -385,7 +412,9 @@ private void writeAppendBlobCurrentBufferToService() throws
IOException {
"writeCurrentBufferToService", "append")) {
AppendRequestParameters reqParams = new AppendRequestParameters(offset,
0,
bytesLength, APPEND_MODE, true, leaseId);
- AbfsRestOperation op = client.append(path, bytes, reqParams,
cachedSasToken.get());
+ AbfsRestOperation op = client
+ .append(path, bytes, reqParams, cachedSasToken.get(),
+ new TracingContext(tracingContext));
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615798)
Time Spent: 6.5h (was: 6h 20m)
> ABFS: Add Identifiers to Client Request Header
> --
>
> Key: HADOOP-17290
> URL: https://issues.apache.org/jira/browse/HADOOP-17290
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Adding unique values to the client request header to assist in correlating
> requests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615799
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:10
Start Date: 28/Jun/21 20:10
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084233
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -221,17 +226,27 @@ private void completeExecute() throws
AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+tracingContext.generateClientRequestId();
+httpOperation.getConnection()
+.setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+tracingContext.constructHeader());
Review comment:
- constructHeader() may be called simply to display IDs, in which case
it should not regenerate new clientReqId each time
- callee setting it would mean adding code to all methods to call this, and
retry case would not be handled
Can be done internally within TC: have a generateClientRequestId function to
create guid, and call it in constructor as well as the setRetryCount which is
invoked per retry. However, had avoided this since we are populating ID
variables only as and when they pass through the corresponding ABFS layers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615799)
Time Spent: 6h 40m (was: 6.5h)
> ABFS: Add Identifiers to Client Request Header
> --
>
> Key: HADOOP-17290
> URL: https://issues.apache.org/jira/browse/HADOOP-17290
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Adding unique values to the client request header to assist in correlating
> requests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[
https://issues.apache.org/jira/browse/HADOOP-17290?focusedWorklogId=615797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615797
]
ASF GitHub Bot logged work on HADOOP-17290:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 20:10
Start Date: 28/Jun/21 20:10
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083909
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java
##
@@ -114,13 +119,15 @@ public AbfsLease(AbfsClient client, String path, int
acquireMaxRetries,
LOG.debug("Acquired lease {} on {}", leaseID, path);
}
- private void acquireLease(RetryPolicy retryPolicy, int numRetries, int
retryInterval, long delay)
+ private void acquireLease(RetryPolicy retryPolicy, int numRetries,
+ int retryInterval, long delay)
throws LeaseException {
LOG.debug("Attempting to acquire lease on {}, retry {}", path, numRetries);
if (future != null && !future.isDone()) {
throw new LeaseException(ERR_LEASE_FUTURE_EXISTS);
}
-future = client.schedule(() -> client.acquireLease(path,
INFINITE_LEASE_DURATION),
+future = client.schedule(() -> client.acquireLease(path,
+INFINITE_LEASE_DURATION, new TracingContext(tracingContext)),
Review comment:
not needed; moved to calling method (constructor) where it is cloned
only once, instead of per retry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615797)
Time Spent: 6h 20m (was: 6h 10m)
> ABFS: Add Identifiers to Client Request Header
> --
>
> Key: HADOOP-17290
> URL: https://issues.apache.org/jira/browse/HADOOP-17290
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Adding unique values to the client request header to assist in correlating
> requests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084305
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -221,17 +226,27 @@ private void completeExecute() throws
AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+tracingContext.generateClientRequestId();
+httpOperation.getConnection()
+.setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+tracingContext.constructHeader());
+ }
+
/**
* Executes a single HTTP operation to complete the REST operation. If it
* fails, there may be a retry. The retryCount is incremented with each
* attempt.
*/
- private boolean executeHttpOperation(final int retryCount) throws
AzureBlobFileSystemException {
+ private boolean executeHttpOperation(final int retryCount,
+TracingContext tracingContext) throws AzureBlobFileSystemException {
AbfsHttpOperation httpOperation = null;
try {
// initialize the HTTP request and open the connection
httpOperation = new AbfsHttpOperation(url, method, requestHeaders);
incrementCounter(AbfsStatistic.CONNECTIONS_MADE, 1);
+ updateClientRequestHeader(httpOperation, tracingContext);
Review comment:
done
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Listener.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+
+/**
+ * Interface for testing identifiers tracked via TracingContext
+ * Implemented in TracingHeaderValidator
+ */
+
+public interface Listener {
Review comment:
yes, it's an interface to trigger header tests through callback when
header is constructed. The tests are run only when listener is registered (not
null), which is done across existing tests for different methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084233
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -221,17 +226,27 @@ private void completeExecute() throws
AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+tracingContext.generateClientRequestId();
+httpOperation.getConnection()
+.setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+tracingContext.constructHeader());
Review comment:
- constructHeader() may be called simply to display IDs, in which case
it should not regenerate new clientReqId each time
- callee setting it would mean adding code to all methods to call this, and
retry case would not be handled
Can be done internally within TC: have a generateClientRequestId function to
create guid, and call it in constructor as well as the setRetryCount which is
invoked per retry. However, had avoided this since we are populating ID
variables only as and when they pass through the corresponding ABFS layers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083974
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+this.outputStreamId = getOutputStreamId();
+this.tracingContext = new TracingContext(tracingContext);
+this.tracingContext.setStreamID(outputStreamId);
+this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
done
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -385,7 +412,9 @@ private void writeAppendBlobCurrentBufferToService() throws
IOException {
"writeCurrentBufferToService", "append")) {
AppendRequestParameters reqParams = new AppendRequestParameters(offset,
0,
bytesLength, APPEND_MODE, true, leaseId);
- AbfsRestOperation op = client.append(path, bytes, reqParams,
cachedSasToken.get());
+ AbfsRestOperation op = client
+ .append(path, bytes, reqParams, cachedSasToken.get(),
+ new TracingContext(tracingContext));
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083909
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java
##
@@ -114,13 +119,15 @@ public AbfsLease(AbfsClient client, String path, int
acquireMaxRetries,
LOG.debug("Acquired lease {} on {}", leaseID, path);
}
- private void acquireLease(RetryPolicy retryPolicy, int numRetries, int
retryInterval, long delay)
+ private void acquireLease(RetryPolicy retryPolicy, int numRetries,
+ int retryInterval, long delay)
throws LeaseException {
LOG.debug("Attempting to acquire lease on {}, retry {}", path, numRetries);
if (future != null && !future.isDone()) {
throw new LeaseException(ERR_LEASE_FUTURE_EXISTS);
}
-future = client.schedule(() -> client.acquireLease(path,
INFINITE_LEASE_DURATION),
+future = client.schedule(() -> client.acquireLease(path,
+INFINITE_LEASE_DURATION, new TracingContext(tracingContext)),
Review comment:
not needed; moved to calling method (constructor) where it is cloned
only once, instead of per retry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3150: HDFS-16092. Avoid creating LayoutFlags redundant objects
hadoop-yetus commented on pull request #3150:
URL: https://github.com/apache/hadoop/pull/3150#issuecomment-869996968
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 56s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 210m 7s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 0m 27s | | ASF License check generated no
output? |
| | | 302m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockRecovery2 |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.TestListOpenFiles |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3150 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7ece09deaa88 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ab0dd989c35136c3b7a025b64a1ef2bf1cf1818e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #3023: HDFS-16028. Add a configuration item for special trash dir
hadoop-yetus commented on pull request #3023:
URL: https://github.com/apache/hadoop/pull/3023#issuecomment-869984905
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 8s | | trunk passed |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 12s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 3s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 3s | | the patch passed |
| +1 :green_heart: | compile | 18m 5s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 8s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3023/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 1 new + 225
unchanged - 0 fixed = 226 total (was 225) |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 1s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 176m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3023/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3023 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux c797495d90fe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 48a439494ba7ca181237e0271f41b28ef477683b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3023/1/testReport/ |
| Max. process+thread count | 1266 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3023/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically
[GitHub] [hadoop] hadoop-yetus commented on pull request #2225: HDFS-15329. Provide FileContext based ViewFSOverloadScheme implementation
hadoop-yetus commented on pull request #2225:
URL: https://github.com/apache/hadoop/pull/2225#issuecomment-869944467
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 10s | | trunk passed |
| +1 :green_heart: | compile | 20m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 32s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 1m 7s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 59s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 59s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 54s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 54s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 24s | | root: The patch generated
0 new + 51 unchanged - 1 fixed = 51 total (was 52) |
| -1 :x: | mvnsite | 0m 38s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvnsite | 1m 9s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | javadoc | 0m 44s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2225/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2414: HDFS-15650. Make the socket timeout for computing checksum of striped…
hadoop-yetus commented on pull request #2414:
URL: https://github.com/apache/hadoop/pull/2414#issuecomment-869917943
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 5s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 34s | | trunk passed |
| -1 :x: | shadedclient | 23m 1s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 23s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | javadoc | 0m 23s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2414/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 23s |
[jira] [Work logged] (HADOOP-17778) CI for Centos 8
[ https://issues.apache.org/jira/browse/HADOOP-17778?focusedWorklogId=615693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615693 ] ASF GitHub Bot logged work on HADOOP-17778: --- Author: ASF GitHub Bot Created on: 28/Jun/21 17:20 Start Date: 28/Jun/21 17:20 Worklog Time Spent: 10m Work Description: GauthamBanasandra commented on a change in pull request #3151: URL: https://github.com/apache/hadoop/pull/3151#discussion_r659974899 ## File path: hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.cc ## @@ -19,6 +19,7 @@ #include "configuration_test.h" #include "common/configuration.h" #include "common/configuration_loader.h" + Review comment: Yeah, it was just to trigger the tests. But it also looks nice to separate the custom headers from library headers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 615693) Time Spent: 1h 10m (was: 1h) > CI for Centos 8 > --- > > Key: HADOOP-17778 > URL: https://issues.apache.org/jira/browse/HADOOP-17778 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 > Environment: Centos 8 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Need to run CI for Centos 8 platform to ensure that further changes are > stable on this platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #3151: HADOOP-17778. CI for Centos 8
GauthamBanasandra commented on a change in pull request #3151: URL: https://github.com/apache/hadoop/pull/3151#discussion_r659974899 ## File path: hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.cc ## @@ -19,6 +19,7 @@ #include "configuration_test.h" #include "common/configuration.h" #include "common/configuration_loader.h" + Review comment: Yeah, it was just to trigger the tests. But it also looks nice to separate the custom headers from library headers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1422: HDFS-14839: Use Java Concurrent BlockingQueue instead of Internal Blo…
hadoop-yetus commented on pull request #1422: URL: https://github.com/apache/hadoop/pull/1422#issuecomment-869850514 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 26s | | https://github.com/apache/hadoop/pull/1422 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1422 | | JIRA Issue | HDFS-14839 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1422/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16680) Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider
[ https://issues.apache.org/jira/browse/HADOOP-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-16680: Labels: pull-request-available (was: ) > Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider > --- > > Key: HADOOP-16680 > URL: https://issues.apache.org/jira/browse/HADOOP-16680 > Project: Hadoop Common > Issue Type: New Feature > Components: auth, common, security >Affects Versions: 3.3.0 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add MicrosoftGraphGroupsMapping that uses > https://developer.microsoft.com/en-us/graph to retrieve user groups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1690: HADOOP-16680. Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider
hadoop-yetus commented on pull request #1690: URL: https://github.com/apache/hadoop/pull/1690#issuecomment-869850376 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 16s | | https://github.com/apache/hadoop/pull/1690 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1690 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1690/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16680) Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider
[ https://issues.apache.org/jira/browse/HADOOP-16680?focusedWorklogId=615684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615684 ] ASF GitHub Bot logged work on HADOOP-16680: --- Author: ASF GitHub Bot Created on: 28/Jun/21 16:58 Start Date: 28/Jun/21 16:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #1690: URL: https://github.com/apache/hadoop/pull/1690#issuecomment-869850376 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 16s | | https://github.com/apache/hadoop/pull/1690 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1690 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1690/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 615684) Remaining Estimate: 0h Time Spent: 10m > Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider > --- > > Key: HADOOP-16680 > URL: https://issues.apache.org/jira/browse/HADOOP-16680 > Project: Hadoop Common > Issue Type: New Feature > Components: auth, common, security >Affects Versions: 3.3.0 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Add MicrosoftGraphGroupsMapping that uses > https://developer.microsoft.com/en-us/graph to retrieve user groups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17778) CI for Centos 8
[
https://issues.apache.org/jira/browse/HADOOP-17778?focusedWorklogId=615663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615663
]
ASF GitHub Bot logged work on HADOOP-17778:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 16:02
Start Date: 28/Jun/21 16:02
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#discussion_r659916854
##
File path:
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.cc
##
@@ -19,6 +19,7 @@
#include "configuration_test.h"
#include "common/configuration.h"
#include "common/configuration_loader.h"
+
Review comment:
Is this just to trigger the tests? Undo?
##
File path: dev-support/Jenkinsfile
##
@@ -56,6 +56,34 @@ pipeline {
}
}
+// This is an optional stage which runs only when there's a change in
+// C++/C++ build/platform.
+// This stage serves as a means of cross platform validation, which is
+// really needed to ensure that any C++ related/platform change doesn't
+// break the Hadoop build on Centos 8.
+stage ('precommit-run Centos 8') {
+environment {
+DOCKERFILE =
"${SOURCEDIR}/dev-support/docker/Dockerfile_centos_8"
+IS_OPTIONAL = 1
+}
+
+steps {
+withCredentials(
+[usernamePassword(credentialsId:
'apache-hadoop-at-github.com',
Review comment:
This is now repeating 3 times in this file, can we reuse?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615663)
Time Spent: 1h (was: 50m)
> CI for Centos 8
> ---
>
> Key: HADOOP-17778
> URL: https://issues.apache.org/jira/browse/HADOOP-17778
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.4.0
> Environment: Centos 8
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to run CI for Centos 8 platform to ensure that further changes are
> stable on this platform.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3151: HADOOP-17778. CI for Centos 8
goiri commented on a change in pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#discussion_r659916854
##
File path:
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.cc
##
@@ -19,6 +19,7 @@
#include "configuration_test.h"
#include "common/configuration.h"
#include "common/configuration_loader.h"
+
Review comment:
Is this just to trigger the tests? Undo?
##
File path: dev-support/Jenkinsfile
##
@@ -56,6 +56,34 @@ pipeline {
}
}
+// This is an optional stage which runs only when there's a change in
+// C++/C++ build/platform.
+// This stage serves as a means of cross platform validation, which is
+// really needed to ensure that any C++ related/platform change doesn't
+// break the Hadoop build on Centos 8.
+stage ('precommit-run Centos 8') {
+environment {
+DOCKERFILE =
"${SOURCEDIR}/dev-support/docker/Dockerfile_centos_8"
+IS_OPTIONAL = 1
+}
+
+steps {
+withCredentials(
+[usernamePassword(credentialsId:
'apache-hadoop-at-github.com',
Review comment:
This is now repeating 3 times in this file, can we reuse?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3150: HDFS-16092. Avoid creating LayoutFlags redundant objects
hadoop-yetus commented on pull request #3150:
URL: https://github.com/apache/hadoop/pull/3150#issuecomment-869749065
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 392m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 490m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestBlockTokenWrappingQOP |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.mover.TestMover |
| |
hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3150 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6bfa681cd8c0 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ab0dd989c35136c3b7a025b64a1ef2bf1cf1818e |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
hadoop-yetus commented on pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#issuecomment-869728986
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 58s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 240m 44s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 324m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3148/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3148 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6ce88f807485 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3cfc29998e305c5ce60bf11cad8fb42a04cf03ea |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3148/3/testReport/ |
| Max. process+thread count | 3369 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3148/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about
[GitHub] [hadoop] hadoop-yetus commented on pull request #3150: HDFS-16092. Avoid creating LayoutFlags redundant objects
hadoop-yetus commented on pull request #3150:
URL: https://github.com/apache/hadoop/pull/3150#issuecomment-869691737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 450m 25s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 537m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.TestViewDistributedFileSystemContract |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3150/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3150 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ab4aeee3e28a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ab0dd989c35136c3b7a025b64a1ef2bf1cf1818e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3154: YARN-10835. Pass user defined variables of yarn.nodemanager.env-whitelist along jobs
hadoop-yetus commented on pull request #3154:
URL: https://github.com/apache/hadoop/pull/3154#issuecomment-869657875
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 7s | | trunk passed |
| +1 :green_heart: | compile | 20m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 11s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 20m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 14s | | the patch passed |
| +1 :green_heart: | compile | 18m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 27m 34s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 8m 38s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 219m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3154 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux af1ca7a11478 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f43d9550c05d835dbc78234a075f2b05e44af17 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
. |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[jira] [Work logged] (HADOOP-17765) ABFS: Use Unique File Paths in Tests
[ https://issues.apache.org/jira/browse/HADOOP-17765?focusedWorklogId=615570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615570 ] ASF GitHub Bot logged work on HADOOP-17765: --- Author: ASF GitHub Bot Created on: 28/Jun/21 11:43 Start Date: 28/Jun/21 11:43 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #3153: URL: https://github.com/apache/hadoop/pull/3153#issuecomment-869613676 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` HNS OAuth [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:182->twoWriters:150 » TestTimedOut [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 1, Skipped: 52 AppendBlob HNS OAuth [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 2, Errors: 1, Skipped: 98 [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 1, Skipped: 76 HNS SharedKey [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Tests run: 556, Failures: 1, Errors: 0, Skipped: 62 [ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 2, Skipped: 40 NonHNS SharedKey [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Tests run: 556, Failures: 1, Errors: 0, Skipped: 276 [ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 2, Skipped: 40 ``` Existing JIRAs to track failures: [AppendBlob StreamOps test](https://issues.apache.org/jira/browse/HADOOP-17716), [LMT](https://issues.apache.org/jira/browse/HADOOP-16752), [ReadWriteandSeek](https://issues.apache.org/jira/browse/HADOOP-15702), [testDistCpWithIterator](https://issues.apache.org/jira/browse/HADOOP-17628) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 615570) Time Spent: 1h 20m (was: 1h 10m) > ABFS: Use Unique File Paths in Tests > > > Key: HADOOP-17765 > URL: https://issues.apache.org/jira/browse/HADOOP-17765 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Many of ABFS driver tests use common names for file paths (e.g., > "/testfile"). This poses a risk of errors during parallel test runs when > static variables (such as those for monitoring stats) affected by file paths > are introduced. > Using unique test file names will avoid possible errors
[GitHub] [hadoop] sumangala-patki commented on pull request #3153: HADOOP-17765. ABFS: Use Unique File Paths in Tests
sumangala-patki commented on pull request #3153: URL: https://github.com/apache/hadoop/pull/3153#issuecomment-869613676 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` HNS OAuth [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:182->twoWriters:150 » TestTimedOut [ERROR] Tests run: 556, Failures: 1, Errors: 1, Skipped: 98 [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 1, Skipped: 52 AppendBlob HNS OAuth [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:173->twoWriters:161 » IO [ERROR] Tests run: 556, Failures: 2, Errors: 1, Skipped: 98 [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 1, Skipped: 76 HNS SharedKey [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Tests run: 556, Failures: 1, Errors: 0, Skipped: 62 [ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 2, Skipped: 40 NonHNS SharedKey [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0 [ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:143->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime [ERROR] Tests run: 556, Failures: 1, Errors: 0, Skipped: 276 [ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut [ERROR] Tests run: 265, Failures: 0, Errors: 2, Skipped: 40 ``` Existing JIRAs to track failures: [AppendBlob StreamOps test](https://issues.apache.org/jira/browse/HADOOP-17716), [LMT](https://issues.apache.org/jira/browse/HADOOP-16752), [ReadWriteandSeek](https://issues.apache.org/jira/browse/HADOOP-15702), [testDistCpWithIterator](https://issues.apache.org/jira/browse/HADOOP-17628) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17772) wasb: delete() should have timeout option
[ https://issues.apache.org/jira/browse/HADOOP-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370548#comment-17370548 ] Steve Loughran commented on HADOOP-17772: - ok. So that's not an abfs issue; moving out and changing title. Can you try using the abfs:// connector against the same store? > wasb: delete() should have timeout option > - > > Key: HADOOP-17772 > URL: https://issues.apache.org/jira/browse/HADOOP-17772 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Zhuangyu Han >Priority: Major > > The delete() API in AzureBlobFileSystem could potentially stuck when trying > to delete a infinitely lease blob file/directory. We hope that there is a > timeout option for this API and the delete() could throw an timeoutException > when specified timeout limit is reached. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17772) wasb: delete() should have timeout option
[ https://issues.apache.org/jira/browse/HADOOP-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17772: Summary: wasb: delete() should have timeout option (was: ABFS: delete() should have timeout option) > wasb: delete() should have timeout option > - > > Key: HADOOP-17772 > URL: https://issues.apache.org/jira/browse/HADOOP-17772 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Zhuangyu Han >Priority: Major > > The delete() API in AzureBlobFileSystem could potentially stuck when trying > to delete a infinitely lease blob file/directory. We hope that there is a > timeout option for this API and the delete() could throw an timeoutException > when specified timeout limit is reached. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17772) wasb: delete() should have timeout option
[ https://issues.apache.org/jira/browse/HADOOP-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17772: Parent: (was: HADOOP-17736) Issue Type: Bug (was: Sub-task) > wasb: delete() should have timeout option > - > > Key: HADOOP-17772 > URL: https://issues.apache.org/jira/browse/HADOOP-17772 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Zhuangyu Han >Priority: Major > > The delete() API in AzureBlobFileSystem could potentially stuck when trying > to delete a infinitely lease blob file/directory. We hope that there is a > timeout option for this API and the delete() could throw an timeoutException > when specified timeout limit is reached. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370546#comment-17370546 ] Steve Loughran commented on HADOOP-17028: - bq. Steve, this looks like incompatible change as it replaced the parameter to a different class... In a world of @Functional params, it is compatible. But I suspect that you are right about link-time matching; the JVM probably isn't going to blindly treat them as equivalent. bq. One good thing about it that you can remove the deprecated classes as they have never been used. not used in hadoop-*. My fear was that they had been picked up/used in google GCS. I've been building that connector and I don't see it in use. Given that I'll target it for removal in 3.3.2 bq. You don't seem to care about older versions, but many people do. I really do. I am the one currently staring at the ABFS Code and a branch-2 build. At the same time, being more isolated (and, let's be ruthless: lower risk), the object storage code has been able to evolve faster than other bits of the codebase. FWIW the main troublespot in backporting cloud storage changes, esp in hadoop-azure, is actually mockito versions. something which works on mockito 2 is still likely to compile on mockito 1.x but then fail with some "impossible" stack trace, and I'm left trying to distinguish between "mockito lib version issues, fix test or cut", "different codepath breaks mockito test" and "test has actually found a regression". It's a key reason why I don't like mockito-based testing. > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17769) Upgrade JUnit to 4.13.2
[
https://issues.apache.org/jira/browse/HADOOP-17769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370547#comment-17370547
]
Steve Loughran commented on HADOOP-17769:
-
yes -thank you!
> Upgrade JUnit to 4.13.2
> ---
>
> Key: HADOOP-17769
> URL: https://issues.apache.org/jira/browse/HADOOP-17769
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1, 3.4.0, 2.10.2, 3.2.3
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> JUnit 4.13.1 has a bug that is reported in Junit
> [issue-1652|https://github.com/junit-team/junit4/issues/1652] _Timeout
> ThreadGroups should not be destroyed_
> After upgrading Junit to 4.13.1 in HADOOP-17602, {{TestBlockRecovery}}
> started to fail regularly in branch-3.x and branch-2.10.
> While investigating the failure in branch-2.10 HDFS-16072, I found out that
> the bug is the main reason {{TestBlockRecovery}} started to fail because the
> timeout of the Junit would try to close a ThreadGroup that has been already
> closed which throws the {{java.lang.IllegalThreadStateException}}.
> The bug has been fixed in Junit-4.13.2
> For branch-3.x, HDFS-15940 did not address the root cause of the problem.
> Eventually, Splitting the {{TestBlockRecovery}} hid the bug, but the upgrade
> needs to be done so that the problem does not show up in another unit test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] symious opened a new pull request #3154: YARN-10835. Pass user defined variables of yarn.nodemanager.env-whitelist along jobs
symious opened a new pull request #3154: URL: https://github.com/apache/hadoop/pull/3154 # What changes were proposed in this pull request? The configuration of "yarn.nodemanager.env-whitelist" has been helpful in running multi version Hadoop on Yarn. But the pain point is that users need to use different configurations to run jobs of different versions of Hadoop. The first idea is to have SRE prepared different versions of configurations and let users specify different environment variables of "HADOOP_CONF_DIR", this shifts the burden from users to SRE that they have to maintain different versions of configurations with only one or two configurations changed. This ticket proposes to export the value of the environment variable in "yarn.nodemanager.env-whitelist" via the environment variable or Java properties of the client submitting jobs. So that SRE only has to maintain one version of configuration and users can easily define Hadoop version with one export in an environment variable or Java property. # What is the link to the Apache JIRA https://issues.apache.org/jira/browse/YARN-10835 # How was this patch tested? Add unit test to pass Java properties to `ContainerLaunchContext`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17764) S3AInputStream read does not re-open the input stream on the second read retry attempt
[
https://issues.apache.org/jira/browse/HADOOP-17764?focusedWorklogId=615506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615506
]
ASF GitHub Bot logged work on HADOOP-17764:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 09:03
Start Date: 28/Jun/21 09:03
Worklog Time Spent: 10m
Work Description: majdyz closed pull request #3132:
URL: https://github.com/apache/hadoop/pull/3132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615506)
Time Spent: 8h 50m (was: 8h 40m)
> S3AInputStream read does not re-open the input stream on the second read
> retry attempt
> --
>
> Key: HADOOP-17764
> URL: https://issues.apache.org/jira/browse/HADOOP-17764
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Assignee: Zamil Majdy
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> *Bug description:*
> The read method in S3AInputStream has this following behaviour when an
> IOException happening during the read:
> * {{reopen and read quickly}}: The client after failing in the first attempt
> of {{read}}, will reopen the stream and try reading again without {{sleep}}.
> * {{reopen and wait for fixed duration}}: The client after failing in the
> attempt of {{read}}, will reopen the stream, sleep for
> {{fs.s3a.retry.interval}} milliseconds (defaults to 500 ms), and then try
> reading from the stream.
> While doing the {{reopen and read quickly}} process, the subsequent read will
> be retried without reopening the input stream in case of the second failure
> happened. This leads to some of the bytes read being skipped which results to
> corrupt/less data than required.
>
> *Scenario to reproduce:*
> * Execute S3AInputStream `read()` or `read(b, off, len)`.
> * The read failed and throws `Connection Reset` exception after reading some
> data.
> * The InputStream is re-opened and another `read()` or `read(b, off, len)`
> is executed
> * The read failed for the second time and throws `Connection Reset`
> exception after reading some data.
> * The InputStream is not re-opened and another `read()` or `read(b, off,
> len)` is executed after sleep
> * The read succeed, but it skips the first few bytes that has already been
> read on the second failure.
>
> *Proposed fix:*
> [https://github.com/apache/hadoop/pull/3109]
> Added the test that reproduces the issue along with the fix
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] majdyz closed pull request #3132: HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt
majdyz closed pull request #3132: URL: https://github.com/apache/hadoop/pull/3132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
virajjasani commented on a change in pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#discussion_r659601136
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -2272,19 +2274,11 @@ public int getActiveTransferThreadCount() {
void incrDatanodeNetworkErrors(String host) {
metrics.incrDatanodeNetworkErrors();
-/*
- * Synchronizing on the whole cache is a big hammer, but since it's only
- * accumulating errors, it should be ok. If this is ever expanded to
include
- * non-error stats, then finer-grained concurrency should be applied.
- */
-synchronized (datanodeNetworkCounts) {
- try {
-final Map curCount = datanodeNetworkCounts.get(host);
-curCount.put("networkErrors", curCount.get("networkErrors") + 1L);
-datanodeNetworkCounts.put(host, curCount);
- } catch (ExecutionException e) {
-LOG.warn("failed to increment network error counts for host: {}",
host);
- }
+try {
+ datanodeNetworkCounts.get(host).compute(NETWORK_ERRORS,
+ (key, errors) -> errors == null ? null : errors + 1L);
Review comment:
> i mean, shouldn't it be made 1 when errors is null (meaning the key
didn't exist before)?
I see. Based on the LoadingCache creation, we will always find value `0L` at
the beginning and the only reason why I handled `errors == null` case is
because findbugs don't complain about missing this. But I think your suggestion
is better, we should return error `1L` when it is null. Let me change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
virajjasani commented on a change in pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#discussion_r659596959
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -2272,19 +2274,11 @@ public int getActiveTransferThreadCount() {
void incrDatanodeNetworkErrors(String host) {
metrics.incrDatanodeNetworkErrors();
-/*
- * Synchronizing on the whole cache is a big hammer, but since it's only
- * accumulating errors, it should be ok. If this is ever expanded to
include
- * non-error stats, then finer-grained concurrency should be applied.
- */
-synchronized (datanodeNetworkCounts) {
- try {
-final Map curCount = datanodeNetworkCounts.get(host);
-curCount.put("networkErrors", curCount.get("networkErrors") + 1L);
-datanodeNetworkCounts.put(host, curCount);
- } catch (ExecutionException e) {
-LOG.warn("failed to increment network error counts for host: {}",
host);
- }
+try {
+ datanodeNetworkCounts.get(host).compute(NETWORK_ERRORS,
+ (key, errors) -> errors == null ? null : errors + 1L);
Review comment:
So everytime we have a network error, instead of locking entire
LoadingCache, with CHM.compute(), we will just take lock on bucket of Map where
the key resides and then error count will be incremented. So this is fine
grained locking and much performant than taking lock on entire `LoadingCache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
virajjasani commented on a change in pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#discussion_r659595214
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -2272,19 +2274,11 @@ public int getActiveTransferThreadCount() {
void incrDatanodeNetworkErrors(String host) {
metrics.incrDatanodeNetworkErrors();
-/*
- * Synchronizing on the whole cache is a big hammer, but since it's only
- * accumulating errors, it should be ok. If this is ever expanded to
include
- * non-error stats, then finer-grained concurrency should be applied.
- */
-synchronized (datanodeNetworkCounts) {
- try {
-final Map curCount = datanodeNetworkCounts.get(host);
-curCount.put("networkErrors", curCount.get("networkErrors") + 1L);
-datanodeNetworkCounts.put(host, curCount);
- } catch (ExecutionException e) {
-LOG.warn("failed to increment network error counts for host: {}",
host);
- }
+try {
+ datanodeNetworkCounts.get(host).compute(NETWORK_ERRORS,
+ (key, errors) -> errors == null ? null : errors + 1L);
Review comment:
Map.compute() is just replacement of below code (and ConcurrentHashMap
does it atomically):
```
* {@code
* V oldValue = map.get(key);
* V newValue = remappingFunction.apply(key, oldValue);
* if (oldValue != null ) {
*if (newValue != null)
* map.put(key, newValue);
*else
* map.remove(key);
* } else {
*if (newValue != null)
* map.put(key, newValue);
*else
* return null;
* }
* }
```
errors will ideally never be null because it is defined as `0L` here:
```
datanodeNetworkCounts =
CacheBuilder.newBuilder()
.maximumSize(dncCacheMaxSize)
.build(new CacheLoader>() {
@Override
public Map load(String key) throws Exception {
final Map ret = new HashMap();
ret.put("networkErrors", 0L);
return ret;
}
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
jojochuang commented on a change in pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#discussion_r659593859
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -2272,19 +2274,11 @@ public int getActiveTransferThreadCount() {
void incrDatanodeNetworkErrors(String host) {
metrics.incrDatanodeNetworkErrors();
-/*
- * Synchronizing on the whole cache is a big hammer, but since it's only
- * accumulating errors, it should be ok. If this is ever expanded to
include
- * non-error stats, then finer-grained concurrency should be applied.
- */
-synchronized (datanodeNetworkCounts) {
- try {
-final Map curCount = datanodeNetworkCounts.get(host);
-curCount.put("networkErrors", curCount.get("networkErrors") + 1L);
-datanodeNetworkCounts.put(host, curCount);
- } catch (ExecutionException e) {
-LOG.warn("failed to increment network error counts for host: {}",
host);
- }
+try {
+ datanodeNetworkCounts.get(host).compute(NETWORK_ERRORS,
+ (key, errors) -> errors == null ? null : errors + 1L);
Review comment:
i mean, shouldn't it be made 1 when errors is null (meaning the key
didn't exist before)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3148: HDFS-16090. Fine grained lock for datanodeNetworkCounts
jojochuang commented on a change in pull request #3148:
URL: https://github.com/apache/hadoop/pull/3148#discussion_r659593248
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -2272,19 +2274,11 @@ public int getActiveTransferThreadCount() {
void incrDatanodeNetworkErrors(String host) {
metrics.incrDatanodeNetworkErrors();
-/*
- * Synchronizing on the whole cache is a big hammer, but since it's only
- * accumulating errors, it should be ok. If this is ever expanded to
include
- * non-error stats, then finer-grained concurrency should be applied.
- */
-synchronized (datanodeNetworkCounts) {
- try {
-final Map curCount = datanodeNetworkCounts.get(host);
-curCount.put("networkErrors", curCount.get("networkErrors") + 1L);
-datanodeNetworkCounts.put(host, curCount);
- } catch (ExecutionException e) {
-LOG.warn("failed to increment network error counts for host: {}",
host);
- }
+try {
+ datanodeNetworkCounts.get(host).compute(NETWORK_ERRORS,
+ (key, errors) -> errors == null ? null : errors + 1L);
Review comment:
not familiar with ConcurrentHashMap.compute() but wouldn't this make
errors null always?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3146: HDFS-16089. EC: Add metric EcReconstructionValidateTimeMillis for Striped…
tomscut commented on pull request #3146: URL: https://github.com/apache/hadoop/pull/3146#issuecomment-869478686 Thanks @jojochuang for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17778) CI for Centos 8
[
https://issues.apache.org/jira/browse/HADOOP-17778?focusedWorklogId=615490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615490
]
ASF GitHub Bot logged work on HADOOP-17778:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 08:17
Start Date: 28/Jun/21 08:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869475751
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 53s | | trunk passed |
| +1 :green_heart: | compile | 2m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 2m 34s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 2m 34s | | the patch passed |
| +1 :green_heart: | golang | 2m 34s | | the patch passed |
| +1 :green_heart: | javac | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 2m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 2m 40s | | the patch passed |
| +1 :green_heart: | golang | 2m 40s | | the patch passed |
| +1 :green_heart: | javac | 2m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | hadolint | 0m 2s | | No new issues. |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 12m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 45s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 92m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3151 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux e13614ad8474 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8dd54013d49993bc4595efc41b56c0ba88363e4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/testReport/ |
| Max. process+thread count | 547 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe,
[GitHub] [hadoop] hadoop-yetus commented on pull request #3151: HADOOP-17778. CI for Centos 8
hadoop-yetus commented on pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869475751
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 53s | | trunk passed |
| +1 :green_heart: | compile | 2m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 2m 34s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 2m 34s | | the patch passed |
| +1 :green_heart: | golang | 2m 34s | | the patch passed |
| +1 :green_heart: | javac | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 2m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 2m 40s | | the patch passed |
| +1 :green_heart: | golang | 2m 40s | | the patch passed |
| +1 :green_heart: | javac | 2m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | hadolint | 0m 2s | | No new issues. |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 12m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 45s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 92m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3151 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux e13614ad8474 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8dd54013d49993bc4595efc41b56c0ba88363e4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/testReport/ |
| Max. process+thread count | 547 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3146: HDFS-16089. EC: Add metric EcReconstructionValidateTimeMillis for Striped…
tomscut commented on pull request #3146: URL: https://github.com/apache/hadoop/pull/3146#issuecomment-869468219 Hi @rakeshadr @tasanuma @jojochuang , could you please review the code? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17765) ABFS: Use Unique File Paths in Tests
[
https://issues.apache.org/jira/browse/HADOOP-17765?focusedWorklogId=615470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615470
]
ASF GitHub Bot logged work on HADOOP-17765:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 07:06
Start Date: 28/Jun/21 07:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3153:
URL: https://github.com/apache/hadoop/pull/3153#issuecomment-869417214
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 7 unchanged - 0
fixed = 8 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 79m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3153 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f5bce17dec6f 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ad689edc99e9f25d17faf0df5896d7407ceb12a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/testReport/ |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
[GitHub] [hadoop] hadoop-yetus commented on pull request #3153: HADOOP-17765. ABFS: Use Unique File Paths in Tests
hadoop-yetus commented on pull request #3153:
URL: https://github.com/apache/hadoop/pull/3153#issuecomment-869417214
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 7 unchanged - 0
fixed = 8 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 79m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3153 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f5bce17dec6f 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ad689edc99e9f25d17faf0df5896d7407ceb12a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/testReport/ |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3153/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
[jira] [Work logged] (HADOOP-17778) CI for Centos 8
[ https://issues.apache.org/jira/browse/HADOOP-17778?focusedWorklogId=615467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615467 ] ASF GitHub Bot logged work on HADOOP-17778: --- Author: ASF GitHub Bot Created on: 28/Jun/21 06:45 Start Date: 28/Jun/21 06:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3151: URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869406124 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 615467) Time Spent: 40m (was: 0.5h) > CI for Centos 8 > --- > > Key: HADOOP-17778 > URL: https://issues.apache.org/jira/browse/HADOOP-17778 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 > Environment: Centos 8 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Need to run CI for Centos 8 platform to ensure that further changes are > stable on this platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3151: HADOOP-17778. CI for Centos 8
hadoop-yetus commented on pull request #3151: URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869406124 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17778) CI for Centos 8
[
https://issues.apache.org/jira/browse/HADOOP-17778?focusedWorklogId=615466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-615466
]
ASF GitHub Bot logged work on HADOOP-17778:
---
Author: ASF GitHub Bot
Created on: 28/Jun/21 06:44
Start Date: 28/Jun/21 06:44
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869405936
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 23m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 2m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 2m 30s | | the patch passed |
| +1 :green_heart: | cc | 2m 30s | | the patch passed |
| +1 :green_heart: | golang | 2m 30s | | the patch passed |
| +1 :green_heart: | javac | 2m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 46s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 124m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3151 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs
hadolint compile cc mvnsite javac unit golang |
| uname | Linux b2b124fb2839 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8dd54013d49993bc4595efc41b56c0ba88363e4 |
| Default Java | Red Hat, Inc.-1.8.0_292-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/testReport/ |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 615466)
Time Spent: 0.5h (was: 20m)
> CI for Centos 8
> ---
>
> Key: HADOOP-17778
> URL: https://issues.apache.org/jira/browse/HADOOP-17778
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.4.0
> Environment: Centos 8
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Need to run CI for Centos 8 platform to
[GitHub] [hadoop] hadoop-yetus commented on pull request #3151: HADOOP-17778. CI for Centos 8
hadoop-yetus commented on pull request #3151:
URL: https://github.com/apache/hadoop/pull/3151#issuecomment-869405936
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 23m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 2m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 2m 30s | | the patch passed |
| +1 :green_heart: | cc | 2m 30s | | the patch passed |
| +1 :green_heart: | golang | 2m 30s | | the patch passed |
| +1 :green_heart: | javac | 2m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 46s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 124m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3151 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs
hadolint compile cc mvnsite javac unit golang |
| uname | Linux b2b124fb2839 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8dd54013d49993bc4595efc41b56c0ba88363e4 |
| Default Java | Red Hat, Inc.-1.8.0_292-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/testReport/ |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3151/1/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
