[GitHub] [hadoop] jojochuang commented on a diff in pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
jojochuang commented on code in PR #4107:
URL: https://github.com/apache/hadoop/pull/4107#discussion_r859381403
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -4914,6 +4914,33 @@ int getNumberOfDatanodes(DatanodeReportType type) {
}
}
+ DatanodeInfo[] slowDataNodesReport() throws IOException {
+String operationName = "slowDataNodesReport";
+DatanodeInfo[] datanodeInfos;
+checkSuperuserPrivilege(operationName);
Review Comment:
does it need to require super user privilege?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:
##
@@ -433,7 +433,7 @@ static int run(DistributedFileSystem dfs, String[] argv,
int idx) throws IOExcep
*/
private static final String commonUsageSummary =
"\t[-report [-live] [-dead] [-decommissioning] " +
-"[-enteringmaintenance] [-inmaintenance]]\n" +
+ "[-enteringmaintenance] [-inmaintenance] [-slownodes]]\n" +
Review Comment:
The corresponding documentation needs to update when CLI commands are
added/updated.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:
##
@@ -632,6 +638,20 @@ private static void
printDataNodeReports(DistributedFileSystem dfs,
}
}
+ private static void printSlowDataNodeReports(DistributedFileSystem dfs,
boolean listNodes,
Review Comment:
Can you provide a sample output? It would be confusing, I guess. I suspect
you would need some kind of header to distinguish from the other data node
reports.
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java:
##
@@ -1868,4 +1868,16 @@ BatchedEntries listOpenFiles(long prevId,
*/
@AtMostOnce
void satisfyStoragePolicy(String path) throws IOException;
+
+ /**
+ * Get report on all of the slow Datanodes. Slow running datanodes are
identified based on
+ * the Outlier detection algorithm, if slow peer tracking is enabled for the
DFS cluster.
+ *
+ * @return Datanode report for slow running datanodes.
+ * @throws IOException If an I/O error occurs.
+ */
+ @Idempotent
+ @ReadOnly
+ DatanodeInfo[] getSlowDatanodeReport() throws IOException;
Review Comment:
I just want to check with every one that it is okay to have an array of
objects as the return value.
I think it's fine but just want to check with every one, because once we
decide the the interface it can't be changed later.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:
##
@@ -433,7 +433,7 @@ static int run(DistributedFileSystem dfs, String[] argv,
int idx) throws IOExcep
*/
private static final String commonUsageSummary =
"\t[-report [-live] [-dead] [-decommissioning] " +
-"[-enteringmaintenance] [-inmaintenance]]\n" +
+ "[-enteringmaintenance] [-inmaintenance] [-slownodes]]\n" +
Review Comment:
In fact it would appear confusion to HDFS administrators. These subcommands
are meant to filter the DNs in these states, and "slownodes" is not a defined
DataNode state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] skysiders opened a new pull request, #4236: MAPREDUCE-7375 set permission for JobSubmissionFiles
skysiders opened a new pull request, #4236: URL: https://github.com/apache/hadoop/pull/4236 ### Description of PR JobSubmissionFiles provide getStagingDir to get Staging Directory.If stagingArea missing, method will create new directory with this. fs.mkdirs(stagingArea, new FsPermission(JOB_DIR_PERMISSION)); It seems create new directory with JOB_DIR_PERMISSION,but this permission will be apply by umask.If umask too strict , this permission may be 000(if umask is 700).So we should change permission after create. ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #4213: HDFS-16554 Remove unused configuration dfs.namenode.block.deletion.increment.
Hexiaoqiao commented on PR #4213: URL: https://github.com/apache/hadoop/pull/4213#issuecomment-1110505405 Committed to trunk. Thanks @smarthanwang for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4228: HDFS-16468. Define ssize_t for Windows
hadoop-yetus commented on PR #4228:
URL: https://github.com/apache/hadoop/pull/4228#issuecomment-1110505106
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 53s | | trunk passed |
| +1 :green_heart: | compile | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 68m 5s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 68m 29s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 3m 32s | | the patch passed |
| +1 :green_heart: | cc | 3m 32s | | the patch passed |
| +1 :green_heart: | golang | 3m 32s | | the patch passed |
| +1 :green_heart: | javac | 3m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 113m 39s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 221m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4228 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux d0aa6b9ffd8b 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b26a9a96131cc3112107e471ed0b13e3a29dffd |
| Default Java | Debian-11.0.14+9-post-Debian-1deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/testReport/ |
| Max. process+thread count | 585 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao merged pull request #4213: HDFS-16554 Remove unused configuration dfs.namenode.block.deletion.increment.
Hexiaoqiao merged PR #4213: URL: https://github.com/apache/hadoop/pull/4213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #4230: HDFS-16558 Consider changing the lock of delegation token from write …
Hexiaoqiao commented on PR #4230: URL: https://github.com/apache/hadoop/pull/4230#issuecomment-1110498309 `In a very busy authed cluster, renewing/caneling/getting delegation token get slow and it will slow down the speed of handling rpcs from client. Since AbstractDelegationTokenSecretManager is a thread-safe manager, we propose to change the fs lock from write lock to read lock(protect editlog rolling)` @yuanboliu Thanks for your proposal, it is a great improvement. I think it is proper for ADTS which is thread-safe as you mentioned above. But I am concern if it is also thread-safe for editlog sync. Consider both renew/cancel/get for different token, is it safe to keep the order when replay editlog? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smarthanwang commented on pull request #4213: HDFS-16554 Remove unused configuration dfs.namenode.block.deletion.increment.
smarthanwang commented on PR #4213: URL: https://github.com/apache/hadoop/pull/4213#issuecomment-1110494557 @Hexiaoqiao thanks for review. The failed UTs are not related with this PR which seems failed for timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #4209: HDFS-16550. [SBN read] Improper cache-size for journal node may cause cluster crash
tomscut commented on PR #4209: URL: https://github.com/apache/hadoop/pull/4209#issuecomment-1110446887 Hi @tasanuma @ayushtkn @sunchao @xkrogen , could you please take a look. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4186: HDFS-16528. Reconfigure slow peer enable for Namenode
virajjasani commented on PR #4186: URL: https://github.com/apache/hadoop/pull/4186#issuecomment-1110437828 Thanks for the review @tomscut ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #4201: HDFS-16547. [SBN read] Namenode in safe mode should not be transfered to observer state
tomscut commented on PR #4201: URL: https://github.com/apache/hadoop/pull/4201#issuecomment-1110433872 Hi @sunchao @xkrogen , could you please take a look? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18104) Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads
[
https://issues.apache.org/jira/browse/HADOOP-18104?focusedWorklogId=762634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762634
]
ASF GitHub Bot logged work on HADOOP-18104:
---
Author: ASF GitHub Bot
Created on: 27/Apr/22 00:21
Start Date: 27/Apr/22 00:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #3964:
URL: https://github.com/apache/hadoop/pull/3964#issuecomment-1110371645
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ feature-vectored-io Compile Tests _ |
| +0 :ok: | mvndep | 16m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 1s | | feature-vectored-io
passed |
| +1 :green_heart: | compile | 22m 49s | | feature-vectored-io passed
with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 27s | | feature-vectored-io passed
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | feature-vectored-io
passed |
| +1 :green_heart: | mvnsite | 3m 46s | | feature-vectored-io passed |
| +1 :green_heart: | javadoc | 2m 58s | | feature-vectored-io passed
with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 39s | | feature-vectored-io passed
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 9s | | feature-vectored-io passed
|
| +1 :green_heart: | shadedclient | 22m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 51s | | the patch passed |
| +1 :green_heart: | compile | 22m 4s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 22m 4s | | the patch passed |
| +1 :green_heart: | compile | 20m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 19s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3964/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
|
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 47s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 250m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3964/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3964 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile
javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux e2a0388503cf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-vectored-io /
a64892e65fb5843506c46e8950ad7c93c933b838 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #3964: HADOOP-18104: S3A: Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads
hadoop-yetus commented on PR #3964:
URL: https://github.com/apache/hadoop/pull/3964#issuecomment-1110371645
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ feature-vectored-io Compile Tests _ |
| +0 :ok: | mvndep | 16m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 1s | | feature-vectored-io
passed |
| +1 :green_heart: | compile | 22m 49s | | feature-vectored-io passed
with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 27s | | feature-vectored-io passed
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | feature-vectored-io
passed |
| +1 :green_heart: | mvnsite | 3m 46s | | feature-vectored-io passed |
| +1 :green_heart: | javadoc | 2m 58s | | feature-vectored-io passed
with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 39s | | feature-vectored-io passed
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 9s | | feature-vectored-io passed
|
| +1 :green_heart: | shadedclient | 22m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 51s | | the patch passed |
| +1 :green_heart: | compile | 22m 4s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 22m 4s | | the patch passed |
| +1 :green_heart: | compile | 20m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 19s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3964/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
|
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 47s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 250m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3964/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3964 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile
javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux e2a0388503cf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-vectored-io /
a64892e65fb5843506c46e8950ad7c93c933b838 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3964/8/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
[GitHub] [hadoop] hadoop-yetus commented on pull request #4228: HDFS-16468. Define ssize_t for Windows
hadoop-yetus commented on PR #4228:
URL: https://github.com/apache/hadoop/pull/4228#issuecomment-1110361697
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 25m 32s | | trunk passed |
| +1 :green_heart: | compile | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 11s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 53m 39s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 48s | | the patch passed |
| +1 :green_heart: | cc | 3m 48s | | the patch passed |
| +1 :green_heart: | golang | 3m 48s | | the patch passed |
| +1 :green_heart: | javac | 3m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 111m 6s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 195m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4228 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 3fb06964645e 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b26a9a96131cc3112107e471ed0b13e3a29dffd |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/testReport/ |
| Max. process+thread count | 601 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16965) Introduce StreamContext for Abfs Input and Output streams.
[ https://issues.apache.org/jira/browse/HADOOP-16965?focusedWorklogId=762629=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762629 ] ASF GitHub Bot logged work on HADOOP-16965: --- Author: ASF GitHub Bot Created on: 26/Apr/22 23:49 Start Date: 26/Apr/22 23:49 Worklog Time Spent: 10m Work Description: arjun4084346 commented on PR #4171: URL: https://github.com/apache/hadoop/pull/4171#issuecomment-1110354907 > I ran the integration tests on branch-2.10 and that also has skipped tests. Test results are same for both the branch-2.10 and my PR in terms of tests run, failures, errors, skipped tests. ``` [INFO] - Issue Time Tracking --- Worklog Id: (was: 762629) Time Spent: 3h 20m (was: 3h 10m) > Introduce StreamContext for Abfs Input and Output streams. > -- > > Key: HADOOP-16965 > URL: https://issues.apache.org/jira/browse/HADOOP-16965 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > The number of configuration keeps growing in AbfsOutputStream and > AbfsInputStream as we keep on adding new features. It is time to refactor the > configurations in a separate class like StreamContext and pass them around. > This is will improve the readability of code and reduce cherry-pick-backport > pain. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arjun4084346 commented on pull request #4171: HADOOP-16965. Refactor abfs stream configuration. (#1956)
arjun4084346 commented on PR #4171: URL: https://github.com/apache/hadoop/pull/4171#issuecomment-1110354907 > I ran the integration tests on branch-2.10 and that also has skipped tests. Test results are same for both the branch-2.10 and my PR in terms of tests run, failures, errors, skipped tests. ``` [INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-azure --- [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCreate.testFilterFSWriteAfterClose:182 » IO java.io.Fi... [ERROR] ITestAzureBlobFileSystemE2E.testFlushWithFileNotFoundException:224 » IO java.i... [ERROR] ITestAzureBlobFileSystemE2E.testWriteWithFileNotFoundException:204 » IO java.i... [INFO] [ERROR] Tests run: 413, Failures: 0, Errors: 3, Skipped: 253 [INFO] --- maven-failsafe-plugin:2.21.0:integration-test (integration-test-abfs-parallel-classes) @ hadoop-azure --- [WARNING] Tests run: 151, Failures: 0, Errors: 0, Skipped: 24 ``` It's the same and behavior doesn't change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4228: HDFS-16468. Define ssize_t for Windows
goiri commented on code in PR #4228: URL: https://github.com/apache/hadoop/pull/4228#discussion_r859202738 ## hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform/types.h: ## @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef NATIVE_LIBHDFSPP_LIB_CROSS_PLATFORM_TYPES +#define NATIVE_LIBHDFSPP_LIB_CROSS_PLATFORM_TYPES + +#if _WIN32 || _WIN64 Review Comment: It might be more readable to do something like: ``` #if _WIN64 typedef long int ssize_t; #elif _WIN32 typedef int ssize_t; #else #include #endif ``` With the comments and so on obviously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4228: HDFS-16468. Define ssize_t for Windows
goiri commented on code in PR #4228:
URL: https://github.com/apache/hadoop/pull/4228#discussion_r859200993
##
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/cat.cc:
##
@@ -62,7 +62,6 @@ int main(int argc, char *argv[]) {
//wrapping file_raw into a unique pointer to guarantee deletion
std::unique_ptr file(file_raw);
- ssize_t total_bytes_read = 0;
Review Comment:
I would prefer to do the cleanup of this unused variables (including the one
in the tools) separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18187) Convert s3a prefetching to use JavaDoc for fields and enums
[ https://issues.apache.org/jira/browse/HADOOP-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528442#comment-17528442 ] Bhalchandra Pandit commented on HADOOP-18187: - [~dannycjones] do you have a reference source file whose comment style I can use? > Convert s3a prefetching to use JavaDoc for fields and enums > --- > > Key: HADOOP-18187 > URL: https://issues.apache.org/jira/browse/HADOOP-18187 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Daniel Carl Jones >Assignee: Bhalchandra Pandit >Priority: Minor > > There's lots of good comments for fields and enum values in the current code. > Let's expose these to your IDE with JavaDoc instead. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18187) Convert s3a prefetching to use JavaDoc for fields and enums
[ https://issues.apache.org/jira/browse/HADOOP-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhalchandra Pandit reassigned HADOOP-18187: --- Assignee: Bhalchandra Pandit > Convert s3a prefetching to use JavaDoc for fields and enums > --- > > Key: HADOOP-18187 > URL: https://issues.apache.org/jira/browse/HADOOP-18187 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Daniel Carl Jones >Assignee: Bhalchandra Pandit >Priority: Minor > > There's lots of good comments for fields and enum values in the current code. > Let's expose these to your IDE with JavaDoc instead. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18182) S3File to store reference to active S3Object in a field.
[
https://issues.apache.org/jira/browse/HADOOP-18182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528440#comment-17528440
]
Bhalchandra Pandit commented on HADOOP-18182:
-
[~ste...@apache.org] I did not understand your comment : "as long as the s3
object reference lifespan is > than that of the stream's use, all is well.".
Can you please clarify?
> S3File to store reference to active S3Object in a field.
>
>
> Key: HADOOP-18182
> URL: https://issues.apache.org/jira/browse/HADOOP-18182
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Bhalchandra Pandit
>Priority: Major
>
> HADOOP-17338 showed us how recent {{S3Object.finalize()}} can call
> stream.close() and so close an active stream if a GC happens during a read.
> replicate the same fix here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18182) S3File to store reference to active S3Object in a field.
[
https://issues.apache.org/jira/browse/HADOOP-18182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhalchandra Pandit reassigned HADOOP-18182:
---
Assignee: Bhalchandra Pandit
> S3File to store reference to active S3Object in a field.
>
>
> Key: HADOOP-18182
> URL: https://issues.apache.org/jira/browse/HADOOP-18182
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Bhalchandra Pandit
>Priority: Major
>
> HADOOP-17338 showed us how recent {{S3Object.finalize()}} can call
> stream.close() and so close an active stream if a GC happens during a read.
> replicate the same fix here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[ https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=762537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762537 ] ASF GitHub Bot logged work on HADOOP-18069: --- Author: ASF GitHub Bot Created on: 26/Apr/22 20:48 Start Date: 26/Apr/22 20:48 Worklog Time Spent: 10m Work Description: ashutoshcipher commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1110235192 > commented. > > * checkstyle needs to be happy, along with javac. > * spotbugs still thinks there is a problem. what is it that it is warning about? > > i'm worried about adding kotlin everywhere. looking at the mvnrepo declarations it is (a) not optional and (b) about 1.5MB including transitive dependencies. so nothing much. my main concern is what pain does it cause downstream. we've had to tag this as an incompatible change just to add in the release notes about where it is used/needed - Checkstyle would be happy with my last commit. - Javac - 1. `hadoop-hdfs-project/hadoop-hdfs client/src/main/java/org/apache/hadoop/hdfs/web/oauth2/CredentialBasedAccessTokenProvider.java:109:36:[deprecation] create(MediaType,String) in RequestBody has been deprecated `- Handled this in latest commit. 2. `hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:229:76:[unchecked] unchecked cast` -I believe this is not due this change, can see the same cast warning in my local on trunk as well. May be we can create a separate JIRA to handle this. - spotbugs still thinks there is a problem. what is it that it is warning about? I already used try-with-resources for OkHttpClient and added request checks as well. The warning is about NPE case. - I agree with your view on Kotlin. Issue Time Tracking --- Worklog Id: (was: 762537) Time Spent: 2h 20m (was: 2h 10m) > CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client > --- > > Key: HADOOP-18069 > URL: https://issues.apache.org/jira/browse/HADOOP-18069 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.3.1 >Reporter: Eugene Shinn (Truveta) >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Our static vulnerability scanner (Fortify On Demand) detected [NVD - > CVE-2021-0341 > (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] > in our application. We traced the vulnerability to a transitive dependency > coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 > ([hadoop/pom.xml at trunk · apache/hadoop > (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). > To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: > [CVE-2021-0341 · Issue #6724 · square/okhttp > (github.com)|https://github.com/square/okhttp/issues/6724]). -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
ashutoshcipher commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1110235192 > commented. > > * checkstyle needs to be happy, along with javac. > * spotbugs still thinks there is a problem. what is it that it is warning about? > > i'm worried about adding kotlin everywhere. looking at the mvnrepo declarations it is (a) not optional and (b) about 1.5MB including transitive dependencies. so nothing much. my main concern is what pain does it cause downstream. we've had to tag this as an incompatible change just to add in the release notes about where it is used/needed - Checkstyle would be happy with my last commit. - Javac - 1. `hadoop-hdfs-project/hadoop-hdfs client/src/main/java/org/apache/hadoop/hdfs/web/oauth2/CredentialBasedAccessTokenProvider.java:109:36:[deprecation] create(MediaType,String) in RequestBody has been deprecated `- Handled this in latest commit. 2. `hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:229:76:[unchecked] unchecked cast` -I believe this is not due this change, can see the same cast warning in my local on trunk as well. May be we can create a separate JIRA to handle this. - spotbugs still thinks there is a problem. what is it that it is warning about? I already used try-with-resources for OkHttpClient and added request checks as well. The warning is about NPE case. - I agree with your view on Kotlin. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4228: HDFS-16468. Define ssize_t for Windows
hadoop-yetus commented on PR #4228:
URL: https://github.com/apache/hadoop/pull/4228#issuecomment-1110234952
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 38s | | trunk passed |
| +1 :green_heart: | compile | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 69m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 70m 2s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 3m 53s | | the patch passed |
| +1 :green_heart: | cc | 3m 53s | | the patch passed |
| +1 :green_heart: | golang | 3m 53s | | the patch passed |
| +1 :green_heart: | javac | 3m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 112m 25s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 212m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4228 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 79f6b6f5312c 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b26a9a96131cc3112107e471ed0b13e3a29dffd |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/testReport/ |
| Max. process+thread count | 603 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4228/5/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18104) Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads
[
https://issues.apache.org/jira/browse/HADOOP-18104?focusedWorklogId=762514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762514
]
ASF GitHub Bot logged work on HADOOP-18104:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 20:10
Start Date: 26/Apr/22 20:10
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on code in PR #3964:
URL: https://github.com/apache/hadoop/pull/3964#discussion_r859104404
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -564,6 +568,23 @@ public void initialize(URI name, Configuration
originalConf)
}
}
+ /**
+ * Populates the configurations related to vectored IO operation
+ * in the context which has to passed down to input streams.
+ * @param conf configuration object.
+ * @return VectoredIOContext.
+ */
+ private VectoredIOContext populateVectoredIOContext(Configuration conf) {
+final int minSeekVectored = conf.getInt(AWS_S3_VECTOR_READS_MIN_SEEK_SIZE,
Review Comment:
Yes. Thanks for pointing out.
Issue Time Tracking
---
Worklog Id: (was: 762514)
Time Spent: 3h 10m (was: 3h)
> Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads
>
>
> Key: HADOOP-18104
> URL: https://issues.apache.org/jira/browse/HADOOP-18104
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, fs
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a diff in pull request #3964: HADOOP-18104: S3A: Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads
mukund-thakur commented on code in PR #3964:
URL: https://github.com/apache/hadoop/pull/3964#discussion_r859104404
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -564,6 +568,23 @@ public void initialize(URI name, Configuration
originalConf)
}
}
+ /**
+ * Populates the configurations related to vectored IO operation
+ * in the context which has to passed down to input streams.
+ * @param conf configuration object.
+ * @return VectoredIOContext.
+ */
+ private VectoredIOContext populateVectoredIOContext(Configuration conf) {
+final int minSeekVectored = conf.getInt(AWS_S3_VECTOR_READS_MIN_SEEK_SIZE,
Review Comment:
Yes. Thanks for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18194) Public dataset class for S3A integration tests
[
https://issues.apache.org/jira/browse/HADOOP-18194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528370#comment-17528370
]
Steve Loughran commented on HADOOP-18194:
-
yeah
> Public dataset class for S3A integration tests
> --
>
> Key: HADOOP-18194
> URL: https://issues.apache.org/jira/browse/HADOOP-18194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Daniel Carl Jones
>Assignee: Daniel Carl Jones
>Priority: Minor
>
> Introduction of PublicDatasetTestUtils as proposed previously in some of the
> ideas for refactoring S3A incrementally. Some of its responsibilities:
> - Source of truth for getting URI based on public data set.
> - Maybe keep the methods specific to their purpose where possible? We might
> need {{s3a://landsat-pds/scene_list.gz}} specifically for some tests, but
> other tests may just need a bucket with a bunch of keys.
> - Introduce test assumptions about the S3 endpoint or AWS partition. If we’re
> not looking at 'aws' partition, skip test.
> How might we make this generic for non-{{aws}} partition S3 or
> S3API-compatible object stores?
> - Ideally allow for future extension to provide some easy ways to override
> the bucket if tester has an alternative source? I see
> "fs.s3a.scale.test.csvfile" already has a little bit of this.
> - We could have something which takes a path to a hadoop XML config file;
> we'd have a default resource but the maven build could be pointed at another
> via a command line property. this file could contain all the settings for a
> test against a partition or internal s3-compatible store
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18216) Document "io.file.buffer.size" must be greater than zero
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18216.
-
Resolution: Fixed
documentation is updated. PR discusses a followup with safety check on use
> Document "io.file.buffer.size" must be greater than zero
>
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18216) Document "io.file.buffer.size" must be greater than zero
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18216:
Summary: Document "io.file.buffer.size" must be greater than zero (was:
Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
data read/write blockage)
> Document "io.file.buffer.size" must be greater than zero
>
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18216) Document "io.file.buffer.size" must be greater than zero
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18216:
Priority: Minor (was: Major)
> Document "io.file.buffer.size" must be greater than zero
>
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18216:
Fix Version/s: 3.4.0
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?focusedWorklogId=762486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762486
]
ASF GitHub Bot logged work on HADOOP-18216:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 18:53
Start Date: 26/Apr/22 18:53
Worklog Time Spent: 10m
Work Description: steveloughran merged PR #4220:
URL: https://github.com/apache/hadoop/pull/4220
Issue Time Tracking
---
Worklog Id: (was: 762486)
Time Spent: 1h 20m (was: 1h 10m)
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4220: HADOOP-18216. io.file.buffer.size must be greater than zero
steveloughran merged PR #4220: URL: https://github.com/apache/hadoop/pull/4220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?focusedWorklogId=762485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762485
]
ASF GitHub Bot logged work on HADOOP-18216:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 18:52
Start Date: 26/Apr/22 18:52
Worklog Time Spent: 10m
Work Description: steveloughran commented on PR #4220:
URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1110139088
@Hexiaoqiao yes, you can add a hadoop.util.Precondition check where it is
loaded, but do it in a new PR. let's merge this doc change in first, as it self
contained and low risk to backport (unless the xml is broken, safe)
Issue Time Tracking
---
Worklog Id: (was: 762485)
Time Spent: 1h 10m (was: 1h)
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4220: HADOOP-18216. io.file.buffer.size must be greater than zero
steveloughran commented on PR #4220: URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1110139088 @Hexiaoqiao yes, you can add a hadoop.util.Precondition check where it is loaded, but do it in a new PR. let's merge this doc change in first, as it self contained and low risk to backport (unless the xml is broken, safe) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[ https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=762481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762481 ] ASF GitHub Bot logged work on HADOOP-18069: --- Author: ASF GitHub Bot Created on: 26/Apr/22 18:50 Start Date: 26/Apr/22 18:50 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1110137354 commented. * checkstyle needs to be happy, along with javac. * spotbugs still thinks there is a problem. what is it that it is warning about? i'm worried about adding kotlin everywhere. looking at the mvnrepo declarations it is (a) not optional and (b) about 1.5MB including transitive dependencies. so nothing much. my main concern is what pain does it cause downstream. we've had to tag this as an incompatible change just to add in the release notes about where it is used/needed Issue Time Tracking --- Worklog Id: (was: 762481) Time Spent: 2h 10m (was: 2h) > CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client > --- > > Key: HADOOP-18069 > URL: https://issues.apache.org/jira/browse/HADOOP-18069 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.3.1 >Reporter: Eugene Shinn (Truveta) >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Our static vulnerability scanner (Fortify On Demand) detected [NVD - > CVE-2021-0341 > (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] > in our application. We traced the vulnerability to a transitive dependency > coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 > ([hadoop/pom.xml at trunk · apache/hadoop > (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). > To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: > [CVE-2021-0341 · Issue #6724 · square/okhttp > (github.com)|https://github.com/square/okhttp/issues/6724]). -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
steveloughran commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1110137354 commented. * checkstyle needs to be happy, along with javac. * spotbugs still thinks there is a problem. what is it that it is warning about? i'm worried about adding kotlin everywhere. looking at the mvnrepo declarations it is (a) not optional and (b) about 1.5MB including transitive dependencies. so nothing much. my main concern is what pain does it cause downstream. we've had to tag this as an incompatible change just to add in the release notes about where it is used/needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10-4.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10-4.patch, > HADOOP-18167-branch-2.10.patch, HADOOP-18167-branch-3.3.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18028) High performance S3A input stream with prefetching & caching
[ https://issues.apache.org/jira/browse/HADOOP-18028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528330#comment-17528330 ] Ahmar Suhail commented on HADOOP-18028: --- [https://github.com/apache/hadoop/pull/4212] fixes the failing tests > High performance S3A input stream with prefetching & caching > > > Key: HADOOP-18028 > URL: https://issues.apache.org/jira/browse/HADOOP-18028 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Bhalchandra Pandit >Assignee: Bhalchandra Pandit >Priority: Major > Labels: pull-request-available > Time Spent: 13h 50m > Remaining Estimate: 0h > > I work for Pinterest. I developed a technique for vastly improving read > throughput when reading from the S3 file system. It not only helps the > sequential read case (like reading a SequenceFile) but also significantly > improves read throughput of a random access case (like reading Parquet). This > technique has been very useful in significantly improving efficiency of the > data processing jobs at Pinterest. > > I would like to contribute that feature to Apache Hadoop. More details on > this technique are available in this blog I wrote recently: > [https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0] > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18177. - Resolution: Fixed > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18177: Fix Version/s: 3.4.0 > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762448 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 17:36 Start Date: 26/Apr/22 17:36 Worklog Time Spent: 10m Work Description: steveloughran merged PR #4205: URL: https://github.com/apache/hadoop/pull/4205 Issue Time Tracking --- Worklog Id: (was: 762448) Time Spent: 2h 40m (was: 2.5h) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4205: HADOOP-18177. Document prefetching architecture.
steveloughran merged PR #4205: URL: https://github.com/apache/hadoop/pull/4205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18028) High performance S3A input stream with prefetching & caching
[ https://issues.apache.org/jira/browse/HADOOP-18028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528323#comment-17528323 ] Steve Loughran commented on HADOOP-18028: - this branch is going to need rebasing. what outstanding prs are there other than the doc one? > High performance S3A input stream with prefetching & caching > > > Key: HADOOP-18028 > URL: https://issues.apache.org/jira/browse/HADOOP-18028 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Bhalchandra Pandit >Assignee: Bhalchandra Pandit >Priority: Major > Labels: pull-request-available > Time Spent: 13h 50m > Remaining Estimate: 0h > > I work for Pinterest. I developed a technique for vastly improving read > throughput when reading from the S3 file system. It not only helps the > sequential read case (like reading a SequenceFile) but also significantly > improves read throughput of a random access case (like reading Parquet). This > technique has been very useful in significantly improving efficiency of the > data processing jobs at Pinterest. > > I would like to contribute that feature to Apache Hadoop. More details on > this technique are available in this blog I wrote recently: > [https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0] > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-3.3.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10.patch, > HADOOP-18167-branch-3.3.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a diff in pull request #4228: HDFS-16468. Define ssize_t for Windows
GauthamBanasandra commented on code in PR #4228:
URL: https://github.com/apache/hadoop/pull/4228#discussion_r858971962
##
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:
##
@@ -487,10 +488,10 @@ static ssize_t wildcard_expandPath(const char* path,
char* expanded)
* allocated after using this function with expandedClasspath=NULL to get the
* right size.
*/
-static ssize_t getClassPath_helper(const char *classpath, char*
expandedClasspath)
+static x_platform_ssize_t getClassPath_helper(const char *classpath, char*
expandedClasspath)
{
-ssize_t length;
-ssize_t retval;
+x_platform_ssize_t length;
Review Comment:
> Is this the best way to do this? Can't we just do the typedef for windows?
I've handled this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4170: HDFS-16540 Data locality is lost when DataNode pod restarts in kubern…
huaxiangsun commented on PR #4170: URL: https://github.com/apache/hadoop/pull/4170#issuecomment-1110051006 Any more comments? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4235: YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name
hadoop-yetus commented on PR #4235:
URL: https://github.com/apache/hadoop/pull/4235#issuecomment-1110048699
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 55s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 59s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/2/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 48s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 106m 27s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 209m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4235 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7a9581c36de9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50a00104796d10b3343880a1aee1123415ef89bc |
| Default Java | Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #4235: YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name
hadoop-yetus commented on PR #4235:
URL: https://github.com/apache/hadoop/pull/4235#issuecomment-1110046794
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/1/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18) |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 105m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 209m 38s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4235 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 64d2371801cd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50a00104796d10b3343880a1aee1123415ef89bc |
| Default Java | Private
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[
https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762417
]
ASF GitHub Bot logged work on HADOOP-18168:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 16:51
Start Date: 26/Apr/22 16:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4140:
URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1110029320
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 0m 38s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with
JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 10 new + 17 unchanged -
0 fixed = 27 total (was 17) |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 0m 32s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 10 new
+ 17 unchanged - 0 fixed = 27 total (was 17) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed
= 2 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 19s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 95m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4140: HADOOP-18168. Fix S3A ITestMarkerTool dependency on purged public bucket
hadoop-yetus commented on PR #4140:
URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1110029320
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 0m 38s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with
JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 10 new + 17 unchanged -
0 fixed = 27 total (was 17) |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 0m 32s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 10 new
+ 17 unchanged - 0 fixed = 27 total (was 17) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed
= 2 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 19s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 95m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4140 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux e105e845a0f4 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision |
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=762413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762413 ] ASF GitHub Bot logged work on HADOOP-18167: --- Author: ASF GitHub Bot Created on: 26/Apr/22 16:20 Start Date: 26/Apr/22 16:20 Worklog Time Spent: 10m Work Description: omalley merged PR #4092: URL: https://github.com/apache/hadoop/pull/4092 Issue Time Tracking --- Worklog Id: (was: 762413) Time Spent: 5h 50m (was: 5h 40m) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley merged pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
omalley merged PR #4092: URL: https://github.com/apache/hadoop/pull/4092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[
https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762384
]
ASF GitHub Bot logged work on HADOOP-18168:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 15:16
Start Date: 26/Apr/22 15:16
Worklog Time Spent: 10m
Work Description: dannycjones commented on code in PR #4140:
URL: https://github.com/apache/hadoop/pull/4140#discussion_r858847636
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java:
##
@@ -108,6 +108,21 @@
*/
String DEFAULT_REQUESTER_PAYS_FILE =
"s3a://usgs-landsat/collection02/catalog.json";
+ /**
+ * Configuration key for an existing bucket with many objects: {@value}.
+ *
+ * This is used for tests depending on buckets with a large number of keys.
+ */
+ String KEY_BUCKET_WITH_MANY_OBJECTS
+ = TEST_FS_S3A + "bucket-with-many-objects";
+
+ /**
+ * Default bucket for when {@value KEY_BUCKET_WITH_MANY_OBJECTS} is not set:
+ * {@value}.
+ */
+ String DEFAULT_BUCKET_MANY_OBJECTS
+ = "s3a://commoncrawl/";
Review Comment:
migrating to `usgs-landsat`
https://registry.opendata.aws/usgs-landsat/index.html
Issue Time Tracking
---
Worklog Id: (was: 762384)
Time Spent: 1h 20m (was: 1h 10m)
> ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket
> content purged
> ---
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Daniel Carl Jones
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4140: HADOOP-18168. Fix S3A ITestMarkerTool dependency on purged public bucket
dannycjones commented on code in PR #4140:
URL: https://github.com/apache/hadoop/pull/4140#discussion_r858847636
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java:
##
@@ -108,6 +108,21 @@
*/
String DEFAULT_REQUESTER_PAYS_FILE =
"s3a://usgs-landsat/collection02/catalog.json";
+ /**
+ * Configuration key for an existing bucket with many objects: {@value}.
+ *
+ * This is used for tests depending on buckets with a large number of keys.
+ */
+ String KEY_BUCKET_WITH_MANY_OBJECTS
+ = TEST_FS_S3A + "bucket-with-many-objects";
+
+ /**
+ * Default bucket for when {@value KEY_BUCKET_WITH_MANY_OBJECTS} is not set:
+ * {@value}.
+ */
+ String DEFAULT_BUCKET_MANY_OBJECTS
+ = "s3a://commoncrawl/";
Review Comment:
migrating to `usgs-landsat`
https://registry.opendata.aws/usgs-landsat/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4186: HDFS-16528. Reconfigure slow peer enable for Namenode
hadoop-yetus commented on PR #4186:
URL: https://github.com/apache/hadoop/pull/4186#issuecomment-1109875631
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 42s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 365m 6s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 485m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4186/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4186 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1314cff2b79d 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cfd7d2df778b8eb0f46d790a6c3856209fb4a3f6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4186/11/testReport/ |
| Max. process+thread count | 2012 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4186/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762347
]
ASF GitHub Bot logged work on HADOOP-18177:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 14:22
Start Date: 26/Apr/22 14:22
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109860466
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 59s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 56s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 93m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 416a76a29726 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
9558361263d879b0bef5f452528dca5f18abdc15 |
| Max. process+thread count | 525 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 762347)
Time Spent: 2.5h (was: 2h 20m)
> document use and architecture design of prefetching s3a input stream
>
>
> Key: HADOOP-18177
> URL: https://issues.apache.org/jira/browse/HADOOP-18177
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Document S3PrefetchingInputStream for users (including any new failure modes
> in troubleshooting) and the architecture for maintainers
> there's some markdown in
> hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md
> already
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4205: HADOOP-18177. Document prefetching architecture.
hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109860466
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 59s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 56s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 93m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 416a76a29726 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
9558361263d879b0bef5f452528dca5f18abdc15 |
| Max. process+thread count | 525 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762317
]
ASF GitHub Bot logged work on HADOOP-18177:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 13:49
Start Date: 26/Apr/22 13:49
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109822081
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 6s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 55s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 93m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 918724e3c41c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
658396acb384ba50103e240e890f31cf1355388a |
| Max. process+thread count | 601 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 762317)
Time Spent: 2h 20m (was: 2h 10m)
> document use and architecture design of prefetching s3a input stream
>
>
> Key: HADOOP-18177
> URL: https://issues.apache.org/jira/browse/HADOOP-18177
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Document S3PrefetchingInputStream for users (including any new failure modes
> in troubleshooting) and the architecture for maintainers
> there's some markdown in
> hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md
> already
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4205: HADOOP-18177. Document prefetching architecture.
hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109822081
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 6s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 55s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 93m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 918724e3c41c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
658396acb384ba50103e240e890f31cf1355388a |
| Max. process+thread count | 601 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth opened a new pull request, #4235: YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name
szilard-nemeth opened a new pull request, #4235: URL: https://github.com/apache/hadoop/pull/4235 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[
https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=762312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762312
]
ASF GitHub Bot logged work on HADOOP-18069:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 13:38
Start Date: 26/Apr/22 13:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1109809947
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 28s | | trunk passed |
| +1 :green_heart: | compile | 24m 55s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 1s | | trunk passed |
| -1 :x: | javadoc | 1m 35s |
[/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 8m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 27s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 57m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 24m 37s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | compile | 24m 35s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 24m 35s |
[/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 2 new + 1815 unchanged - 0
fixed = 1817 total (was 1815) |
| +1 :green_heart: | compile | 21m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 21m 34s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1690
unchanged - 0 fixed = 1692 total (was 1690) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 19m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 1m 28s |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
hadoop-yetus commented on PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1109809947
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 28s | | trunk passed |
| +1 :green_heart: | compile | 24m 55s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 1s | | trunk passed |
| -1 :x: | javadoc | 1m 35s |
[/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 8m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 27s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 57m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 24m 37s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | compile | 24m 35s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 24m 35s |
[/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 2 new + 1815 unchanged - 0
fixed = 1817 total (was 1815) |
| +1 :green_heart: | compile | 21m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 21m 34s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1690
unchanged - 0 fixed = 1692 total (was 1690) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 19m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 1m 28s |
[/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/5/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.
|
| +1 :green_heart: | javadoc | 8m 20s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 26s | | hadoop-project has no data from
spotbugs |
| -1 :x: | spotbugs | 2m 52s |
[jira] [Commented] (HADOOP-18194) Public dataset class for S3A integration tests
[
https://issues.apache.org/jira/browse/HADOOP-18194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528168#comment-17528168
]
Daniel Carl Jones commented on HADOOP-18194:
[~ste...@apache.org], as you mention there are a number of settings for landsat
in test/resources/core-site.xml.
I was wondering if we should add two configurations to every public data set
bucket:
* {{fs.s3a.bucket..audit.add.referrer.header}} to prevent filling up
any access logs?
* {{{}fs.s3a.bucket..multipart.purge{}}}. to prevent log spam if the
tests try to upload here
Does that make sense?
> Public dataset class for S3A integration tests
> --
>
> Key: HADOOP-18194
> URL: https://issues.apache.org/jira/browse/HADOOP-18194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Daniel Carl Jones
>Assignee: Daniel Carl Jones
>Priority: Minor
>
> Introduction of PublicDatasetTestUtils as proposed previously in some of the
> ideas for refactoring S3A incrementally. Some of its responsibilities:
> - Source of truth for getting URI based on public data set.
> - Maybe keep the methods specific to their purpose where possible? We might
> need {{s3a://landsat-pds/scene_list.gz}} specifically for some tests, but
> other tests may just need a bucket with a bunch of keys.
> - Introduce test assumptions about the S3 endpoint or AWS partition. If we’re
> not looking at 'aws' partition, skip test.
> How might we make this generic for non-{{aws}} partition S3 or
> S3API-compatible object stores?
> - Ideally allow for future extension to provide some easy ways to override
> the bucket if tester has an alternative source? I see
> "fs.s3a.scale.test.csvfile" already has a little bit of this.
> - We could have something which takes a path to a hadoop XML config file;
> we'd have a default resource but the maven build could be pointed at another
> via a command line property. this file could contain all the settings for a
> test against a partition or internal s3-compatible store
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?focusedWorklogId=762299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762299
]
ASF GitHub Bot logged work on HADOOP-18216:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 13:04
Start Date: 26/Apr/22 13:04
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on PR #4220:
URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109772315
Hi @steveloughran and @liever18 Sorry for late comments. I found that this
pr just update comments, right? Is it better to improve the logic to guard the
illegal configuration? Thanks.
Issue Time Tracking
---
Worklog Id: (was: 762299)
Time Spent: 1h (was: 50m)
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #4220: HADOOP-18216. io.file.buffer.size must be greater than zero
Hexiaoqiao commented on PR #4220: URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109772315 Hi @steveloughran and @liever18 Sorry for late comments. I found that this pr just update comments, right? Is it better to improve the logic to guard the illegal configuration? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?focusedWorklogId=762276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762276
]
ASF GitHub Bot logged work on HADOOP-18216:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 12:48
Start Date: 26/Apr/22 12:48
Worklog Time Spent: 10m
Work Description: liever18 commented on PR #4220:
URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109754227
> i am about to merge this. should i give the credit to Jingxuan Fu?
yeap! "Jingxuan Fu" is my jira's account.
Issue Time Tracking
---
Worklog Id: (was: 762276)
Time Spent: 50m (was: 40m)
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liever18 commented on pull request #4220: HADOOP-18216. io.file.buffer.size must be greater than zero
liever18 commented on PR #4220: URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109754227 > i am about to merge this. should i give the credit to Jingxuan Fu? yeap! "Jingxuan Fu" is my jira's account. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?focusedWorklogId=762273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762273
]
ASF GitHub Bot logged work on HADOOP-18216:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 12:44
Start Date: 26/Apr/22 12:44
Worklog Time Spent: 10m
Work Description: steveloughran commented on PR #4220:
URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109750693
i am about to merge this. should i give the credit to Jingxuan Fu?
Issue Time Tracking
---
Worklog Id: (was: 762273)
Time Spent: 40m (was: 0.5h)
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4220: HADOOP-18216. io.file.buffer.size must be greater than zero
steveloughran commented on PR #4220: URL: https://github.com/apache/hadoop/pull/4220#issuecomment-1109750693 i am about to merge this. should i give the credit to Jingxuan Fu? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762246
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 11:49
Start Date: 26/Apr/22 11:49
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109697812
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 52s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 49s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 55s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 42s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 44s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 33s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 54s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 100m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 5f5921be7f77 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
e1eb23ed6dd6b7ff0661411e82d4b0d92ddd3051 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/5/testReport/ |
| Max. process+thread count | 675 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109697812
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 52s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 49s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 55s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 42s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 44s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 33s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 54s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 100m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 5f5921be7f77 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
e1eb23ed6dd6b7ff0661411e82d4b0d92ddd3051 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/5/testReport/ |
| Max. process+thread count | 675 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762241=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762241
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 11:35
Start Date: 26/Apr/22 11:35
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109686826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 20s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 0m 59s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 46s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 58s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 40s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 46s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 36s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 43s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 101m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 806dec1ccaf6 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
f0d25f761238670646262c0866094324a02e8e23 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/4/testReport/ |
| Max. process+thread count | 740 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109686826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 20s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 0m 59s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 46s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 58s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 40s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 46s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 36s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 43s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 101m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 806dec1ccaf6 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
f0d25f761238670646262c0866094324a02e8e23 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/4/testReport/ |
| Max. process+thread count | 740 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762239 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 11:28 Start Date: 26/Apr/22 11:28 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858597582 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,192 @@ + + +# S3A Prefetching + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input +stream. +A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0). + +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. +It also potentially caches these blocks. + +### Basic Concepts + +* **Remote File**: A binary blob of data stored on some storage device. +* **Block File**: Local file containing a block of the remote file. +* **Block**: A file is divided into a number of blocks. +The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. +That is, either an entire block is read and returned or none at all. +Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | +|`fs.s3a.prefetch.block.size`|Size of a block|`8M`| +|`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| + +### Key Components + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of +this class as the input stream. +Depending on the remote file size, it will either use +the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the remote file size < configured block +size. +Will read the entire remote file into memory. + +`S3CachingInputStream` - Underlying input stream used when remote file size > configured block size. +Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a remote file, such as: + +* Number of blocks in the remote file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). +Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). +Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. +It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to +the remote file in S3. + +`S3Reader` - Implements reading from the stream opened by `S3File`. +Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. +Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. +Each cache block is stored on the local disk as a separate block file. + +### Operation + + S3InMemoryInputStream + +For a remote file with size 5MB, and block size = 8MB, since file size is less than the block size, +the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. +The `S3InMemoryInputStream` gets the data in this remote file by calling the `ensureCurrentBuffer()` +method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)`. +* `S3Reader` uses `S3File` to open an input stream to the remote file in S3 by making + a `getObject()` request with range as `(0, filesize)`. +* The `S3Reader` reads the entire remote
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4205: HADOOP-18177. Document prefetching architecture.
dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858597582 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,192 @@ + + +# S3A Prefetching + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input +stream. +A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0). + +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. +It also potentially caches these blocks. + +### Basic Concepts + +* **Remote File**: A binary blob of data stored on some storage device. +* **Block File**: Local file containing a block of the remote file. +* **Block**: A file is divided into a number of blocks. +The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. +That is, either an entire block is read and returned or none at all. +Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | +|`fs.s3a.prefetch.block.size`|Size of a block|`8M`| +|`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| + +### Key Components + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of +this class as the input stream. +Depending on the remote file size, it will either use +the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the remote file size < configured block +size. +Will read the entire remote file into memory. + +`S3CachingInputStream` - Underlying input stream used when remote file size > configured block size. +Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a remote file, such as: + +* Number of blocks in the remote file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). +Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). +Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. +It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to +the remote file in S3. + +`S3Reader` - Implements reading from the stream opened by `S3File`. +Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. +Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. +Each cache block is stored on the local disk as a separate block file. + +### Operation + + S3InMemoryInputStream + +For a remote file with size 5MB, and block size = 8MB, since file size is less than the block size, +the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. +The `S3InMemoryInputStream` gets the data in this remote file by calling the `ensureCurrentBuffer()` +method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)`. +* `S3Reader` uses `S3File` to open an input stream to the remote file in S3 by making + a `getObject()` request with range as `(0, filesize)`. +* The `S3Reader` reads the entire remote file into the provided buffer, and once reading is complete + closes the S3 stream and frees all underlying resources. +* Now the entire remote file is in a buffer, set this data in `FilePosition` so it can be accessed + by the input stream. + +The read operation now just gets the required bytes from the buffer in `FilePosition`. + +When the second read is issued, there is already a valid buffer which can be used.
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762237
]
ASF GitHub Bot logged work on HADOOP-18177:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 11:18
Start Date: 26/Apr/22 11:18
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109670724
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 23s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-eol.txt)
| The patch has 30 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 94m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux afff6318d18d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
125152ef64d24083abb7308978302285c1720fb6 |
| Max. process+thread count | 589 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 762237)
Time Spent: 2h (was: 1h 50m)
> document use and architecture design of prefetching s3a input stream
>
>
> Key: HADOOP-18177
> URL: https://issues.apache.org/jira/browse/HADOOP-18177
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Document S3PrefetchingInputStream for users (including any new failure modes
> in troubleshooting) and the architecture for maintainers
> there's some markdown in
> hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md
> already
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4205: HADOOP-18177. Document prefetching architecture.
hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109670724
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 23s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 53s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-eol.txt)
| The patch has 30 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 94m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux afff6318d18d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
125152ef64d24083abb7308978302285c1720fb6 |
| Max. process+thread count | 589 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762232
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 11:06
Start Date: 26/Apr/22 11:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109660722
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 45s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 1m 3s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 57s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 50s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 1m 2s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 48s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 54s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 44s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 53s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 100m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 617a41e90ade 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
ce15ebc251e87aa88b900283aaec4633181511fa |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/3/testReport/ |
| Max. process+thread count | 547 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
hadoop-yetus commented on PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#issuecomment-1109660722
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 45s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 1m 3s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 57s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 50s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 1m 2s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 0m 48s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 54s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 44s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 22m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 53s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 100m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4212 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 617a41e90ade 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
ce15ebc251e87aa88b900283aaec4633181511fa |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/3/testReport/ |
| Max. process+thread count | 547 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4212/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
[GitHub] [hadoop] hadoop-yetus commented on pull request #4178: HDFS-16543. Keep default value of dfs.datanode.directoryscan.throttle.limit.ms.pe…
hadoop-yetus commented on PR #4178:
URL: https://github.com/apache/hadoop/pull/4178#issuecomment-1109647004
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 20s | | trunk passed |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 364m 25s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 483m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4178/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4178 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 84d7ab623bcd 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d69b76429f17c4bf3f962ca1d2b0ec4b725f66c4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4178/4/testReport/ |
| Max. process+thread count | 2208 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4178/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762219
]
ASF GitHub Bot logged work on HADOOP-18177:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 10:38
Start Date: 26/Apr/22 10:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109635269
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 38s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 56s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 94m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 9472cd5389b4 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
ba1d26a0f77b6d3795cdeb923c1403dfa31e7ded |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 762219)
Time Spent: 1h 50m (was: 1h 40m)
> document use and architecture design of prefetching s3a input stream
>
>
> Key: HADOOP-18177
> URL: https://issues.apache.org/jira/browse/HADOOP-18177
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Document S3PrefetchingInputStream for users (including any new failure modes
> in troubleshooting) and the architecture for maintainers
> there's some markdown in
> hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md
> already
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4205: HADOOP-18177. Document prefetching architecture.
hadoop-yetus commented on PR #4205:
URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109635269
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 38s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 0m 56s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 66m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 94m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4205 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 9472cd5389b4 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
ba1d26a0f77b6d3795cdeb923c1403dfa31e7ded |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762204
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 09:37
Start Date: 26/Apr/22 09:37
Worklog Time Spent: 10m
Work Description: monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858504016
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
just noticed we can use `FSDataInputStream.hasCapability(...)`, I will
change to it
Issue Time Tracking
---
Worklog Id: (was: 762204)
Time Spent: 1h 10m (was: 1h)
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Monthon Klongklaew
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] monthonk commented on a diff in pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858504016
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
just noticed we can use `FSDataInputStream.hasCapability(...)`, I will
change to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[
https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=762203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762203
]
ASF GitHub Bot logged work on HADOOP-18069:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 09:36
Start Date: 26/Apr/22 09:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1109576214
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 33s | | trunk passed |
| +1 :green_heart: | compile | 25m 8s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 3s | | trunk passed |
| -1 :x: | javadoc | 1m 36s |
[/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 8m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 28s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 58m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 24m 59s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | compile | 24m 36s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 24m 36s |
[/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 2 new + 1810 unchanged - 0
fixed = 1812 total (was 1810) |
| +1 :green_heart: | compile | 21m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 21m 37s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1686
unchanged - 0 fixed = 1688 total (was 1686) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 19m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 1m 26s |
[/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.
|
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed with JDK
Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
hadoop-yetus commented on PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1109576214
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 33s | | trunk passed |
| +1 :green_heart: | compile | 25m 8s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 3s | | trunk passed |
| -1 :x: | javadoc | 1m 36s |
[/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 8m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 28s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 58m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 24m 59s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | compile | 24m 36s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 24m 36s |
[/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 2 new + 1810 unchanged - 0
fixed = 1812 total (was 1810) |
| +1 :green_heart: | compile | 21m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 21m 37s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1686
unchanged - 0 fixed = 1688 total (was 1686) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 19m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 1m 26s |
[/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.
|
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 26s | | hadoop-project has no data from
spotbugs |
| -1 :x: | spotbugs | 2m 57s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4229/6/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html)
| hadoop-hdfs-project/hadoop-hdfs-client generated 3 new + 0
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762200
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 09:25
Start Date: 26/Apr/22 09:25
Worklog Time Spent: 10m
Work Description: monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858492522
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java:
##
@@ -71,11 +72,16 @@ public void testRequesterPaysOptionSuccess() throws
Throwable {
inputStream.seek(0);
inputStream.readByte();
- // Verify > 1 call was made, so we're sure it is correctly configured
for each request
- IOStatisticAssertions
- .assertThatStatisticCounter(inputStream.getIOStatistics(),
- StreamStatisticNames.STREAM_READ_OPENED)
- .isGreaterThan(1);
+ if (conf.getBoolean(PREFETCH_ENABLED_KEY, true)) {
+// For S3PrefetchingInputStream, verify a call was made
+
IOStatisticAssertions.assertThatStatisticCounter(inputStream.getIOStatistics(),
+StreamStatisticNames.STREAM_READ_OPENED).isEqualTo(1);
+ } else {
+// For S3InputStream, verify > 1 call was made,
Review Comment:
you're right, updated the comment
Issue Time Tracking
---
Worklog Id: (was: 762200)
Time Spent: 1h (was: 50m)
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Monthon Klongklaew
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] monthonk commented on a diff in pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858492522
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java:
##
@@ -71,11 +72,16 @@ public void testRequesterPaysOptionSuccess() throws
Throwable {
inputStream.seek(0);
inputStream.readByte();
- // Verify > 1 call was made, so we're sure it is correctly configured
for each request
- IOStatisticAssertions
- .assertThatStatisticCounter(inputStream.getIOStatistics(),
- StreamStatisticNames.STREAM_READ_OPENED)
- .isGreaterThan(1);
+ if (conf.getBoolean(PREFETCH_ENABLED_KEY, true)) {
+// For S3PrefetchingInputStream, verify a call was made
+
IOStatisticAssertions.assertThatStatisticCounter(inputStream.getIOStatistics(),
+StreamStatisticNames.STREAM_READ_OPENED).isEqualTo(1);
+ } else {
+// For S3InputStream, verify > 1 call was made,
Review Comment:
you're right, updated the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762199
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 09:22
Start Date: 26/Apr/22 09:22
Worklog Time Spent: 10m
Work Description: monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858489694
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
At this point we still don't know what kind of inputStream it is, the method
`hasCapability` only visible after we cast it to S3AInputStream or
S3PrefetchingInputStream
Issue Time Tracking
---
Worklog Id: (was: 762199)
Time Spent: 50m (was: 40m)
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Monthon Klongklaew
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] monthonk commented on a diff in pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
monthonk commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858489694
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
At this point we still don't know what kind of inputStream it is, the method
`hasCapability` only visible after we cast it to S3AInputStream or
S3PrefetchingInputStream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4104: HDFS-16520. Improve EC pread: avoid potential reading whole block
hadoop-yetus commented on PR #4104:
URL: https://github.com/apache/hadoop/pull/4104#issuecomment-1109548468
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 23s | | trunk passed |
| +1 :green_heart: | compile | 6m 20s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 6m 1s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 6m 9s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 6m 9s | | the patch passed |
| +1 :green_heart: | compile | 5m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 5m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 42s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 245m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not
generate ASF License warnings. |
| | | 394m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4104 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0fd20f4378da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 719ae6e859d2af3d72e0a75c4854b3f6734ec6f8 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/8/testReport/ |
| Max. process+thread count | 3470 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/8/console |
| versions |
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762184 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 09:04 Start Date: 26/Apr/22 09:04 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858455675 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. -With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. It also potentially +caches these blocks. ### Basic Concepts -* **File** : A binary blob of data stored on some storage device. -* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. -* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. +* **Remote File**: A binary blob of data stored on some storage device. +* **Block**: A file is divided into a number of blocks. The size of the first n-1 blocks is same, + and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. That is, either an entire block is + read and returned or none at all. Multiple blocks may be read in parallel. ### Configuring the stream |Property|Meaning|Default| |--- |---|---| -|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | -|fs.s3a.prefetch.block.size|Size of a block|8MB| -|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|`true` | +|fs.s3a.prefetch.block.size|Size of a block|`8M`| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|`8`| Review Comment: We want backticks around the configuration keys too. ```suggestion |`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | |`fs.s3a.prefetch.block.size`|Size of a block|`8M`| |`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| ``` ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. Review Comment: drop `blogpost` as it is already mentioned earlier ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4205: HADOOP-18177. Document prefetching architecture.
dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858455675 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. -With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. It also potentially +caches these blocks. ### Basic Concepts -* **File** : A binary blob of data stored on some storage device. -* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. -* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. +* **Remote File**: A binary blob of data stored on some storage device. +* **Block**: A file is divided into a number of blocks. The size of the first n-1 blocks is same, + and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. That is, either an entire block is + read and returned or none at all. Multiple blocks may be read in parallel. ### Configuring the stream |Property|Meaning|Default| |--- |---|---| -|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | -|fs.s3a.prefetch.block.size|Size of a block|8MB| -|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|`true` | +|fs.s3a.prefetch.block.size|Size of a block|`8M`| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|`8`| Review Comment: We want backticks around the configuration keys too. ```suggestion |`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | |`fs.s3a.prefetch.block.size`|Size of a block|`8M`| |`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| ``` ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. Review Comment: drop `blogpost` as it is already mentioned earlier ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in
[jira] [Work logged] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?focusedWorklogId=762181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762181
]
ASF GitHub Bot logged work on HADOOP-18175:
---
Author: ASF GitHub Bot
Created on: 26/Apr/22 08:58
Start Date: 26/Apr/22 08:58
Worklog Time Spent: 10m
Work Description: ahmarsuhail commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858421027
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java:
##
@@ -71,11 +72,16 @@ public void testRequesterPaysOptionSuccess() throws
Throwable {
inputStream.seek(0);
inputStream.readByte();
- // Verify > 1 call was made, so we're sure it is correctly configured
for each request
- IOStatisticAssertions
- .assertThatStatisticCounter(inputStream.getIOStatistics(),
- StreamStatisticNames.STREAM_READ_OPENED)
- .isGreaterThan(1);
+ if (conf.getBoolean(PREFETCH_ENABLED_KEY, true)) {
+// For S3PrefetchingInputStream, verify a call was made
+
IOStatisticAssertions.assertThatStatisticCounter(inputStream.getIOStatistics(),
+StreamStatisticNames.STREAM_READ_OPENED).isEqualTo(1);
+ } else {
+// For S3InputStream, verify > 1 call was made,
Review Comment:
should this be For S3AInputStream ?
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
You should just be able to use
`inputStream.hasCapability(StreamCapabilities.UNBUFFER)` instead of this method
Issue Time Tracking
---
Worklog Id: (was: 762181)
Time Spent: 40m (was: 0.5h)
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Monthon Klongklaew
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #4212: HADOOP-18175. fix test failures with prefetching s3a input stream
ahmarsuhail commented on code in PR #4212:
URL: https://github.com/apache/hadoop/pull/4212#discussion_r858421027
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java:
##
@@ -71,11 +72,16 @@ public void testRequesterPaysOptionSuccess() throws
Throwable {
inputStream.seek(0);
inputStream.readByte();
- // Verify > 1 call was made, so we're sure it is correctly configured
for each request
- IOStatisticAssertions
- .assertThatStatisticCounter(inputStream.getIOStatistics(),
- StreamStatisticNames.STREAM_READ_OPENED)
- .isGreaterThan(1);
+ if (conf.getBoolean(PREFETCH_ENABLED_KEY, true)) {
+// For S3PrefetchingInputStream, verify a call was made
+
IOStatisticAssertions.assertThatStatisticCounter(inputStream.getIOStatistics(),
+StreamStatisticNames.STREAM_READ_OPENED).isEqualTo(1);
+ } else {
+// For S3InputStream, verify > 1 call was made,
Review Comment:
should this be For S3AInputStream ?
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java:
##
@@ -192,6 +197,12 @@ private boolean isObjectStreamOpen(FSDataInputStream
inputStream) {
return ((S3AInputStream)
inputStream.getWrappedStream()).isObjectStreamOpen();
}
+ private void skipIfCannotUnbuffer(InputStream inputStream) {
Review Comment:
You should just be able to use
`inputStream.hasCapability(StreamCapabilities.UNBUFFER)` instead of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He reassigned HADOOP-18216:
Assignee: Jingxuan Fu
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18216) Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to data read/write blockage
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527943#comment-17527943
]
Xiaoqiao He commented on HADOOP-18216:
--
[~fujx] Thanks for raising this issue. Add to contributor list and assign this
jira to you.
> Ensure "io.file.buffer.size" is greater than zero. Otherwise, it will lead to
> data read/write blockage
> --
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
