[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817184#comment-13817184 ] Hudson commented on HDFS-5364: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #386 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/386/]) HDFS-5364. Add OpenFileCtx cache. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1539834) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestOutOfOrderWrite.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestOpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.2.1 Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch, HDFS-5364.009.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817253#comment-13817253 ] Hudson commented on HDFS-5364: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1603 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1603/]) HDFS-5364. Add OpenFileCtx cache. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1539834) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestOutOfOrderWrite.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestOpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.2.1 Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch, HDFS-5364.009.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817283#comment-13817283 ] Hudson commented on HDFS-5364: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1577 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1577/]) HDFS-5364. Add OpenFileCtx cache. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1539834) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestOutOfOrderWrite.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestOpenFileCtxCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.2.1 Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch, HDFS-5364.009.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816367#comment-13816367
]
Brandon Li commented on HDFS-5364:
--
{quote}2 and 3 are optimization of the eviction method. As we discussed
offline, I will file a following up JIRA for that.{quote}
The new patch added the optimization of the eviction method. Also the scan()
method is not holding the lock all the time. A unit test is added to test the
scan() method.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch,
HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816399#comment-13816399
]
Hadoop QA commented on HDFS-5364:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612683/HDFS-5364.008.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5356//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5356//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch,
HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816421#comment-13816421 ] Jing Zhao commented on HDFS-5364: - Thanks for addressing all the comments, Brandon! The new patch looks good to me. +1 pending Jenkins. Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch, HDFS-5364.009.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816753#comment-13816753
]
Hadoop QA commented on HDFS-5364:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612696/HDFS-5364.009.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5358//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch,
HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch,
HDFS-5364.009.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816912#comment-13816912 ] Brandon Li commented on HDFS-5364: -- Thank you, Jing. I've committed the patch. Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.2.1 Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch, HDFS-5364.007.patch, HDFS-5364.008.patch, HDFS-5364.009.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814335#comment-13814335 ] Brandon Li commented on HDFS-5364: -- Thanks for the review. 1. done 2 and 3 are optimization of the eviction method. As we discussed offline, I will file a following up JIRA for that. 4. done. The lock needs to be held there to synchronize with insert operation. 5. done. nice catch! 6. done. Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch, HDFS-5364.006.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814357#comment-13814357
]
Hadoop QA commented on HDFS-5364:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612265/HDFS-5364.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5338//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5338//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5338//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch,
HDFS-5364.006.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814459#comment-13814459
]
Hadoop QA commented on HDFS-5364:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612290/HDFS-5364.007.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5339//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5339//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch,
HDFS-5364.006.patch, HDFS-5364.007.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813469#comment-13813469
]
Hadoop QA commented on HDFS-5364:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612061/HDFS-5364.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5330//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5330//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5330//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813521#comment-13813521
]
Hadoop QA commented on HDFS-5364:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612080/HDFS-5364.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5331//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5331//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch, HDFS-5364.004.patch, HDFS-5364.005.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805094#comment-13805094
]
Jing Zhao commented on HDFS-5364:
-
Some comments so far:
# OpenFileCtxCache#scan holds the monitor of OpenFileCtxCache. In the
meanwhile, it calls OpenFileCtx#streamCleanup, which may be blocked for
relatively long time in some I/O ops. This can block new OpenFileCtx from being
added to the cache.
# It may be better to move class StreamMonitor from WriteManager to
OpenFileCtxCache.
# In WriteManager.java,
{code}
FileHandle handle = request.getHandle();
if (LOG.isDebugEnabled()) {
LOG.debug(handleWrite fileId: + handle.getFileId() + offset:
- + offset + length: + count + stableHow: +
stableHow.getValue());
+ + offset + length: + count + stableHow: + stableHow.name());
}
{code}
Do you want to use the new added WRITE3Request#toString here?
# Looks like DFSClient#append does not unwrap the AlreadyBeingCreatedException,
thus we may not be able to catch the AlreadyBeingCreatedException directly in
the following code.
{code}
+ } catch (AlreadyBeingCreatedException e) {
+LOG.warn(Can't append file: + fileIdPath
++ . Possibly the file is being closed. Drop the request:
++ request + , wait for the client to retry...);
+return;
{code}
# If the following code, we first create fos for appending data (through an NN
RPC), then check if we can add a new OpenFileCtx into the cache, if not, we
will close fos (through NN RPC again). So here can we first check the cache
availablity, and create fos only when cache has available spot?
{code}
+ if (!addOpenFileStream(fileHandle, openFileCtx)) {
+LOG.info(Can't add new stream. Close it. Tell client to retry.);
+try {
+ fos.close();
+} catch (IOException e) {
+ LOG.error(Can't close stream for fileId: + handle.getFileId());
+}
{code}
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805879#comment-13805879
]
Hadoop QA commented on HDFS-5364:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610417/HDFS-5364.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5285//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5285//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5285//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
HDFS-5364.003.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803606#comment-13803606 ] Brandon Li commented on HDFS-5364: -- The first patch used guava cache and it turned out to be a bad choice because we can't control the eviction policy. Ideally we should evict the idlest stream. However, our idleness means write access, while guava cache's idleness is cache r/w access. Also, we can't use entry auto-expiration feature provided by guava cache to replace StreamMonitor thread. This is because guava cache uses lazy eviction. When an entry is expired, guava cache evicts it (and then invokes removal-listener to do cleanup) when the data structure is touched again. Uploaded a new patch which uses a simple map to implement the cache. Add OpenFileCtx cache - Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., 1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803635#comment-13803635
]
Hadoop QA commented on HDFS-5364:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609964/HDFS-5364.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5269//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5269//console
This message is automatically generated.
Add OpenFileCtx cache
-
Key: HDFS-5364
URL: https://issues.apache.org/jira/browse/HDFS-5364
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch
NFS gateway can run out of memory when the stream timeout is set to a
relatively long period(e.g., 1 minute) and user uploads thousands of files
in parallel. Each stream DFSClient creates a DataStreamer thread, and will
eventually run out of memory by creating too many threads.
NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
