[jira] [Comment Edited] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781714#comment-16781714
]
BELUGA BEHR edited comment on HDFS-14292 at 3/1/19 2:25 PM:
Hello Team,
Only one test failure on the latest go.
Does anyone have any thoughts on this? Even if someone where to add in a
full-fledged Application/Web server (which would be in big support of), the
author would still need to address the issues I worked through here regarding
Threads being killed off as a signaling device. With what I have here (and
more planned), it should be a lot easier for someone to swap this out in the
future.
Thanks.
was (Author: belugabehr):
Hello Team,
Only one test failure on the latest go.
Does anyone have any thoughts on this? Even if someone where to add in a
full-fledged Application/Web serve (which would be in big support of), the
author would still need to address the issues I worked through here regarding
Threads being killed off as a signaling device. With what I have here (and
more planned), it should be a lot easier for someone to swap this out in the
future.
Thanks.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781714#comment-16781714
]
BELUGA BEHR commented on HDFS-14292:
Hello Team,
Only one test failure on the latest go.
Does anyone have any thoughts on this? Even if someone where to add in a
full-fledged Application/Web serve (which would be in big support of), the
author would still need to address the issues I worked through here regarding
Threads being killed off as a signaling device. With what I have here (and
more planned), it should be a lot easier for someone to swap this out in the
future.
Thanks.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13623) getContentSummary to return ContentSummary without hidden files
[
https://issues.apache.org/jira/browse/HDFS-13623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781689#comment-16781689
]
BELUGA BEHR commented on HDFS-13623:
I would suggest updating the existing:
{code}
public ContentSummary getContentSummary(Path f, boolean includeHidden)
throws IOException
Return the ContentSummary of a given Path.
Parameters:
f - path to use
includeHidden - Whether or not to include hidden files in the return values
{code}
> getContentSummary to return ContentSummary without hidden files
> ---
>
> Key: HDFS-13623
> URL: https://issues.apache.org/jira/browse/HDFS-13623
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
>Affects Versions: 3.1.0
>Reporter: Miklos Szurap
>Priority: Major
>
> Improve the
> [FileSystem.getContentSummary()|http://hadoop.apache.org/docs/r3.1.0/api/org/apache/hadoop/fs/FileSystem.html#getContentSummary-org.apache.hadoop.fs.Path-]
> method to return ContentSummary object with
> "getFileCountWithoutHiddenFiles()" and "getLengthWithoutHiddenFiles()".
> That two new counter should not include hidden files and hidden directories
> (and it's sub-contents).
> {code:java}
> public static final PathFilter HIDDEN_FILES_PATH_FILTER = new PathFilter() {
> public boolean accept(Path p) {
>String name = p.getName();
>return !name.startsWith("_") && !name.startsWith(".");
> }
> };{code}
> This would be especially useful for Hive: to compute table statistics with a
> single {{contentSummary}} call instead of {{globStatus}} (multiple
> {{listStatus}} calls) and iterating over multiple thousand of objects on
> client side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779960#comment-16779960
]
BELUGA BEHR commented on HDFS-14295:
Ya that code could use some TLC. I'm working on [HDFS-14292]. I'll
address it there, or in a follow-up to [HDFS-14292].
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779950#comment-16779950
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Not sure if you saw my last comment. We submitted at the same time.
:)
The DataNode already implements this for the main threadpool. So if anything,
we are bringing the behavior inline with the other pool.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779937#comment-16779937
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Unfortunately, we cannot. The running threads depend on (interact
with) some of these other services, so we need to wait for these threads to
complete (max 30s) before closing down the other services. There are going to
be a lot of race conditions if these threads are running while the rest of the
services are being closed.
This may be worth its own JIRA: To ensure that all of the components are being
shutdown in a proper order.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779943#comment-16779943
]
BELUGA BEHR commented on HDFS-14295:
Also note, that this already happens for the main thread pool. There is a
"sleep until threads are done":
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2061-L2084
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779376#comment-16779376
]
BELUGA BEHR commented on HDFS-14292:
Also, the PR is up to date if it's easier to view there.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779363#comment-16779363
]
BELUGA BEHR edited comment on HDFS-14292 at 2/27/19 2:37 PM:
-
Alright, well, there are still three tests failing, but those are known to be
flaky. The good news is that I fixed up
{{hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes}} in this patch and
have not seen it fail since (knock on wood). Please consider this patch
[^HDFS-14292.8.patch] for inclusion into the project.
was (Author: belugabehr):
Alright, well, there are still three tests failing, but those are known to be
flaky. The good news is that I fixed up
{{hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes}} in this patch and
have not seen it fail since (knock on wood). Please consider this patch
[^HDFS-14295.8.patch] for inclusion into the project.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779375#comment-16779375
]
BELUGA BEHR commented on HDFS-14292:
Update: Woops. The last patch I submitted was for a different JIRA. I'm
submitting the {{.8.patch}} again.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: (was: HDFS-14292.8.patch)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.8.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: (was: HDFS-14295.8.patch)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779363#comment-16779363
]
BELUGA BEHR commented on HDFS-14292:
Alright, well, there are still three tests failing, but those are known to be
flaky. The good news is that I fixed up
{{hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes}} in this patch and
have not seen it fail since (knock on wood). Please consider this patch
[^HDFS-14295.8.patch] for inclusion into the project.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.8.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778844#comment-16778844
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri]
So, with this patch, the threads still are Daemon threads.
{code:java}
this.xferService =
Executors.newCachedThreadPool(new Daemon.DaemonFactory())
{code}
I think that it is most polite to give the threads some time to finish. Yes,
they are daemon threads and therefore will not hold up the JVM from an exit,
but it still puts some stress on the active clients as they will have to
perform a re-try once their service thread is terminated. I think as-is is the
best approach. Block new requests but allow exiting client tasks some glimmer
of hope of succeeding.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778533#comment-16778533
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Thanks for the clarification. It seems to me to be 'too much' to
separate the two calls to {{shutdown}}. If we move the first call higher up in
the overall DN shutdown process, then we'll probably want to mark the time
stamp of that interaction so that it can be calculated how much time has passed
between the {{shutdown}} call and the {{awaitTermination}} call. This would
allow the software to wait: 30s - Xs where X is the amount of time that elapsed
between the two calls. This ensure that the shutdown requires no more than 30s
between the two calls. However, I'm not sure how much time would even pass
between the two calls, 1s? 2s? 5s? Seems like too much complexity and
overheard in the code to save a few seconds on shutdown (which rarely occurs).
Regarding the logging, it's not so obvious. The JavaDoc states regarding
{{shutdownNow}}:
bq. Attempts to stop all actively executing tasks, halts the processing of
waiting tasks, and returns a list of the tasks that were *awaiting* execution.
So, the list that is returned is not a list of running tasks that got killed.
It is a list of all the tasks remaining in the queue. However, note, that this
current setup does not queue any tasks. The maximum number of running threads
is 4096 (by default). If anything comes in after that, it will be rejected,
not queued. Therefore, I would not expect this method to ever return anything.
Please note that any threads that are still running will be interrupted by the
call to {{shutdownNow}} and will most likely generate some sort of "I've been
interrupted. Exiting." log message from within each Thread.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: (was: HDFS-14292.6.patch)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14295.8.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778517#comment-16778517
]
BELUGA BEHR commented on HDFS-14293:
[~shwetayakkali] Can you please lend me a hand in getting [HDFS-14294]
approved? I think that will be a good start.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
OK. I believe these last two unit tests are flaky and not related to the
changes in this patch. I have just submitted the patch once more to see what
we get. Also, the checkstyle errors are things that are a bit beyond my
control,... i.e., the {{Unused Import}} is already in the code, not caused by
this patch, and some of the line-length issues are due to the existing formats
and to change my code would make it not match the current stuff.
Please consider this patch for inclusion into the project.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch,
> HDFS-14292.8.patch, HDFS-14295.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778503#comment-16778503
]
BELUGA BEHR edited comment on HDFS-14295 at 2/26/19 7:45 PM:
-
[~elgoiri] The way it is setup in the patch is the correct way to interact with
a thread pool. Calling {{shutdownNow()}} is the last resort, not the first
option. Shut-down-now will interrupt the running threads whereas calling
{{shutdown}} simply blocks new requests from entering the queue and allows the
existing threads some time to finish their work without interrupting them.
Like I said, if there's some flaw in the code here, the most likely outcome is
that the threads are killed off early, which is the behavior the DataNode faces
currently.
was (Author: belugabehr):
[~elgoiri] The way it is setup in the patch is the correct way to interact with
a thread pool. Calling {{shutdownNow()}} is the last resort, not the first
option. Shut-down-now will interrupt the running threads whereas calling
{{shutdown}} simply blocks new requests from entering the queue and allows the
existing threads some time to finish their work without interrupting them.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778503#comment-16778503
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] The way it is setup in the patch is the correct way to interact with
a thread pool. Calling {{shutdownNow()}} is the last resort, not the first
option. Shut-down-now will interrupt the running threads whereas calling
{{shutdown}} simply blocks new requests from entering the queue and allows the
existing threads some time to finish their work without interrupting them.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.8.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
[~elgoiri] Ugh. Thanks for pointing that out. New patch provided.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch, HDFS-14295.8.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
Sorry. Posted a new patch. Last one had a typo in the logging message.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.7.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: (was: HDFS-14252.7.patch)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14252.7.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch,
> HDFS-14295.6.patch, HDFS-14295.7.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778059#comment-16778059
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Latest patch has logging.
The new behavior is that it waits for at most 30 seconds on shutdown to allow
existing work to finish gracefully. I'm not sure we want to enforce the
behavior with a test. The current behavior is to let the JVM kill the daemon
thread on the way out, so it is very low risk that this behavior will be worse
than that.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.6.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
With the last patch I provided, the
{{hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes}} unit test failed.
It's a bit weird because the test itself failed with a NPE. I took a look at
the test and it looks a bit flaky so I tightened it up a bit in this latest
patch.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.8.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch, HDFS-14292.8.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: (was: HDFS-14295.5.patch)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.6.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.6.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.6.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.6.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.6.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777160#comment-16777160
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Hello!
I'm sorry, but I don't quite follow. What is the ask regarding a 30 second
timeout?
There is logging added in the patch.
{code}
LOG.info("Waiting up to 30 seconds for transfer threads to complete");
{code}
I can add logging for {{shutdownNow()}}
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775165#comment-16775165
]
BELUGA BEHR edited comment on HDFS-14292 at 2/22/19 2:23 PM:
-
Hello Watchers.
There are three 3 checkstyle warnings. Please ignore these. To clear one of
them, the fix breaks unit tests. The others are minor infractions and are
artifacts of the existing code base.
The unit test "TestNamenodeCapacityReport" is failing because it is getting
incorrect XCeiver numbers. I'm not sure where this failure is coming from, it
passes on my local machine, but it could be addressed by other changes such as
HDFS-14295 because the thread group that this thread pool uses is also used in
other areas of the DataNode and therefore are not controlled or limited by this
pool.
Please consider the latest patch for inclusion into the project.
PR on GitHub is updated with the latest proposed changes.
was (Author: belugabehr):
Hello Watchers.
There are three 3 checkstyle warnings. Please ignore these. To clear one of
them, the fix breaks unit tests. The others are minor infractions and are
artifacts of the existing code base.
The unit test "TestNamenodeCapacityReport" is failing because it is getting
incorrect XCeiver numbers. I'm not sure where this failure is coming from, it
passes on my local machine, but it could be addressed by other changes such as
HDFS-14295 because the thread group that this thread pool uses is also used in
other areas of the DataNode and therefore are not controlled or limited by this
pool.
Please consider the latest patch for inclusion into the project.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775171#comment-16775171
]
BELUGA BEHR commented on HDFS-14293:
[~shwetayakkali] I'm not sure the best way to go about deprecating all of these
various components. However, if you can assist with getting HDFS-14294
accepted, I can start tearing out that piece of the puzzle.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775165#comment-16775165
]
BELUGA BEHR commented on HDFS-14292:
Hello Watchers.
There are three 3 checkstyle warnings. Please ignore these. To clear one of
them, the fix breaks unit tests. The others are minor infractions and are
artifacts of the existing code base.
The unit test "TestNamenodeCapacityReport" is failing because it is getting
incorrect XCeiver numbers. I'm not sure where this failure is coming from, it
passes on my local machine, but it could be addressed by other changes such as
HDFS-14295 because the thread group that this thread pool uses is also used in
other areas of the DataNode and therefore are not controlled or limited by this
pool.
Please consider the latest patch for inclusion into the project.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775153#comment-16775153
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] - Checkstyles and FindBugs are happy again. Unit tests are passed
(save a couple of known-flaky tests). Please consider for inclusion into the
project.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
[~elgoiri] I think the code is simple enough to pass with code-review alone. I
also think that unit tests shouldn't be too concerned with implementation
details like caching.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774692#comment-16774692
]
BELUGA BEHR commented on HDFS-14292:
Pushed a new patch to fix some of the unit tests.
Please note that this patch also includes changes to the {{Peer}} interface to
allow for some cleaner code.
As part of that, the logging has changed a bit and I think it is more clear.
It certainly is closer to a spec that most people would recognize.
{code}
from peer: NioInetPeer [isLocal=true, localURI=hdfs+dn://127.0.0.1:35375,
remoteURI=hdfs+dn://127.0.0.1:38376]
from peer: BasicInetPeer [isLocal=true, localURI=hdfs+dn://127.0.0.1:35375,
remoteURI=hdfs+dn://127.0.0.1:38376]
from peer: DomainPeer [isLocal=true,
localURI=hdfs+dn+unix://127.0.0.1/tmp/socket,
remoteURI=hdfs+dn+unix://127.0.0.1/tmp/socket]
{code}
These are stored in actual {{URI}} objects.
The {{hdfs+dn}} scheme is typical datagram socket stuff (host and port). The
{{hdfs+dn+unix}} specifies that the DataNode is communicating over a Unix
domain socket (file). There is no port obviously, but the URI path is the path
to the socket file.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.5.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.7.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
Ya, [~elgoiri]. I think we're better off with the cast into a Future and then
someday, when the API is fixed, the cast can simply be removed.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.5.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774469#comment-16774469
]
BELUGA BEHR edited comment on HDFS-14293 at 2/21/19 8:00 PM:
-
[~shwetayakkali] They are used all over the place in the code base. Some of
the stuff can be deprecated: [HDFS-14294], and if these two can condense down
into a single configuration, that is also a good thing. However, I don't think
the entire feature can be removed because, as I said, the code does use it a
lot. Just search for {{getIoFileBufferSize}} in the code.
was (Author: belugabehr):
[~shwetayakkali] They are used all over the place in the code base. Some of
the stuff can be deprecated: [HDFS-14294], and we can condense these two down
into a single configuration, that is also a good thing. However, I don't think
the entire feature can be removed because, as I said, the code does use it a
lot. Just search for {{getIoFileBufferSize}} in the code.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774469#comment-16774469
]
BELUGA BEHR commented on HDFS-14293:
[~shwetayakkali] They are used all over the place in the code base. Some of
the stuff can be deprecated: [HDFS-14294], and we can condense these two down
into a single configuration, that is also a good thing. However, I don't think
the entire feature can be removed because, as I said, the code does use it a
lot. Just search for {{getIoFileBufferSize}} in the code.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774456#comment-16774456
]
BELUGA BEHR commented on HDFS-14292:
Posted the same patch again for YETUS. Several of these failed UT pass
locally, so trying again.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.6.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774357#comment-16774357
]
BELUGA BEHR commented on HDFS-14295:
The find-bugs is complaining that we're not waiting on the outcome of the
threads,... which is OK here.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
New patch... Checkstyle said I should make a class final, but that causes
Mockito (and therefore some unit tests) to fail.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.6.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch, HDFS-14292.6.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774098#comment-16774098
]
BELUGA BEHR commented on HDFS-14295:
[~elgoiri] Alright. I've made the requested changes. Thanks again, as always.
Please consider this patch for inclusion into the project.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.5.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.4.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773583#comment-16773583
]
BELUGA BEHR commented on HDFS-14293:
[~shwetayakkali] Same goes for: {{io.file.buffer.size}}
https://github.com/apache/hadoop/blob/d43af8b3db4743b4b240751b6f29de6c20cfd6e5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java#L236-L239
https://github.com/apache/hadoop/blob/fb8932a727f757b2e9c1c61a18145878d0eb77bd/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java#L740-L744
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
I don't think I got all of the UT passing, but could use a new baseline of all
the test having run.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.4.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773208#comment-16773208
]
BELUGA BEHR commented on HDFS-14292:
Ya, finally found it. I'm not sure that this is the issue with all of my
failing unit tests, but it's a start:
{code:java|title=LocalReplicaInPipeline}
Thread thread = writer.get();
if ((thread == null) || (thread == Thread.currentThread()) ||
(!thread.isAlive())) {
if (writer.compareAndSet(thread, null)) {
return; // Done
}
// The writer changed. Go back to the start of the loop and attempt to
// stop the new writer.
continue;
}
thread.interrupt();
try {
thread.join(xceiverStopTimeout);
if (thread.isAlive()) {
// Our thread join timed out.
final String msg = "Join on writer thread " + thread + " timed out";
DataNode.LOG.warn(msg + "\n" + StringUtils.getStackTrace(thread));
throw new IOException(msg);
}
{code}
[https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java#L265-L268]
Interrupts the thread and waits for it to die ({{join}}), only now I've got a
thread pool in my branch so the thread is re-used, it does not die. It waits
here until timeout then fails. I'm not sure what the fix is, but I'm thinking
about it.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773187#comment-16773187
]
BELUGA BEHR commented on HDFS-14292:
[~xkrogen] Thanks for the feedback. Well, I don't know the history, but I too
seemed like it was the obvious choice, but I will tell you that it's not easy
to do... unit tests are failing and I haven't pinned it down quite yet (ugh),
but I think some of the code is expecting that threads are not-reused via
ThreadLocal and other mechanisms. I'm still trying to hunt it down exactly,
but the unit tests pass when I use a thread pool that does not re-use threads.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773187#comment-16773187
]
BELUGA BEHR edited comment on HDFS-14292 at 2/20/19 4:52 PM:
-
[~xkrogen] Thanks for the feedback. Well, I don't know the history, but I too
seemed thought it was the obvious choice, but I will tell you that it's not
easy to do... unit tests are failing and I haven't pinned it down quite yet
(ugh), but I think some of the code is expecting that threads are not-reused
via ThreadLocal and other mechanisms. I'm still trying to hunt it down
exactly, but the unit tests pass when I use a thread pool that does not re-use
threads.
was (Author: belugabehr):
[~xkrogen] Thanks for the feedback. Well, I don't know the history, but I too
seemed like it was the obvious choice, but I will tell you that it's not easy
to do... unit tests are failing and I haven't pinned it down quite yet (ugh),
but I think some of the code is expecting that threads are not-reused via
ThreadLocal and other mechanisms. I'm still trying to hunt it down exactly,
but the unit tests pass when I use a thread pool that does not re-use threads.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772388#comment-16772388
]
BELUGA BEHR commented on HDFS-14295:
{{StringUtils#join}} is not the go-to option here because it requires a
{{Collection}} of Strings. That is not the case here.
A list of objects must be iterated to extract the required String first, then
joined. Nice to be able to do it in one-line.
In regards to the cast, I believe it's a necessary evil because of a mistake in
the Java API:
https://stackoverflow.com/questions/34233375/return-completablefuturevoid-or-completablefuture/34233790
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772387#comment-16772387
]
BELUGA BEHR commented on HDFS-14295:
{{StringUtils#join}} is not the go-to option here because it requires a
{{Collection}} of Strings. That is not the case here.
A list of objects must be iterated to extract the required String first, then
joined. Nice to be able to do it in one-line.
In regards to the cast, I believe it's a necessary evil because of a mistake in
the Java API:
https://stackoverflow.com/questions/34233375/return-completablefuturevoid-or-completablefuture/34233790
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Description:
When a DataNode data transfers a block, is spins up a new thread for each
transfer.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
and
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
Instead, add the threads to a {{CachedThreadPool}} so that when their
threads complete the transfer, they can be re-used for another transfer. This
should save resources spent on creating and spinning up transfer threads.
One thing I'll point out that's a bit off, which I address in this patch, ...
There are two places in the code where a {{DataTransfer}} thread is started. In
[one
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
it's started in a default thread group. In [another
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
it's started in the
[dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
thread group.
I do not think it's correct to include any of these threads in the
{{dataXceiverServer}} thread group. Anything submitted to the
{{dataXceiverServer}} should probably be tied to the
{{dfs.datanode.max.transfer.threads}} configurations, and neither of these
methods are. Instead, they should be submitted into the same thread pool with
its own thread group (probably the default thread group, unless someone
suggests otherwise) and is what I have included in this patch.
was:When a DataNode data transfers a block, is spins up a new thread for each
transfer.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
and
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
Instead, add the threads to a {{CachedThreadPool}} so that when their
threads complete the transfer, they can be re-used for another transfer. This
should save resources spent on creating and spinning up transfer threads.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the sam
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772007#comment-16772007
]
BELUGA BEHR commented on HDFS-14295:
Again, it took me a few tries locally, but I did get it to pass eventually:
{code}
>mvn clean install -Dtest=TestEditLogTailer,TestJournalNodeSync
...
INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ hadoop-hdfs ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.403
s - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
[INFO] Running org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeSync
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.479
s - in org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeSync
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
{code}
Please consider accepting this patch into the product.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.3.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771471#comment-16771471
]
BELUGA BEHR commented on HDFS-14296:
This is a good reference.
https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
[ https://issues.apache.org/jira/browse/HDFS-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771417#comment-16771417 ] BELUGA BEHR commented on HDFS-9596: --- [~giovanni.fumarola] [~elgoiri] Can either of you fine folks finish this one up for me? :) > Remove "Shuffle" Method From DFSUtil > > > Key: HDFS-9596 > URL: https://issues.apache.org/jira/browse/HDFS-9596 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-9596.2.patch, HDFS-9596.3.patch, HDFS-9596.4.patch, > HDFS-9596.5.patch, HDFS-9596.6.patch, HDFS-9596.7.patch, HDFS-9596.7.patch, > HDFS-9596.7.patch, Shuffle.HDFS-9596.patch > > > DFSUtil contains a "shuffle" routine that shuffles arrays. With a little > wrapping, Java Collections class can shuffle. This code is superfluous and > undocumented. The method returns an array, though it does not specify if it > returns the same array or a new, copy, array of shuffled items. > It is only used in one place, so it is better to simply remove and substitute > this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
BELUGA BEHR created HDFS-14296:
--
Summary: Prefer ArrayList over LinkedList in VolumeScanner
Key: HDFS-14296
URL: https://issues.apache.org/jira/browse/HDFS-14296
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Assignee: BELUGA BEHR
Attachments: HDFS-14296.1.patch
{quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
{{listIterator}} operations run in constant time. - ArrayList
{quote}
However, for a {{LinkedList}}, the entire list must be traversed to get to the
desired index.
Like
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
Most of the time, the List is being iterated, which is quicker over a primitive
array than walking a LinkedList.
There is one place where an item is removed, potentially from the middle of the
list
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
but the speed of removing from the middle of the list isn't bad; it's a system
native array shift, and it only happens on the off chance that a block pool is
removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14296:
---
Status: Patch Available (was: Open)
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
[
https://issues.apache.org/jira/browse/HDFS-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14296:
---
Attachment: HDFS-14296.1.patch
> Prefer ArrayList over LinkedList in VolumeScanner
> -
>
> Key: HDFS-14296
> URL: https://issues.apache.org/jira/browse/HDFS-14296
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14296.1.patch
>
>
> {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
> {{listIterator}} operations run in constant time. - ArrayList
> {quote}
> However, for a {{LinkedList}}, the entire list must be traversed to get to
> the desired index.
> Like
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
> Most of the time, the List is being iterated, which is quicker over a
> primitive array than walking a LinkedList.
> There is one place where an item is removed, potentially from the middle of
> the list
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
> but the speed of removing from the middle of the list isn't bad; it's a
> system native array shift, and it only happens on the off chance that a block
> pool is removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771385#comment-16771385
]
BELUGA BEHR edited comment on HDFS-14295 at 2/18/19 9:25 PM:
-
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Check it out
here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
was (Author: belugabehr):
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771385#comment-16771385
]
BELUGA BEHR commented on HDFS-14295:
Also note that this means that some of these threads are being counted in the
MXBean, so you could see cases where the MXBean reported a number higher than
{{dfs.datanode.max.transfer.thread}} which is a bit confusing.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2197-L2201]
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771376#comment-16771376
]
BELUGA BEHR commented on HDFS-14295:
One thing I'll point out that's a bit off, which I address in this patch, ...
There are two places in the code where a {{DataTransfer}} thread is started. In
[one
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
it's started in a default thread group. In [another
place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
it's started in the
[dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
thread group.
I do not think it's correct to include any of these threads in the
{{dataXceiverServer}} thread group. Anything submitted to the
{{dataXceiverServer}} should probably be tied to the
{{dfs.datanode.max.transfer.threads}} configurations, and neither of these
methods are. Instead, they should be submitted into the same thread pool with
its own thread group (probably the default thread group, unless someone
suggests otherwise) and is what I have included in this patch.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14294) Deprecate DFSUtilClient#getSmallBufferSize
[
https://issues.apache.org/jira/browse/HDFS-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771378#comment-16771378
]
BELUGA BEHR commented on HDFS-14294:
{{javac}} is complaining about the use of deprecated methods... which makes
sense since this patch proposes to deprecate a method that previously was not.
> Deprecate DFSUtilClient#getSmallBufferSize
> --
>
> Key: HDFS-14294
> URL: https://issues.apache.org/jira/browse/HDFS-14294
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14294.1.patch
>
>
> {code:java|title=DFSUtilClient.java}
> public static int getIoFileBufferSize(Configuration conf) {
> return conf.getInt(
> CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY,
> CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT);
> }
> public static int getSmallBufferSize(Configuration conf) {
> return Math.min(getIoFileBufferSize(conf) / 2, 512);
> }
> {code}
> This concept of a "small buffer size" seems a bit overkill. First of all,
> it's not documented that such a thing exists and that by adjusting
> {{dfs.stream-buffer-size}} an administrator is also scaling these other
> buffer sizes. Seconds, I think any "small" buffer size should just use the
> default JDK buffer sizes. Anything that benefits from being larger than the
> default JDK size should be the controlled by {{IO_FILE_BUFFER_SIZE_KEY}} /
> {{dfs.stream-buffer-size}}. For reference, the default JDK size is 8K :
> [HDFS-14293]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
