[jira] [Created] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message
GAO Rui created HDFS-8936: - Summary: Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message Key: HDFS-8936 URL: https://issues.apache.org/jira/browse/HDFS-8936 Project: Hadoop HDFS Issue Type: Sub-task Reporter: GAO Rui Assignee: GAO Rui When a EC directory exceed DiskSpace quota, the error message is along with DFSStripedOutputStream inner exception message. Error messages should be as simple and clear as normal hdfs directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8388) Time and Date format need to be in sync in Namenode UI page
[ https://issues.apache.org/jira/browse/HDFS-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706512#comment-14706512 ] Surendra Singh Lilhore commented on HDFS-8388: -- Thanks [~vinayrpet] for review bq. I feel, if we are unifying, then unify everywhere. agree? yes I will update the patch. > Time and Date format need to be in sync in Namenode UI page > --- > > Key: HDFS-8388 > URL: https://issues.apache.org/jira/browse/HDFS-8388 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Minor > Attachments: HDFS-8388-002.patch, HDFS-8388-003.patch, > HDFS-8388-004.patch, HDFS-8388.patch, HDFS-8388_1.patch, > ScreenShot-InvalidDate.png > > > In NameNode UI Page, Date and Time FORMAT displayed on the page are not in > sync currently. > Started:Wed May 13 12:28:02 IST 2015 > Compiled:23 Apr 2015 12:22:59 > Block Deletion Start Time 13 May 2015 12:28:02 > We can keep a common format in all the above places. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8932) NPE thrown in NameNode when try to get "TotalSyncCount" metric before editLogStream initialization
[ https://issues.apache.org/jira/browse/HDFS-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706506#comment-14706506 ] Surendra Singh Lilhore commented on HDFS-8932: -- Thanks [~vinayrpet] for review > NPE thrown in NameNode when try to get "TotalSyncCount" metric before > editLogStream initialization > -- > > Key: HDFS-8932 > URL: https://issues.apache.org/jira/browse/HDFS-8932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > Attachments: HDFS-8932.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-6939: - Status: Patch Available (was: Open) > Support path-based filtering of inotify events > -- > > Key: HDFS-6939 > URL: https://issues.apache.org/jira/browse/HDFS-6939 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, qjm >Reporter: James Thomas >Assignee: Surendra Singh Lilhore > Attachments: HDFS-6939-001.patch > > > Users should be able to specify that they only want events involving > particular paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[
https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706503#comment-14706503
]
Surendra Singh Lilhore commented on HDFS-6939:
--
>From inotify design doc.
bq. 1. Allow clients to watch events from specific directories rather than from
the entirety of HDFS. We will need to decide whether undesired events should be
filtered out on the NameNode or client side.
I think its good to add filtering on the Namenode side. Client side user only
can do filtering in his application.
*Patch 001:*
1. Added *Regex* based filterting, User can pass regular expression for
filtering.
2. User can watch events for perticular file, by passing file path.
3. Added two new API in *DistributedFileSystem*
{noformat}
public DFSInotifyEventInputStream
getInotifyEventStream(InotifyFilter filter);
public DFSInotifyEventInputStream getInotifyEventStream(long
lastReadTxid, InotifyFilter filter)
{noformat}
4. In feature we can add more filter option in *InotifyFilter* object
without changing API, for example event based filtering HDFS-8933
Please review and give suggestions .
> Support path-based filtering of inotify events
> --
>
> Key: HDFS-6939
> URL: https://issues.apache.org/jira/browse/HDFS-6939
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode, qjm
>Reporter: James Thomas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-6939-001.patch
>
>
> Users should be able to specify that they only want events involving
> particular paths.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-6939: - Attachment: HDFS-6939-001.patch > Support path-based filtering of inotify events > -- > > Key: HDFS-6939 > URL: https://issues.apache.org/jira/browse/HDFS-6939 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, qjm >Reporter: James Thomas >Assignee: Surendra Singh Lilhore > Attachments: HDFS-6939-001.patch > > > Users should be able to specify that they only want events involving > particular paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706489#comment-14706489 ] Vinayakumar B commented on HDFS-7116: - bq. 1. Would you remove the sentence starts with "The new value is not persistent ... "? If we add the metric, the new value does persist on the DataNodeMetrics. IMO, exposing via metric is not same as persisting. It just shows out the current value. Here not actually meant, new value is only till lifetime of the datanode. On restart it will be reset. so i feel still NOTE is valid. May be the sentence related to exposing can be added before the note. To make it clear. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706489#comment-14706489 ] Vinayakumar B edited comment on HDFS-7116 at 8/21/15 9:57 AM: -- bq. 1. Would you remove the sentence starts with "The new value is not persistent ... "? If we add the metric, the new value does persist on the DataNodeMetrics. IMO, exposing via metric is not same as persisting. It just shows out the current value. Here NOTE actually meant, new value is only till lifetime of the datanode. On restart it will be reset. so i feel still NOTE is valid. May be the sentence related to exposing can be added before the note. To make it clear. was (Author: vinayrpet): bq. 1. Would you remove the sentence starts with "The new value is not persistent ... "? If we add the metric, the new value does persist on the DataNodeMetrics. IMO, exposing via metric is not same as persisting. It just shows out the current value. Here not actually meant, new value is only till lifetime of the datanode. On restart it will be reset. so i feel still NOTE is valid. May be the sentence related to exposing can be added before the note. To make it clear. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8388) Time and Date format need to be in sync in Namenode UI page
[ https://issues.apache.org/jira/browse/HDFS-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706481#comment-14706481 ] Vinayakumar B commented on HDFS-8388: - Patch looks great. I feel there are some other places in NameNode UI where this unification could be done. ex: Datanode last contact time, file last modification time, datanode volume failure time, etc. I feel, if we are unifying, then unify everywhere. agree? > Time and Date format need to be in sync in Namenode UI page > --- > > Key: HDFS-8388 > URL: https://issues.apache.org/jira/browse/HDFS-8388 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Minor > Attachments: HDFS-8388-002.patch, HDFS-8388-003.patch, > HDFS-8388-004.patch, HDFS-8388.patch, HDFS-8388_1.patch, > ScreenShot-InvalidDate.png > > > In NameNode UI Page, Date and Time FORMAT displayed on the page are not in > sync currently. > Started:Wed May 13 12:28:02 IST 2015 > Compiled:23 Apr 2015 12:22:59 > Block Deletion Start Time 13 May 2015 12:28:02 > We can keep a common format in all the above places. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8829) DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
[
https://issues.apache.org/jira/browse/HDFS-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HDFS-8829:
Attachment: HDFS-8829.0002.patch
Added configuration to hdfs-default.xml
> DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
> ---
>
> Key: HDFS-8829
> URL: https://issues.apache.org/jira/browse/HDFS-8829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.3.0, 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
> Attachments: HDFS-8829.0001.patch, HDFS-8829.0002.patch
>
>
> {code:java}
> private void initDataXceiver(Configuration conf) throws IOException {
> // find free port or use privileged port provided
> TcpPeerServer tcpPeerServer;
> if (secureResources != null) {
> tcpPeerServer = new TcpPeerServer(secureResources);
> } else {
> tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
> DataNode.getStreamingAddr(conf));
> }
>
> tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
> {code}
> The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on
> some system.
> Shall we make this behavior configurable?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706461#comment-14706461 ] Rakesh R commented on HDFS-7116: OK, thank you for the explanation. Attached patch addressing the comments. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-7116: --- Attachment: HDFS-7116-05.patch > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8892) ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
[
https://issues.apache.org/jira/browse/HDFS-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706455#comment-14706455
]
Ravikumar commented on HDFS-8892:
-
Iterating over all replicas is a cause of concern. Is it possible for client to
ack a slot made invalid by DN? But my guess is that, it would needlessly
complicate this issue.
May be the patch itself is applicable for specific use-cases…
How about exposing a boolean config
"dfs.client.read.shortcircuit.replica.stale.cleanup" and wrapping this small
patch in a conditional block?
> ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
> -
>
> Key: HDFS-8892
> URL: https://issues.apache.org/jira/browse/HDFS-8892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.7.1
>Reporter: Ravikumar
>Assignee: kanaka kumar avvaru
>Priority: Minor
>
> Currently CacheCleaner thread checks only for cache-expiry times. It would be
> nice if it handles an invalid-slot too in an extra-pass of evictable map…
> for(ShortCircuitReplica replica:evictable.values()) {
> if(!scr.getSlot().isValid()) {
> purge(replica);
> }
> }
> //Existing code...
> int numDemoted = demoteOldEvictableMmaped(curMs);
> int numPurged = 0;
> Long evictionTimeNs = Long.valueOf(0);
> ….
> …..
> Apps like HBase can tweak the expiry/staleness/cache-size params in
> DFS-Client, so that ShortCircuitReplica will never be closed except when Slot
> is declared invalid.
> I assume slot-invalidation will happen during block-invalidation/deletes
> {Primarily triggered by compaction/shard-takeover etc..}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[
https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706452#comment-14706452
]
Akira AJISAKA commented on HDFS-7116:
-
Thanks [~rakeshr] for updating the patch.
{code}
> -| `-setBalancerBandwidth` \ | Changes the
> network bandwidth used by each datanode during HDFS block balancing.
> \ is the maximum number of bytes per second that will be used by
> each datanode. This value overrides the dfs.balance.bandwidthPerSec
> parameter.NOTE: The new value is not persistent on the DataNode. |
> +| `-setBalancerBandwidth` \ | Changes the
> network bandwidth used by each datanode during HDFS block balancing.
> \ is the maximum number of bytes per second that will be used by
> each datanode. This value overrides the dfs.balance.bandwidthPerSec
> parameter.NOTE: The new value is not persistent on the DataNode. The
> value is exposed via BalancerBandwidth metric. |
{code}
1. Would you remove the sentence starts with "The new value is not persistent
... "? If we add the metric, the new value does persist on the DataNodeMetrics.
2. (minor) Would you replace {{0xC2A0}} (whitespace in UTF-8) before "NOTE:"
with whitespace as well?
> Add a metric to expose the bandwidth of balancer
>
>
> Key: HDFS-7116
> URL: https://issues.apache.org/jira/browse/HDFS-7116
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: balancer & mover
>Reporter: Akira AJISAKA
>Assignee: Rakesh R
> Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch,
> HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch
>
>
> Now reading logs is the only way to check how the balancer bandwidth is set.
> It would be useful for administrators if they can get the value of the same.
> This jira to discuss & implement the way to access the balancer bandwidth
> value of the datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706441#comment-14706441 ] Rakesh R commented on HDFS-7116: Attached another patch for the HDFSCommands doc changes. Thanks again! > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-7116: --- Attachment: HDFS-7116-04.patch > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8853:
-
Attachment: HDFS-8853-HDFS-7285-merge-04.patch
Thanks [~vinayrpet] and [~zhz] for the review comments.
Updated the patch . Please review.
Have raised a separate jira for passing policy name for creating EC zone
(HDFS-8935)
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-merge-02.patch, HDFS-8853-HDFS-7285-merge-03.patch,
> HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8935) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy
J.Andreina created HDFS-8935: Summary: Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy Key: HDFS-8935 URL: https://issues.apache.org/jira/browse/HDFS-8935 Project: Hadoop HDFS Issue Type: Sub-task Reporter: J.Andreina Assignee: J.Andreina Current behavior : User has to specify ErasureCodingPolicy as an argument for createErasureCodingZone api . This can be made in sync with creation of EC zone through CLI , where user need to specify only the policy name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Open (was: Patch Available) > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Patch Available (was: Open) to trigger jenkins > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[
https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706408#comment-14706408
]
Akira AJISAKA commented on HDFS-7116:
-
Thanks [~rakeshr] for updating the patch. I built a pseudo-distributed cluster
with your patch and the metric worked well.
One comment: Would you update the description of "dfsadmin
-setBalancerBandwidth" in HDFSCommands document?
{code}
NOTE: The new value is not persistent on the DataNode.
{code}
Now the new value is exposed via BalancerBandwidth metric.
I'm +1 if that is addressed.
> Add a metric to expose the bandwidth of balancer
>
>
> Key: HDFS-7116
> URL: https://issues.apache.org/jira/browse/HDFS-7116
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: balancer & mover
>Reporter: Akira AJISAKA
>Assignee: Rakesh R
> Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch,
> HDFS-7116-02.patch, HDFS-7116-03.patch
>
>
> Now reading logs is the only way to check how the balancer bandwidth is set.
> It would be useful for administrators if they can get the value of the same.
> This jira to discuss & implement the way to access the balancer bandwidth
> value of the datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8899) Erasure Coding: use threadpool for EC recovery tasks
[
https://issues.apache.org/jira/browse/HDFS-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706398#comment-14706398
]
Rakesh R commented on HDFS-8899:
Test case failures and findbug warnings are not related to the patch, please
ignore it.
> Erasure Coding: use threadpool for EC recovery tasks
>
>
> Key: HDFS-8899
> URL: https://issues.apache.org/jira/browse/HDFS-8899
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8899-HDFS-7285-merge-00.patch
>
>
> The idea is to use threadpool for processing erasure coding recovery tasks at
> the datanode.
> {code}
> new Daemon(new ReconstructAndTransferBlock(recoveryInfo)).start();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706361#comment-14706361
]
Zhe Zhang commented on HDFS-8909:
-
Jenkins [job | https://builds.apache.org/job/Hadoop-HDFS-8909/] started.
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706336#comment-14706336 ] Vinayakumar B commented on HDFS-7116: - Thanks @Akira for confirming. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706338#comment-14706338 ] Rakesh R commented on HDFS-7116: Thanks [~ajisakaa], [~vinayrpet] for the comments. Attached another patch addressing the comments. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-7116: --- Attachment: HDFS-7116-03.patch > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch, HDFS-7116-03.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8829) DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
[
https://issues.apache.org/jira/browse/HDFS-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706335#comment-14706335
]
Vinayakumar B commented on HDFS-8829:
-
Changes looks great.
1. Need to add this new configuration in the {{hdfs-default.xml}} and In the
description need to mention about AutoTuning, it would help users. I believe
there is also a testcase which will fail, if entry not found in xml.
> DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
> ---
>
> Key: HDFS-8829
> URL: https://issues.apache.org/jira/browse/HDFS-8829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.3.0, 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
> Attachments: HDFS-8829.0001.patch
>
>
> {code:java}
> private void initDataXceiver(Configuration conf) throws IOException {
> // find free port or use privileged port provided
> TcpPeerServer tcpPeerServer;
> if (secureResources != null) {
> tcpPeerServer = new TcpPeerServer(secureResources);
> } else {
> tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
> DataNode.getStreamingAddr(conf));
> }
>
> tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
> {code}
> The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on
> some system.
> Shall we make this behavior configurable?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706329#comment-14706329 ] Akira AJISAKA commented on HDFS-7116: - bq. Can someone confirm whether adding a new metric is a compatible change? Adding a new metric is compatible. http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#MetricsJMX > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706326#comment-14706326 ] Vinayakumar B commented on HDFS-7116: - Thanks [~rakeshr], Patch looks great for adding the metric. This is the simple way to expose the bandwidth value. +1 for the patch. Can someone confirm whether adding a new metric is a compatible change? I am asking because, a new entry will show up in jmx json. If incompatible, need to mark this jira as incompatible. > Add a metric to expose the bandwidth of balancer > > > Key: HDFS-7116 > URL: https://issues.apache.org/jira/browse/HDFS-7116 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Akira AJISAKA >Assignee: Rakesh R > Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, > HDFS-7116-02.patch > > > Now reading logs is the only way to check how the balancer bandwidth is set. > It would be useful for administrators if they can get the value of the same. > This jira to discuss & implement the way to access the balancer bandwidth > value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[
https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706325#comment-14706325
]
Akira AJISAKA commented on HDFS-7116:
-
Thanks [~rakeshr] for creating the patch.
{code}
private AtomicLong balancerBandwidth = new AtomicLong(0);
{code}
There is a class {{MutableGaugeLong}}, which is similar to
{{MutableCounterLong}} and can set arbitrary long value. Would you use the
class instead of {{AtomicLong}}? The rest of the patch looks good to me.
> Add a metric to expose the bandwidth of balancer
>
>
> Key: HDFS-7116
> URL: https://issues.apache.org/jira/browse/HDFS-7116
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: balancer & mover
>Reporter: Akira AJISAKA
>Assignee: Rakesh R
> Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch,
> HDFS-7116-02.patch
>
>
> Now reading logs is the only way to check how the balancer bandwidth is set.
> It would be useful for administrators if they can get the value of the same.
> This jira to discuss & implement the way to access the balancer bandwidth
> value of the datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
