[jira] [Updated] (HDFS-12783) [branch-2] "dfsrouter" should use hdfsScript
[ https://issues.apache.org/jira/browse/HDFS-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12783: Attachment: HDFS-12783-branch-2.patch Uploaded the patch.Kindly review > [branch-2] "dfsrouter" should use hdfsScript > > > Key: HDFS-12783 > URL: https://issues.apache.org/jira/browse/HDFS-12783 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula > Attachments: HDFS-12783-branch-2.patch > > > *when we start "dfsrouter" with "hadoop-daemon.sh"* it will fail with > following error (Found during 2.9 verification) > brahma@brahma:/opt/hadoop-2.9.0/sbin$ ./hadoop-daemon.sh start dfsrouter > starting dfsrouter, logging to > /opt/hadoop-2.9.0/logs/hadoop-brahma-dfsrouter-brahma.out > Error: Could not find or load main class dfsrouter -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12783) [branch-2] "dfsrouter" should use hdfsScript
[ https://issues.apache.org/jira/browse/HDFS-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-12783: --- Assignee: Brahma Reddy Battula > [branch-2] "dfsrouter" should use hdfsScript > > > Key: HDFS-12783 > URL: https://issues.apache.org/jira/browse/HDFS-12783 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12783-branch-2.patch > > > *when we start "dfsrouter" with "hadoop-daemon.sh"* it will fail with > following error (Found during 2.9 verification) > brahma@brahma:/opt/hadoop-2.9.0/sbin$ ./hadoop-daemon.sh start dfsrouter > starting dfsrouter, logging to > /opt/hadoop-2.9.0/logs/hadoop-brahma-dfsrouter-brahma.out > Error: Could not find or load main class dfsrouter -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12783) [branch-2] "dfsrouter" should use hdfsScript
[ https://issues.apache.org/jira/browse/HDFS-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12783: Status: Patch Available (was: Open) > [branch-2] "dfsrouter" should use hdfsScript > > > Key: HDFS-12783 > URL: https://issues.apache.org/jira/browse/HDFS-12783 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula > Attachments: HDFS-12783-branch-2.patch > > > *when we start "dfsrouter" with "hadoop-daemon.sh"* it will fail with > following error (Found during 2.9 verification) > brahma@brahma:/opt/hadoop-2.9.0/sbin$ ./hadoop-daemon.sh start dfsrouter > starting dfsrouter, logging to > /opt/hadoop-2.9.0/logs/hadoop-brahma-dfsrouter-brahma.out > Error: Could not find or load main class dfsrouter -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12783) [branch-2] "dfsrouter" should use hdfsScript
Brahma Reddy Battula created HDFS-12783: --- Summary: [branch-2] "dfsrouter" should use hdfsScript Key: HDFS-12783 URL: https://issues.apache.org/jira/browse/HDFS-12783 Project: Hadoop HDFS Issue Type: Bug Reporter: Brahma Reddy Battula *when we start "dfsrouter" with "hadoop-daemon.sh"* it will fail with following error (Found during 2.9 verification) brahma@brahma:/opt/hadoop-2.9.0/sbin$ ./hadoop-daemon.sh start dfsrouter starting dfsrouter, logging to /opt/hadoop-2.9.0/logs/hadoop-brahma-dfsrouter-brahma.out Error: Could not find or load main class dfsrouter -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12781) After Datanode down, In Namenode UI Datanode tab is throwing warning message.
[
https://issues.apache.org/jira/browse/HDFS-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239944#comment-16239944
]
Brahma Reddy Battula commented on HDFS-12781:
-
[~Harsha1206] thanks for reporting.
As per [datable doc|https://datatables.net/manual/tech-notes/4], Each cell in
DataTables requests data, and when DataTables tries to obtain data for a cell
and is unable to do so, it will trigger a warning, telling you that data is not
available where it was expected to be.
when node is dead,as we can't get the values() it through's this warn.
When I give {{defaultContent}},this error will disappear.I feel, we should add
{{defaultContent}} to all the cells.
{code}
{ 'orderDataType': 'ng-value', 'searchable': true , "defaultContent":
"" },
{ 'orderDataType': 'ng-value', 'searchable': true ,
"defaultContent": ""},
{ 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent":
0},
{ 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent":
0},
{ 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent":
0},
{ 'type': 'num' , "defaultContent": 0},
{ 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent":
0},
{ 'type': 'string' , "defaultContent": ""}
{code}
> After Datanode down, In Namenode UI Datanode tab is throwing warning message.
> -
>
> Key: HDFS-12781
> URL: https://issues.apache.org/jira/browse/HDFS-12781
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>
> Scenario:
> Stop one Datanode
> Refresh or click on the Datanode tab in namenode UI.
> Actual Output:
> ==
> it's throwing the warning message. please find the bellow warning message.
> DataTables warning: table id=table-datanodes - Requested unknown parameter
> '7' for row 2. For more information about this error, please see
> http://datatables.net/tn/4
> Expected Output:
>
> whenever you click on Datanode tab,it should be display the datanodes
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12781) After Datanode down, In Namenode UI Datanode tab is throwing warning message.
[ https://issues.apache.org/jira/browse/HDFS-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-12781: --- Assignee: Brahma Reddy Battula > After Datanode down, In Namenode UI Datanode tab is throwing warning message. > - > > Key: HDFS-12781 > URL: https://issues.apache.org/jira/browse/HDFS-12781 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0-alpha1 >Reporter: Harshakiran Reddy >Assignee: Brahma Reddy Battula > > Scenario: > Stop one Datanode > Refresh or click on the Datanode tab in namenode UI. > Actual Output: > == > it's throwing the warning message. please find the bellow warning message. > DataTables warning: table id=table-datanodes - Requested unknown parameter > '7' for row 2. For more information about this error, please see > http://datatables.net/tn/4 > Expected Output: > > whenever you click on Datanode tab,it should be display the datanodes > information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12733) Option to disable to namenode local edits
[
https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12733:
Attachment: HDFS-12733-001.patch
Introduced {{DFS_NAMENODE_EDITS_LOCAL_WRITE_DISABLE_KEY}} which will {{false}}
by default (so that existing behaviour will not changed).
Kindly Review.
> Option to disable to namenode local edits
> -
>
> Key: HDFS-12733
> URL: https://issues.apache.org/jira/browse/HDFS-12733
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12733-001.patch
>
>
> As of now, Edits will be written in local and shared locations which will be
> redundant and local edits never used in HA setup.
> Disabling local edits gives little performance improvement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12733: Status: Patch Available (was: Open) > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12733-001.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12733) Option to disable to namenode local edits
Brahma Reddy Battula created HDFS-12733: --- Summary: Option to disable to namenode local edits Key: HDFS-12733 URL: https://issues.apache.org/jira/browse/HDFS-12733 Project: Hadoop HDFS Issue Type: Improvement Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula As of now, Edits will be written in local and shared locations which will be redundant and local edits never used in HA setup. Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12532: Status: Patch Available (was: In Progress) > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12532.patch > > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12532: Attachment: HDFS-12532.patch Attaching patch with above approach.Kindly review. > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12532.patch > > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12532 started by Brahma Reddy Battula. --- > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reopened HDFS-12532: - Assignee: Brahma Reddy Battula > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[
https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218114#comment-16218114
]
Brahma Reddy Battula commented on HDFS-12532:
-
bq.I recommend either switching your interface and aliased ips
Looks It's not possible in our environment. It used for another components also.
bq. set dfs.namenode.datanode.registration.ip-hostname-check=false.
This can aviod the ERROR, but it will overide the [folating
IP|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java#L1025]
hence clients will get this IP from Namenode but Datanode will listen on
another IP?
I feel, when principal having hostname which can't resolvable (i.e bindAddr =
null),we can have one config like below..? This can be configured DualIP
machines.
{code}
if (bindAddr == null) {
String bindAddrIp = conf.get(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_LOCAL_BIND_IP_KEY);
if (bindAddrIp != null && !bindAddrIp.isEmpty()) {
bindAddr = new InetSocketAddress(bindAddrIp, 0);
}
}
{code}
> DN Reg can Fail when principal doesn't contain hostname and floatingIP is
> configured.
> -
>
> Key: HDFS-12532
> URL: https://issues.apache.org/jira/browse/HDFS-12532
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>
> Configure principal without hostname (i.e hdfs/had...@hadoop.com)
> Configure floatingIP
> Start Cluster.
> Here DN will fail to register as it can take IP which is not in "/etc/hosts".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11146: Attachment: HDFS-11146-005.patch bq.I think checking in HeartbeatManager#heartbeatCheck is more appropriate Ok.Uploaded patch. KIndly review. > Excess replicas will not be deleted until all storages's FBR received after > failover > > > Key: HDFS-11146 > URL: https://issues.apache.org/jira/browse/HDFS-11146 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-11146-002.patch, HDFS-11146-003.patch, > HDFS-11146-004.patch, HDFS-11146-005.patch, HDFS-11146.patch > > > Excess replicas will not be deleted until all storages's FBR received after > failover. > Thinking following soultion can help. > *Solution:* > I think after failover, As DNs aware of failover ,so they can send another > block report (FBR) irrespective of interval.May be some shuffle can be done, > similar to initial delay. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12596) Add TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt back to branch-2.7
[
https://issues.apache.org/jira/browse/HDFS-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193357#comment-16193357
]
Brahma Reddy Battula commented on HDFS-12596:
-
+1,Pending for jenkins..thanks.
> Add TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt back to branch-2.7
> --
>
> Key: HDFS-12596
> URL: https://issues.apache.org/jira/browse/HDFS-12596
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.7.4
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12596.branch-2.7.01.patch
>
>
> {{TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt}} was reverted by
> HDFS-11743, but it is unrelated to HDFS-7933 and pretty contained by
> HDFS-11445. We should add it back.
> See
> https://issues.apache.org/jira/browse/HDFS-11743?focusedCommentId=16186328=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16186328
> for details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[
https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192951#comment-16192951
]
Brahma Reddy Battula commented on HDFS-12084:
-
[~kihwal] thanks a lot for taking a look. Nice Catches..
{{Delete before commit}} and {{pipeline updates}} will handle in next patch.
> Scheduled Count will not decrement when file is deleted before all IBR's
> received
> -
>
> Key: HDFS-12084
> URL: https://issues.apache.org/jira/browse/HDFS-12084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch,
> HDFS-12084-003.patch, HDFS-12084-branch-2.patch
>
>
> When small files creation && deletion happens so frequently and DN's did not
> report blocks to NN before deletion, then scheduled count will keep on
> increment and which will not deleted as blocks are deleted.
> *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be
> more as so many operations.
> when batchIBR enabled with committed allowed=1 this will be observed more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11743) Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7
[
https://issues.apache.org/jira/browse/HDFS-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192944#comment-16192944
]
Brahma Reddy Battula commented on HDFS-11743:
-
I think, we need to handle in seperate jira..? since {{2.7.4}} release is
already given..?
> Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7
> --
>
> Key: HDFS-11743
> URL: https://issues.apache.org/jira/browse/HDFS-11743
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Blocker
> Fix For: 2.7.4
>
> Attachments: HDFS-11743.addendum.patch,
> HDFS-11743-branch-2.7.00.patch, HDFS-11743-branch-2.7.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved HDFS-12532. - Resolution: Not A Bug > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192939#comment-16192939 ] Brahma Reddy Battula commented on HDFS-12532: - [~daryn] thanks a lot. you are correct. I tried both the ways,it's worked. > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-12532: --- Assignee: (was: Brahma Reddy Battula) > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HDFS-12532 > URL: https://issues.apache.org/jira/browse/HDFS-12532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11146: Attachment: HDFS-11146-004.patch bq.It can call setForceRegistration on the datanode descriptor and the next heartbeat will trigger the DNA_REGISTER. Nice Idea..Uploading the patch.Kindly review. > Excess replicas will not be deleted until all storages's FBR received after > failover > > > Key: HDFS-11146 > URL: https://issues.apache.org/jira/browse/HDFS-11146 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-11146-002.patch, HDFS-11146-003.patch, > HDFS-11146-004.patch, HDFS-11146.patch > > > Excess replicas will not be deleted until all storages's FBR received after > failover. > Thinking following soultion can help. > *Solution:* > I think after failover, As DNs aware of failover ,so they can send another > block report (FBR) irrespective of interval.May be some shuffle can be done, > similar to initial delay. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12456) TestNamenodeMetrics.testSyncAndBlockReportMetric fails
[
https://issues.apache.org/jira/browse/HDFS-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183946#comment-16183946
]
Brahma Reddy Battula commented on HDFS-12456:
-
Linking broken jira for reference.Correct me,If I am wrong.
> TestNamenodeMetrics.testSyncAndBlockReportMetric fails
> --
>
> Key: HDFS-12456
> URL: https://issues.apache.org/jira/browse/HDFS-12456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, metrics
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-12456.00.patch
>
>
> testSyncAndBlockReportMetric test is failure in trunk.
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testSyncAndBlockReportMetric
> Failing for the past 6 builds (Since Failed#21132 )
> Took 0.62 sec.
> Error Message:
> {code:java}
> Bad value for metric SyncsNumOps expected:<3> but was:<4>
> Stacktrace
> java.lang.AssertionError: Bad value for metric SyncsNumOps expected:<3> but
> was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:227)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testSyncAndBlockReportMetric(TestNameNodeMetrics.java:892
> {code}
> )
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11743) Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7
[ https://issues.apache.org/jira/browse/HDFS-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183660#comment-16183660 ] Brahma Reddy Battula commented on HDFS-11743: - Oh,Yes.[~xiaochen] thanks for finding. Looks it's unintenational. > Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7 > -- > > Key: HDFS-11743 > URL: https://issues.apache.org/jira/browse/HDFS-11743 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Zhe Zhang >Assignee: Zhe Zhang >Priority: Blocker > Fix For: 2.7.4 > > Attachments: HDFS-11743-branch-2.7.00.patch, > HDFS-11743-branch-2.7.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-5040:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}}.[~raghu.hb...@gmail.com] thanks for reporting and
[~kshukla] thanks a lot for your contribution,are you plan to raise follow up
jira's considering Vinay comment also..
[~vinayrpet] thanks a lot for additional review.
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Raghu C Doppalapudi
>Assignee: Kuhu Shukla
> Fix For: 3.1.0
>
> Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch,
> HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch,
> HDFS-5040.008.patch, HDFS-5040.009.patch, HDFS-5040.patch, HDFS-5040.patch,
> HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181053#comment-16181053
]
Brahma Reddy Battula edited comment on HDFS-5040 at 9/26/17 5:12 PM:
-
{{TestFailures}} are unrelated. +1 on latest patch.Going to commit shortly.
was (Author: brahmareddy):
{{TestFailures}} are unrelated. +1 on latest patch.Going commit shortly.
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Raghu C Doppalapudi
>Assignee: Kuhu Shukla
> Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch,
> HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch,
> HDFS-5040.008.patch, HDFS-5040.009.patch, HDFS-5040.patch, HDFS-5040.patch,
> HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-5040: --- Labels: (was: BB2015-05-TBR) > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.008.patch, HDFS-5040.009.patch, HDFS-5040.patch, HDFS-5040.patch, > HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181053#comment-16181053
]
Brahma Reddy Battula commented on HDFS-5040:
{{TestFailures}} are unrelated. +1 on latest patch.Going commit shortly.
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Raghu C Doppalapudi
>Assignee: Kuhu Shukla
> Labels: BB2015-05-TBR
> Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch,
> HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch,
> HDFS-5040.008.patch, HDFS-5040.009.patch, HDFS-5040.patch, HDFS-5040.patch,
> HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-5040: --- Issue Type: Improvement (was: New Feature) > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.008.patch, HDFS-5040.009.patch, HDFS-5040.patch, HDFS-5040.patch, > HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[
https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179095#comment-16179095
]
Brahma Reddy Battula commented on HDFS-12532:
-
[~daryn] thanks a lot for taking a look into this issue.
bq.That principal does have a hostname. Did you mean h...@hadoop.com?
Supposed to tell,{{hostname}} which is not resolvable i.e
hdfs/hadoop.hadoop@hadoop.com. Here hadoop.hadoop.com is n't configured as
hostname
bq.Is floating ip a dynamic dhcp address? If yes, is this for testing? Anyway,
I need to understand more about X.Y.Y.1 and X.Y.Y.100. I'm assuming X.Y.Y.1 is
localhost/127.0.0.1? X.Y.Y.100 is the dhcp assigned address?
Floating IP configured for HA purpose.Such that Active and standby can use same
IP to access webUI.*Here datanode registered with X.Y.Y.1 and X.Y.Y.100 is
floating IP which is used for further communication which will fail.*
bq.Here's the problem with your proposal: "getLocalHost" will attempt to
resolve the system assigned hostname, which you are assuming will always be in
/etc/hosts,
I was in assumption of host mapping will be configured for hadoop cluster.
Ok, then we can make it configurable..?
> DN Reg can Fail when principal doesn't contain hostname and floatingIP is
> configured.
> -
>
> Key: HDFS-12532
> URL: https://issues.apache.org/jira/browse/HDFS-12532
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> Configure principal without hostname (i.e hdfs/had...@hadoop.com)
> Configure floatingIP
> Start Cluster.
> Here DN will fail to register as it can take IP which is not in "/etc/hosts".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[
https://issues.apache.org/jira/browse/HDFS-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176440#comment-16176440
]
Brahma Reddy Battula commented on HDFS-12532:
-
*Log From the DN*
{noformat}
2017-09-18 01:54:15,762 | ERROR | DataNode:
[[[DISK]file:/srv/BigData/hadoop/data5/dn/,
[DISK]file:/srv/BigData/hadoop/data6/dn/,
[DISK]file:/srv/BigData/hadoop/data7/dn/,
[DISK]file:/srv/BigData/hadoop/data8/dn/,
[DISK]file:/srv/BigData/hadoop/data11/dn/,
[DISK]file:/srv/BigData/hadoop/data2/dn/,
[DISK]file:/srv/BigData/hadoop/data3/dn/,
[DISK]file:/srv/BigData/hadoop/data4/dn/,
[DISK]file:/srv/BigData/hadoop/data1/dn/,
[DISK]file:/srv/BigData/hadoop/data9/dn/,
[DISK]file:/srv/BigData/hadoop/data10/dn/]] heartbeating to
X-Y-Z-1/X.Y.Z.1:25000 | Initialization failed for Block pool
BP-141158008-X.Y.Y.1-1504865702183 (Datanode Uuid
4d9717af-7d34-4cfb-9c24-8e7e0e12d8e8) service to X-Y-Z-1/X.Y.Z.1:25000 Datanode
denied communication with namenode because hostname cannot be resolved
(ip=X.Y.Y.100, hostname=X.Y.Y.100): DatanodeRegistration(X.Y.Y.1:25009,
datanodeUuid=4d9717af-7d34-4cfb-9c24-8e7e0e12d8e8, infoPort=0,
infoSecurePort=25011, ipcPort=25008,
storageInfo=lv=-56;cid=myhacluster;nsid=1749012596;c=1504865702183)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:945)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:4503)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1419)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:96)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:29308)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2163)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2159)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2157)
| BPServiceActor.java:697
{noformat}
Thinking like below while setup the connection
at org.apache.hadoop.ipc.Client.Connection#setupConnection
1) Instead of assigning the {{null}} to {{bindAddr}}
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L665]
can we update with like below..?
{code}
InetSocketAddress bindAddr= new
InetSocketAddress(InetAddress.getLocalHost(),0);
{code}
2) Introduce one config for {{bindAddr}}..?
[~kihwal]/[~daryn] any thoughts on this as you worked on HADOOP-14578..? Please
correct me if I am wrong.
> DN Reg can Fail when principal doesn't contain hostname and floatingIP is
> configured.
> -
>
> Key: HDFS-12532
> URL: https://issues.apache.org/jira/browse/HDFS-12532
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> Configure principal without hostname (i.e hdfs/had...@hadoop.com)
> Configure floatingIP
> Start Cluster.
> Here DN will fail to register as it can take IP which is not in "/etc/hosts".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12532) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
Brahma Reddy Battula created HDFS-12532: --- Summary: DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured. Key: HDFS-12532 URL: https://issues.apache.org/jira/browse/HDFS-12532 Project: Hadoop HDFS Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Configure principal without hostname (i.e hdfs/had...@hadoop.com) Configure floatingIP Start Cluster. Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11034) Provide a command line tool to clear decommissioned DataNode information from the NameNode without restarting.
[
https://issues.apache.org/jira/browse/HDFS-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173117#comment-16173117
]
Brahma Reddy Battula commented on HDFS-11034:
-
can we add one argument to {{hdfs dfsadmin -refreshNodes}} itself instead of
new admin command..?
May be something like {{clearDeadNodes}}..?
> Provide a command line tool to clear decommissioned DataNode information from
> the NameNode without restarting.
> --
>
> Key: HDFS-11034
> URL: https://issues.apache.org/jira/browse/HDFS-11034
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chris Nauroth
>Assignee: Gergely Novák
>
> Information about decommissioned DataNodes remains tracked in the NameNode
> for the entire NameNode process lifetime. Currently, the only way to clear
> this information is to restart the NameNode. This issue proposes to add a
> way to clear this information online, without requiring a process restart.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[
https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172762#comment-16172762
]
Brahma Reddy Battula commented on HDFS-12084:
-
No Problem.I feel,this should go in {{2.8.2}} too.
> Scheduled Count will not decrement when file is deleted before all IBR's
> received
> -
>
> Key: HDFS-12084
> URL: https://issues.apache.org/jira/browse/HDFS-12084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch,
> HDFS-12084-003.patch, HDFS-12084-branch-2.patch
>
>
> When small files creation && deletion happens so frequently and DN's did not
> report blocks to NN before deletion, then scheduled count will keep on
> increment and which will not deleted as blocks are deleted.
> *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be
> more as so many operations.
> when batchIBR enabled with committed allowed=1 this will be observed more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172045#comment-16172045
]
Brahma Reddy Battula edited comment on HDFS-12480 at 9/19/17 6:43 PM:
--
Committed to {{trunk}} and {{branch-3.0}}. [~hanishakoneru] thanks for
contribution and thanks to [~vagarychen] for additional review.
was (Author: brahmareddy):
Committed to {{trunk} and {{branch-3.0}}. [~hanishakoneru] thanks for
contribution and thanks to [~vagarychen] for additional review.
> TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
> --
>
> Key: HDFS-12480
> URL: https://issues.apache.org/jira/browse/HDFS-12480
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Hanisha Koneru
>Priority: Blocker
> Fix For: 3.0.0-beta1, 3.1.0
>
> Attachments: HDFS-12480.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
> expected:<3> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[
https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11799:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
2.8.3
3.0.0-beta1
2.9.0
Release Note:
Added new configuration
"dfs.client.block.write.replace-datanode-on-failure.min-replication".
The minimum number of replications that are needed to not to fail
the write pipeline if new datanodes can not be found to replace
failed datanodes (could be due to network failure) in the write pipeline.
If the number of the remaining datanodes in the write pipeline is greater
than or equal to this property value, continue writing to the remaining
nodes.
Otherwise throw exception.
If this is set to 0, an exception will be thrown, when a replacement
can not be found.
Status: Resolved (was: Patch Available)
Committed to {{trunk}},{{branch-3.0}},{{branch-2}} and {{branch-2.8}}.
[~yzhangal] thanks a lot for continuous review. Resolved minor conflicts for
{{branch-2}} and {{branch-2.8}} and ran the testcase locally.
> Introduce a config to allow setting up write pipeline with fewer nodes than
> replication factor
> --
>
> Key: HDFS-11799
> URL: https://issues.apache.org/jira/browse/HDFS-11799
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Brahma Reddy Battula
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0
>
> Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch,
> HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch,
> HDFS-11799-007.patch, HDFS-11799-008.patch, HDFS-11799-009.patch,
> HDFS-11799.patch
>
>
> During pipeline recovery, if not enough DNs can be found, if
> dfs.client.block.write.replace-datanode-on-failure.best-effort
> is enabled, we let the pipeline to continue, even if there is a single DN.
> Similarly, when we create the write pipeline initially, if for some reason we
> can't find enough DNs, we can have a similar config to enable writing with a
> single DN.
> More study will be done.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[
https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172046#comment-16172046
]
Brahma Reddy Battula commented on HDFS-11799:
-
{{testfailures}} are unrelated. will commit shortly.
> Introduce a config to allow setting up write pipeline with fewer nodes than
> replication factor
> --
>
> Key: HDFS-11799
> URL: https://issues.apache.org/jira/browse/HDFS-11799
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch,
> HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch,
> HDFS-11799-007.patch, HDFS-11799-008.patch, HDFS-11799-009.patch,
> HDFS-11799.patch
>
>
> During pipeline recovery, if not enough DNs can be found, if
> dfs.client.block.write.replace-datanode-on-failure.best-effort
> is enabled, we let the pipeline to continue, even if there is a single DN.
> Similarly, when we create the write pipeline initially, if for some reason we
> can't find enough DNs, we can have a similar config to enable writing with a
> single DN.
> More study will be done.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12480:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed to {{trunk} and {{branch-3.0}}. [~hanishakoneru] thanks for
contribution and thanks to [~vagarychen] for additional review.
> TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
> --
>
> Key: HDFS-12480
> URL: https://issues.apache.org/jira/browse/HDFS-12480
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Hanisha Koneru
>Priority: Blocker
> Fix For: 3.0.0-beta1, 3.1.0
>
> Attachments: HDFS-12480.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
> expected:<3> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12480:
Priority: Blocker (was: Major)
> TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
> --
>
> Key: HDFS-12480
> URL: https://issues.apache.org/jira/browse/HDFS-12480
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Hanisha Koneru
>Priority: Blocker
> Attachments: HDFS-12480.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
> expected:<3> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172017#comment-16172017
]
Brahma Reddy Battula commented on HDFS-12480:
-
Test failures are unrelated. will commit shortly.
> TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
> --
>
> Key: HDFS-12480
> URL: https://issues.apache.org/jira/browse/HDFS-12480
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Hanisha Koneru
> Attachments: HDFS-12480.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
> expected:<3> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11799: Attachment: HDFS-11799-009.patch Uploaded the patch to fix the checkstyle,I ignored this,yongjun thanks for reminding. > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799-007.patch, HDFS-11799-008.patch, HDFS-11799-009.patch, > HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171105#comment-16171105 ] Brahma Reddy Battula commented on HDFS-11799: - Test Failures are unrelated. [~yzhangal] can you take look now..? > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799-007.patch, HDFS-11799-008.patch, HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171083#comment-16171083
]
Brahma Reddy Battula commented on HDFS-12480:
-
+1, Re-triggered the jenkins.
> TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
> --
>
> Key: HDFS-12480
> URL: https://issues.apache.org/jira/browse/HDFS-12480
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Hanisha Koneru
> Attachments: HDFS-12480.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
> expected:<3> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12480) TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk
Brahma Reddy Battula created HDFS-12480:
---
Summary: TestNameNodeMetrics#testTransactionAndCheckpointMetrics
Fails in trunk
Key: HDFS-12480
URL: https://issues.apache.org/jira/browse/HDFS-12480
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
{noformat}
java.lang.AssertionError: Bad value for metric LastWrittenTransactionId
expected:<3> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
at
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testTransactionAndCheckpointMetrics(TestNameNodeMetrics.java:854)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11799: Attachment: HDFS-11799-008.patch Uploaded the patch to address above comment. > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799-007.patch, HDFS-11799-008.patch, HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12395) Support erasure coding policy operations in namenode edit log
[ https://issues.apache.org/jira/browse/HDFS-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169867#comment-16169867 ] Brahma Reddy Battula commented on HDFS-12395: - Thanks for taking care, could have fixed the testcases in seperate jira since HDFS-12460 states "Make addErasureCodingPolicy an idempotent operation". Anyway it's committed,should be fine I feel. > Support erasure coding policy operations in namenode edit log > - > > Key: HDFS-12395 > URL: https://issues.apache.org/jira/browse/HDFS-12395 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-beta1 > > Attachments: editsStored, HDFS-12395.001.patch, HDFS-12395.002.patch, > HDFS-12395.003.patch, HDFS-12395.004.patch > > > Support add, remove, disable, enable erasure coding policy operation in edit > log. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169660#comment-16169660 ] Brahma Reddy Battula commented on HDFS-11799: - Testfailures are unrelated. Kindly Review. TestNamenodeRetryCache and TestRetryCacheWithHA failed after HDFS-12395 TestLeaseRecoveryStriped is tracked HDFS-12437. Rest are passing locally. > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799-007.patch, HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12395) Support erasure coding policy operations in namenode edit log
[ https://issues.apache.org/jira/browse/HDFS-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169658#comment-16169658 ] Brahma Reddy Battula commented on HDFS-12395: - Following two test fails after this commit. TestNamenodeRetryCache.testRetryCacheRebuild TestRetryCacheWithHA.testRetryCacheOnStandbyNN *Reference:* https://builds.apache.org/job/PreCommit-HDFS-Build/21189/testReport/ *Trace* java.lang.AssertionError: Retry cache size is wrong expected:<26> but was:<34> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache.testRetryCacheRebuild(TestNamenodeRetryCache.java:439) > Support erasure coding policy operations in namenode edit log > - > > Key: HDFS-12395 > URL: https://issues.apache.org/jira/browse/HDFS-12395 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-beta1 > > Attachments: editsStored, HDFS-12395.001.patch, HDFS-12395.002.patch, > HDFS-12395.003.patch, HDFS-12395.004.patch > > > Support add, remove, disable, enable erasure coding policy operation in edit > log. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169398#comment-16169398 ] Brahma Reddy Battula commented on HDFS-11576: - [~lukmajercak] thanks for reporting and working on this issue. latest patch lgtm. [~shv] do you any comments on latest patch..? > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.009.patch, HDFS-11576.010.patch, HDFS-11576.011.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11799: Attachment: HDFS-11799-007.patch Uploaded patch as per above suggestion. > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799-007.patch, HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169201#comment-16169201 ] Brahma Reddy Battula commented on HDFS-12381: - Test Failures are unrelated,me too +1 on latest patch. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch, HDFS-12381-HDFS-10467.002.patch, > HDFS-12381-HDFS-10467.003.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168475#comment-16168475 ] Brahma Reddy Battula commented on HDFS-12381: - [~goirix] thanks for updating the patch. I am thinking following also,sorry to trouble you. 1) can we add stop command for router..? may be start command also update like " hdfs --daemon start router"..? 2) how about adding default value to each property in the table(so totally three colums)..? and change description remaining boolean as i mentioned in earlier comment..? 3) Fix following two typos also,like below..? bq.Advanced functions like snapshotting, encryption Advanced functions like snapshot, encryption bq.Adminstrators can query information Administrators can query information > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12450) Fixing TestNamenodeHeartbeat and support non-HA
[
https://issues.apache.org/jira/browse/HDFS-12450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168405#comment-16168405
]
Brahma Reddy Battula commented on HDFS-12450:
-
+1, on Latetest patch.Pending for jenkins.Even I ran locally.
> Fixing TestNamenodeHeartbeat and support non-HA
> ---
>
> Key: HDFS-12450
> URL: https://issues.apache.org/jira/browse/HDFS-12450
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12450-HDFS-10467.000.patch,
> HDFS-12450-HDFS-10467.001.patch
>
>
> The way the service RPC address is obtained changed and showed a problem with
> {{TestNamenodeHeartbeat}} where the address wasn't properly set for the unit
> tests.
> In addition, the {{NamenodeHeartbeatService}} did not provide a good
> experience for non-HA nameservices. This also covers a better logging for
> those.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12378) TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166882#comment-16166882
]
Brahma Reddy Battula edited comment on HDFS-12378 at 9/14/17 7:53 PM:
--
+1,bisecting the good option thanks [~andrew.wang] and [~eddyxu] where I spent
time on debugging this.( I mean ,bisect can reduce the debugging efforts for
test failures).
was (Author: brahmareddy):
+1,bisecting the good option thanks [~andrew.wang] and [~eddyxu] where I spent
time on debugging this.
> TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
> --
>
> Key: HDFS-12378
> URL: https://issues.apache.org/jira/browse/HDFS-12378
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: Xiao Chen
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Labels: flaky-test
> Attachments: HDFS-12378.00.patch
>
>
> Saw on
> https://builds.apache.org/job/PreCommit-HDFS-Build/20928/testReport/org.apache.hadoop.hdfs/TestClientProtocolForPipelineRecovery/testZeroByteBlockRecovery/:
> Error Message
> {noformat}
> Failed to replace a bad datanode on the existing pipeline due to no more good
> datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> {noformat}
> Stacktrace
> {noformat}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1322)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1388)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1488)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1470)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1274)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {noformat}
> Standard Output
> {noformat}
> 2017-08-30 18:02:37,714 [main] INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:(469)) - starting cluster: numNameNodes=1,
> numDataNodes=3
> Formatting using clusterid: testClusterID
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSEditLog
> (FSEditLog.java:newInstance(224)) - Edit logging is async:false
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(742)) - KeyProvider: null
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(120)) - fsLock is fair: true
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(136)) - Detailed lock hold time metrics
> enabled: false
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(763)) - fsOwner = jenkins (auth:SIMPLE)
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(764)) - supergroup = supergroup
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(765)) - isPermissionEnabled = true
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(776)) - HA Enabled: false
> 2017-08-30 18:02:37,718 [main] INFO common.Util
> (Util.java:isDiskStatsEnabled(395)) -
> dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO
> profiling
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(301)) - dfs.block.invalidate.limit:
> configured=1000, counted=60, effected=1000
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(309)) -
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
>
[jira] [Commented] (HDFS-12378) TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166882#comment-16166882
]
Brahma Reddy Battula commented on HDFS-12378:
-
+1,bisecting the good option thanks [~andrew.wang] and [~eddyxu] where I spent
time on debugging this.
> TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
> --
>
> Key: HDFS-12378
> URL: https://issues.apache.org/jira/browse/HDFS-12378
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: Xiao Chen
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Labels: flaky-test
> Attachments: HDFS-12378.00.patch
>
>
> Saw on
> https://builds.apache.org/job/PreCommit-HDFS-Build/20928/testReport/org.apache.hadoop.hdfs/TestClientProtocolForPipelineRecovery/testZeroByteBlockRecovery/:
> Error Message
> {noformat}
> Failed to replace a bad datanode on the existing pipeline due to no more good
> datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> {noformat}
> Stacktrace
> {noformat}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1322)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1388)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1488)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1470)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1274)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {noformat}
> Standard Output
> {noformat}
> 2017-08-30 18:02:37,714 [main] INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:(469)) - starting cluster: numNameNodes=1,
> numDataNodes=3
> Formatting using clusterid: testClusterID
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSEditLog
> (FSEditLog.java:newInstance(224)) - Edit logging is async:false
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(742)) - KeyProvider: null
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(120)) - fsLock is fair: true
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(136)) - Detailed lock hold time metrics
> enabled: false
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(763)) - fsOwner = jenkins (auth:SIMPLE)
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(764)) - supergroup = supergroup
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(765)) - isPermissionEnabled = true
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(776)) - HA Enabled: false
> 2017-08-30 18:02:37,718 [main] INFO common.Util
> (Util.java:isDiskStatsEnabled(395)) -
> dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO
> profiling
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(301)) - dfs.block.invalidate.limit:
> configured=1000, counted=60, effected=1000
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(309)) -
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(76)) -
> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(82)) - The block deletion
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166711#comment-16166711 ] Brahma Reddy Battula commented on HDFS-12084: - Hi [~kihwal] If you get chance,can you please review the patch?? Thanks > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch, > HDFS-12084-003.patch, HDFS-12084-branch-2.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166698#comment-16166698 ] Brahma Reddy Battula commented on HDFS-11799: - [~yzhangal] if you get chance ,can you review once? I feel,we can push to branch-2.8.2 also since this will be useful > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12450) Fixing TestNamenodeHeartbeat and support non-HA
[
https://issues.apache.org/jira/browse/HDFS-12450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166658#comment-16166658
]
Brahma Reddy Battula commented on HDFS-12450:
-
Thanks for reporting,I supposed to raise this.will review your patch.
> Fixing TestNamenodeHeartbeat and support non-HA
> ---
>
> Key: HDFS-12450
> URL: https://issues.apache.org/jira/browse/HDFS-12450
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12450-HDFS-10467.000.patch
>
>
> The way the service RPC address is obtained changed and showed a problem with
> {{TestNamenodeHeartbeat}} where the address wasn't properly set for the unit
> tests.
> In addition, the {{NamenodeHeartbeatService}} did not provide a good
> experience for non-HA nameservices. This also covers a better logging for
> those.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166644#comment-16166644 ] Brahma Reddy Battula commented on HDFS-5040: +1,will commit on next week as I want somebody else also look into this. Test failures are unrelated as you noted. Hope you will raise follow up jira for earlier comments. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.008.patch, HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12430) Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164498#comment-16164498
]
Brahma Reddy Battula commented on HDFS-12430:
-
+1,LGTM. Changed the summary ,hope that should be fine for you.
> Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218
> ---
>
> Key: HDFS-12430
> URL: https://issues.apache.org/jira/browse/HDFS-12430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12430-HDFS-10467.000.patch
>
>
> HDFS-10467 is broken because of the new methods added to {{ClientProtocol}}
> from HDFS-12269 and HDFS-12218.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12430) Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12430:
Summary: Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218 (was:
Rebasing HDFS-10467 (3))
> Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218
> ---
>
> Key: HDFS-12430
> URL: https://issues.apache.org/jira/browse/HDFS-12430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12430-HDFS-10467.000.patch
>
>
> HDFS-10467 is broken because of the new methods added to {{ClientProtocol}}
> from HDFS-12269 and HDFS-12218.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[
https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164178#comment-16164178
]
Brahma Reddy Battula commented on HDFS-12381:
-
bq.Apparently there is yet one more method addition to ClientProtocol related
to EC.
Yes, it's after HDFS-12381. we need to rebase.
*Compilation Errors for reference*
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project h
adoop-hdfs: Compilation failure: Compilation failure:
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[69,39] cannot find symbol
[ERROR] symbol: class BlocksStats
[ERROR] location: package org.apache.hadoop.hdfs.protocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[79,39] cannot find symbol
[ERROR] symbol: class ECBlockGroupsStats
[ERROR] location: package org.apache.hadoop.hdfs.protocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1882,10] cannot find symbol
[ERROR] symbol: class ECBlockGroupsStats
[ERROR] location: class
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1894,10] cannot find symbol
[ERROR] symbol: class BlocksStats
[ERROR] location: class
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[138,8]
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer is not abstract
and doe
s not override abstract method getECBlockGroupStats() in
org.apache.hadoop.hdfs.protocol.ClientProtocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1881,3] method does not override or implement a
method from a supertype
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1893,3] method does not override or implement a
method from a supertype
{noformat}
> [Documentation] Adding configuration keys for the Router
>
>
> Key: HDFS-12381
> URL: https://issues.apache.org/jira/browse/HDFS-12381
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: HDFS-10467
>
> Attachments: HDFS-12381-HDFS-10467.000.patch,
> HDFS-12381-HDFS-10467.001.patch
>
>
> Adding configuration options in tabular format.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12436) TestClientProtocolForPipelineRecovery fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164109#comment-16164109
]
Brahma Reddy Battula commented on HDFS-12436:
-
Dupe of HDFS-12378..?
> TestClientProtocolForPipelineRecovery fails in trunk
>
>
> Key: HDFS-12436
> URL: https://issues.apache.org/jira/browse/HDFS-12436
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Arpit Agarwal
>Priority: Blocker
>
> Fails consistently in trunk with the following exception:
> {code}
> Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 71.317 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> testZeroByteBlockRecovery(org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery)
> Time elapsed: 11.422 sec <<< ERROR!
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1321)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1387)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1586)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1487)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1469)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1273)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163550#comment-16163550 ] Brahma Reddy Battula commented on HDFS-12381: - There are compilations errors , can you look once? I am on mobile.Errors might not related to this jira. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163094#comment-16163094
]
Brahma Reddy Battula commented on HDFS-6489:
bq.Where is the code you posted last? I wasn't able to find it in trunk or
branch-2
it's from the patch
[HDFS-6489.007.patch|https://issues.apache.org/jira/secure/attachment/12803016/HDFS-6489.007.patch]
you uploaded.
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch,
> HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> FilesystemSize Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162770#comment-16162770 ] Brahma Reddy Battula commented on HDFS-12381: - [~goirix] thanks for reporting and working on this. *Minor nits:* 1) Please correct the typo's bq.182 | Propery | Description| 2) How about chage like *"If true ,the Router should monitor the namenode in the local machine" which will be sync with existing conf* .If you agree, please change hdfs-default.xml and remaining configs also. bq.| dfs.federation.router.monitor.localnamenode.enable | If the Router should monitor the namenode in the local machine. | 3) Are you going update with this some more details..? bq.recommendations/best-practices about setting up the mount table, I think I can extend this JIRA to cover this a little more. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162755#comment-16162755 ] Brahma Reddy Battula commented on HDFS-5040: Would any one else like to chime in? [~kshukla] do update the patch..? > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161089#comment-16161089
]
Brahma Reddy Battula commented on HDFS-12416:
-
Dupe of HDFS-12177 and HDFS-11827..?
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> .
> chooseDataNode(scope, excludedNodes)
> .
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12384) Fixing compilation issue with BanDuplicateClasses
[
https://issues.apache.org/jira/browse/HDFS-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159917#comment-16159917
]
Brahma Reddy Battula commented on HDFS-12384:
-
My Delayed +1, I was in leave these days.Looks you didn't mentioned that it's
committed.
> Fixing compilation issue with BanDuplicateClasses
> -
>
> Key: HDFS-12384
> URL: https://issues.apache.org/jira/browse/HDFS-12384
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12384-HDFS-10467-000.patch,
> HDFS-12384-HDFS-10467-001.patch, HDFS-12384-HDFS-10467-002.patch,
> HDFS-12384-HDFS-10467-003.patch, HDFS-12384-HDFS-10467-004.patch,
> HDFS-12384-HDFS-10467-005.patch
>
>
> Build is failing because of changes in {{ClientProtocol}} and dependencies
> from {{CuratorManager}} indirectly added to {{hadoop-client-modules}}:
> {code}
> [INFO] Adding ignore: *
> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses
> failed with message:
> Duplicate classes found:
> Found in:
>
> org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-beta1-SNAPSHOT:compile
> org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-beta1-SNAPSHOT:compile
> Duplicate classes:
>
> org/apache/hadoop/shaded/org/apache/curator/framework/api/DeleteBuilder.class
>
> org/apache/hadoop/shaded/org/apache/curator/framework/CuratorFramework.class
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11799: Attachment: HDFS-11799-006.patch Uploaded the patch to address checkstyle and testcase fix. bq.hadoop.hdfs.TestClientProtocolForPipelineRecovery unrelated,there is tracking jira also HDFS-12378 bq. hadoop.tools.TestHdfsConfigFields handled Rest all are passing locally. I feel, 3 checkstyles can be ignored (2 are more than 80 and VisibilityModifier) > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799-006.patch, > HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[ https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11799: Attachment: HDFS-11799-005.patch Uploaded the patch to address above comments. now named like "dfs.client.block.write.replace-datanode-on-failure.min.replication" to unify with other. > Introduce a config to allow setting up write pipeline with fewer nodes than > replication factor > -- > > Key: HDFS-11799 > URL: https://issues.apache.org/jira/browse/HDFS-11799 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yongjun Zhang >Assignee: Brahma Reddy Battula > Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch, > HDFS-11799-004.patch, HDFS-11799-005.patch, HDFS-11799.patch > > > During pipeline recovery, if not enough DNs can be found, if > dfs.client.block.write.replace-datanode-on-failure.best-effort > is enabled, we let the pipeline to continue, even if there is a single DN. > Similarly, when we create the write pipeline initially, if for some reason we > can't find enough DNs, we can have a similar config to enable writing with a > single DN. > More study will be done. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12136) BlockSender performance regression due to volume scanner edge case
[
https://issues.apache.org/jira/browse/HDFS-12136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145433#comment-16145433
]
Brahma Reddy Battula edited comment on HDFS-12136 at 8/29/17 3:00 PM:
--
bq.so we could target 2.8.3 for the fix.
updated {{2.8.3}} as target version.
was (Author: brahmareddy):
bq.so we could target 2.8.3 for the fix.
updated as target version.
> BlockSender performance regression due to volume scanner edge case
> --
>
> Key: HDFS-12136
> URL: https://issues.apache.org/jira/browse/HDFS-12136
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-12136.branch-2.patch, HDFS-12136.trunk.patch
>
>
> HDFS-11160 attempted to fix a volume scan race for a file appended mid-scan
> by reading the last checksum of finalized blocks within the {{BlockSender}}
> ctor. Unfortunately it's holding the exclusive dataset lock to open and read
> the metafile multiple times Block sender instantiation becomes serialized.
> Performance completely collapses under heavy disk i/o utilization or high
> xceiver activity. Ex. lost node replication, balancing, or decommissioning.
> The xceiver threads congest creating block senders and impair the heartbeat
> processing that is contending for the same lock. Combined with other lock
> contention issues, pipelines break and nodes sporadically go dead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12136) BlockSender performance regression due to volume scanner edge case
[
https://issues.apache.org/jira/browse/HDFS-12136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145433#comment-16145433
]
Brahma Reddy Battula commented on HDFS-12136:
-
bq.so we could target 2.8.3 for the fix.
updated as target version.
> BlockSender performance regression due to volume scanner edge case
> --
>
> Key: HDFS-12136
> URL: https://issues.apache.org/jira/browse/HDFS-12136
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-12136.branch-2.patch, HDFS-12136.trunk.patch
>
>
> HDFS-11160 attempted to fix a volume scan race for a file appended mid-scan
> by reading the last checksum of finalized blocks within the {{BlockSender}}
> ctor. Unfortunately it's holding the exclusive dataset lock to open and read
> the metafile multiple times Block sender instantiation becomes serialized.
> Performance completely collapses under heavy disk i/o utilization or high
> xceiver activity. Ex. lost node replication, balancing, or decommissioning.
> The xceiver threads congest creating block senders and impair the heartbeat
> processing that is contending for the same lock. Combined with other lock
> contention issues, pipelines break and nodes sporadically go dead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12136) BlockSender performance regression due to volume scanner edge case
[
https://issues.apache.org/jira/browse/HDFS-12136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12136:
Target Version/s: 2.8.3 (was: 2.8.2)
> BlockSender performance regression due to volume scanner edge case
> --
>
> Key: HDFS-12136
> URL: https://issues.apache.org/jira/browse/HDFS-12136
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-12136.branch-2.patch, HDFS-12136.trunk.patch
>
>
> HDFS-11160 attempted to fix a volume scan race for a file appended mid-scan
> by reading the last checksum of finalized blocks within the {{BlockSender}}
> ctor. Unfortunately it's holding the exclusive dataset lock to open and read
> the metafile multiple times Block sender instantiation becomes serialized.
> Performance completely collapses under heavy disk i/o utilization or high
> xceiver activity. Ex. lost node replication, balancing, or decommissioning.
> The xceiver threads congest creating block senders and impair the heartbeat
> processing that is contending for the same lock. Combined with other lock
> contention issues, pipelines break and nodes sporadically go dead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143870#comment-16143870
]
Brahma Reddy Battula commented on HDFS-11896:
-
Pushed to {{branch-2.8.2}} as well. Compiled the ran the testcase locally.
{noformat}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.49 sec - in
org.apache.hadoop.hdfs.server.namenode.Te
stDeadDatanode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Non-dfsUsed will be doubled on dead node re-registration
>
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2, 2.8.3
>
> Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch,
> HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch,
> HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch,
> HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch,
> HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch,
> HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[ https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11896: Fix Version/s: 2.8.2 > Non-dfsUsed will be doubled on dead node re-registration > > > Key: HDFS-11896 > URL: https://issues.apache.org/jira/browse/HDFS-11896 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2, 2.8.3 > > Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch, > HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch, > HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch, > HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch, > HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch, > HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch, > HDFS-11896.patch > > > *Scenario:* > i)Make you sure you've non-dfs data. > ii) Stop Datanode > iii) wait it becomes dead > iv) now restart and check the non-dfs data -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Fix the Compile Error after HDFS-12299
[
https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12364:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
Status: Resolved (was: Patch Available)
Committed to {{branch-2.8.2}},[~yangjiandan] thanks for your contribution.
> [branch-2.8.2] Fix the Compile Error after HDFS-12299
> -
>
> Key: HDFS-12364
> URL: https://issues.apache.org/jira/browse/HDFS-12364
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Fix For: 2.8.2
>
> Attachments: HDFS-12364-branch-2.8.2.001.patch
>
>
> error line ：dn1.setHeartbeatsDisabledForTests(true)
> use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true);
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Fix the Compile Error after HDFS-12299
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Summary: [branch-2.8.2] Fix the Compile Error after HDFS-12299 (was: [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg) > [branch-2.8.2] Fix the Compile Error after HDFS-12299 > - > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143771#comment-16143771
]
Brahma Reddy Battula commented on HDFS-11896:
-
IMO, this should be backported to {{branch-2.8.2}} as well..?
> Non-dfsUsed will be doubled on dead node re-registration
>
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.3
>
> Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch,
> HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch,
> HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch,
> HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch,
> HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch,
> HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143735#comment-16143735 ] Brahma Reddy Battula commented on HDFS-12084: - [~kihwal] thanks for taking a look. bq. but I see reserve RBW space gets stuck and never going downon certain datanode. are you telling about DN side reserveSpaceForReplica..? > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch, > HDFS-12084-003.patch, HDFS-12084-branch-2.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Priority: Blocker (was: Major) > [branch-2.8.2] Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > --- > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143657#comment-16143657 ] Brahma Reddy Battula commented on HDFS-12364: - Test failures are unrelated, will commit soon. > [branch-2.8.2] Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > --- > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Summary: [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg (was: [branch-2.8]Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg) > [branch-2.8.2] Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > --- > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8]Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Summary: [branch-2.8]Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg (was: Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg) Target Version/s: 2.8.2 > [branch-2.8]Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12364) Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143492#comment-16143492 ] Brahma Reddy Battula commented on HDFS-12364: - [~yangjiandan] thanks for reporting. +1 , Pending for jenkins. > Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12248) SNN will not upload fsimage on IOE and Interrupted exceptions
[
https://issues.apache.org/jira/browse/HDFS-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12248:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed {{trunk}}.. [~vinayrpet] thanks lot for review and thanks to others
for additional review.
> SNN will not upload fsimage on IOE and Interrupted exceptions
> -
>
> Key: HDFS-12248
> URL: https://issues.apache.org/jira/browse/HDFS-12248
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12248-002.patch, HDFS-12248-003.patch,
> HDFS-12248.patch
>
>
> Related to HDFS-9787. When fsimage uploading to ANN, if there is any
> interrupt or IOE comes {{isPrimaryCheckPointer}} set to
> {{false}}.Rollingupgrade triggered same time then It does the checkpoint
> without sending the fsimage since {{sendRequest}} will be {{false}}.
> So,here {{rollback}} image will not sent to ANN.
> {code}
> } catch (ExecutionException e) {
> ioe = new IOException("Exception during image upload: " +
> e.getMessage(),
> e.getCause());
> break;
> } catch (InterruptedException e) {
> ie = e;
> break;
> }
> }
> lastUploadTime = monotonicNow();
> // we are primary if we successfully updated the ANN
> this.isPrimaryCheckPointer = success;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12299) Race Between update pipeline and DN Re-Registration
[
https://issues.apache.org/jira/browse/HDFS-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142523#comment-16142523
]
Brahma Reddy Battula commented on HDFS-12299:
-
[~kihwal] thanks for review and commit.
> Race Between update pipeline and DN Re-Registration
> ---
>
> Key: HDFS-12299
> URL: https://issues.apache.org/jira/browse/HDFS-12299
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HDFS-12299-branch-2-002.patch,
> HDFS-12299-branch-2.patch, HDFS-12299.patch
>
>
> *Scenario*
> - Started pipeline with DN1->DN2->DN3
> - DN1 is re-reg and update pipeline is called
> - Update pipeline will success with DN1->DN3->DN4
> - Again update pipeline is called,which will fail with NPE.
> In step3 updatepipeline will set the storages as null since DN1 re-reg(which
> will remove and add the storages)
> {{FSNamesystem#updatePipelineInternal}}
> {code}
>lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
> storages, lastBlock.getBlockType())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142521#comment-16142521
]
Brahma Reddy Battula commented on HDFS-12319:
-
[~arpitagarwal] thanks for review and commit. [~vinayrpet] thanks for
additional review.
bq.Committed this through to branch-2.8
Even you committed to {{branch-2.8.2}},looks you forgot to mention.
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HDFS-12319-001.patch, HDFS-12319-002.patch,
> TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[
https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12084:
Attachment: HDFS-12084-003.patch
Uploading the {{trunk}} patch. Updated the testcase to reproduce all the time
without fix. Earlier there was race in test.
> Scheduled Count will not decrement when file is deleted before all IBR's
> received
> -
>
> Key: HDFS-12084
> URL: https://issues.apache.org/jira/browse/HDFS-12084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch,
> HDFS-12084-003.patch, HDFS-12084-branch-2.patch
>
>
> When small files creation && deletion happens so frequently and DN's did not
> report blocks to NN before deletion, then scheduled count will keep on
> increment and which will not deleted as blocks are deleted.
> *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be
> more as so many operations.
> when batchIBR enabled with committed allowed=1 this will be observed more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12319:
Attachment: HDFS-12319-002.patch
[~vinayrpet] thanks for taking look.
bq.In Test, scanner.reconcile(); is called multiple times
Refactored now..Uploaded the patch.Kindly Review.
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: HDFS-12319-001.patch, HDFS-12319-002.patch,
> TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[
https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12084:
Attachment: HDFS-12084-branch-2.patch
Uploading the {{branch-2}} patch.IMO, this should go to {{2.8.2}}
release.Kindly review.
> Scheduled Count will not decrement when file is deleted before all IBR's
> received
> -
>
> Key: HDFS-12084
> URL: https://issues.apache.org/jira/browse/HDFS-12084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch,
> HDFS-12084-branch-2.patch
>
>
> When small files creation && deletion happens so frequently and DN's did not
> report blocks to NN before deletion, then scheduled count will keep on
> increment and which will not deleted as blocks are deleted.
> *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be
> more as so many operations.
> when batchIBR enabled with committed allowed=1 this will be observed more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139528#comment-16139528
]
Brahma Reddy Battula commented on HDFS-12319:
-
[~arpitagarwal] can you please review..? {{perfTimer}} and {{throttleTimer}}
should start per volume not per BP since both the BP's are under same dir.
Please correct me If I am wrong.
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: HDFS-12319-001.patch, TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139466#comment-16139466
]
Brahma Reddy Battula commented on HDFS-12319:
-
IMO, this should be blocker for {{2.8.2}} release since {{Directory scanner}}
might not work in federated cluster.
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: HDFS-12319-001.patch, TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12299) Race Between update pipeline and DN Re-Registration
[
https://issues.apache.org/jira/browse/HDFS-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12299:
Attachment: HDFS-12299-branch-2-002.patch
Updated patch to fix the {{checkstyle}} and testcase to reproduce all the time
without fix.
Kindly review.IMHO, this should go to {{2.8.2}} release.
> Race Between update pipeline and DN Re-Registration
> ---
>
> Key: HDFS-12299
> URL: https://issues.apache.org/jira/browse/HDFS-12299
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-12299-branch-2-002.patch,
> HDFS-12299-branch-2.patch, HDFS-12299.patch
>
>
> *Scenario*
> - Started pipeline with DN1->DN2->DN3
> - DN1 is re-reg and update pipeline is called
> - Update pipeline will success with DN1->DN3->DN4
> - Again update pipeline is called,which will fail with NPE.
> In step3 updatepipeline will set the storages as null since DN1 re-reg(which
> will remove and add the storages)
> {{FSNamesystem#updatePipelineInternal}}
> {code}
>lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
> storages, lastBlock.getBlockType())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9941) Do not log StandbyException on NN, other minor logging fixes
[
https://issues.apache.org/jira/browse/HDFS-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135033#comment-16135033
]
Brahma Reddy Battula commented on HDFS-9941:
[~arpitagarwal] nice work!!
can we backport this to {{branch-2.7}}..? Found during backport of HDFS-11817,
this is required by HDFS-11817.
> Do not log StandbyException on NN, other minor logging fixes
>
>
> Key: HDFS-9941
> URL: https://issues.apache.org/jira/browse/HDFS-9941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9941.01.patch, HDFS-9941.02.patch,
> HDFS-9941.03.patch, HDFS-9941-branch-2.03.patch
>
>
> The NameNode can skip logging StandbyException messages. These are seen
> regularly in normal operation and convey no useful information.
> We no longer log the locations of newly allocated blocks in 2.8.0. The DN IDs
> can be useful for debugging so let's add that back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12299) Race Between update pipeline and DN Re-Registration
[
https://issues.apache.org/jira/browse/HDFS-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12299:
Priority: Critical (was: Major)
Target Version/s: 2.8.2
> Race Between update pipeline and DN Re-Registration
> ---
>
> Key: HDFS-12299
> URL: https://issues.apache.org/jira/browse/HDFS-12299
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-12299-branch-2.patch, HDFS-12299.patch
>
>
> *Scenario*
> - Started pipeline with DN1->DN2->DN3
> - DN1 is re-reg and update pipeline is called
> - Update pipeline will success with DN1->DN3->DN4
> - Again update pipeline is called,which will fail with NPE.
> In step3 updatepipeline will set the storages as null since DN1 re-reg(which
> will remove and add the storages)
> {{FSNamesystem#updatePipelineInternal}}
> {code}
>lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
> storages, lastBlock.getBlockType())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12319:
Target Version/s: 3.0.0-beta1, 2.8.2 (was: 3.0.0-beta1)
Status: Patch Available (was: Open)
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: HDFS-12319-001.patch, TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present
[
https://issues.apache.org/jira/browse/HDFS-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12319:
Attachment: HDFS-12319-001.patch
Uploaded the patch.Kindly Review.
> DirectoryScanner will throw IllegalStateException when Multiple BP's are
> present
>
>
> Key: HDFS-12319
> URL: https://issues.apache.org/jira/browse/HDFS-12319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: HDFS-12319-001.patch, TestCase_to_Reproduce.patch
>
>
> *Scenario:*
> Configure "*dfs.datanode.directoryscan.interval*" as *60* and start federated
> cluster atleast with two nameservices.
> {noformat}
> 2017-08-18 19:06:37,150
> [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty
> queue]] ERROR datanode.DirectoryScanner
> (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the
> volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> StopWatch is already running
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: StopWatch is already running
> at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12299) Race Between update pipeline and DN Re-Registration
[
https://issues.apache.org/jira/browse/HDFS-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129004#comment-16129004
]
Brahma Reddy Battula edited comment on HDFS-12299 at 8/21/17 10:16 AM:
---
Uploading the test to reproduce this issue.
HDFS-9040 fixes this issue fully in {{trunk}}, So only testcase is added.
HDFS-11817 Partially (null storages were included) fixes in {{branch-2}} and
{{branch-2.8}}. Updated patch to add only {{non-null DatanodeStorageInfo}} at
BlockUnderConstructionFeature#setExpectedLocations
Kindly Review.
was (Author: brahmareddy):
Uploading the test to reproduce this issue.
HDFS-9040 fixes this issue fully in {{trunk}}, So only testcase is added.
HDFS-11817 Partially fixes in {{branch-2}} and {{branch-2.8}}. Updated patch
for full fix(null storages were included).
Kindly Review.
> Race Between update pipeline and DN Re-Registration
> ---
>
> Key: HDFS-12299
> URL: https://issues.apache.org/jira/browse/HDFS-12299
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12299-branch-2.patch, HDFS-12299.patch
>
>
> *Scenario*
> - Started pipeline with DN1->DN2->DN3
> - DN1 is re-reg and update pipeline is called
> - Update pipeline will success with DN1->DN3->DN4
> - Again update pipeline is called,which will fail with NPE.
> In step3 updatepipeline will set the storages as null since DN1 re-reg(which
> will remove and add the storages)
> {{FSNamesystem#updatePipelineInternal}}
> {code}
>lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
> storages, lastBlock.getBlockType())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12299) Race Between update pipeline and DN Re-Registration
[
https://issues.apache.org/jira/browse/HDFS-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129004#comment-16129004
]
Brahma Reddy Battula edited comment on HDFS-12299 at 8/21/17 10:14 AM:
---
Uploading the test to reproduce this issue.
HDFS-9040 fixes this issue fully in {{trunk}}, So only testcase is added.
HDFS-11817 Partially fixes in {{branch-2}} and {{branch-2.8}}. Updated patch
for full fix(null storages were included).
Kindly Review.
was (Author: brahmareddy):
Uploading the test to reproduce this issue.
HDFS-9040 fixes this issue fully in {{trunk}}, So only testcase is added.
HDFS-11817 Partially fixes in {{branch-2} and {{branch-2.8}}. Updated patch for
full fix(null storages were included).
Kindly Review.
> Race Between update pipeline and DN Re-Registration
> ---
>
> Key: HDFS-12299
> URL: https://issues.apache.org/jira/browse/HDFS-12299
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12299-branch-2.patch, HDFS-12299.patch
>
>
> *Scenario*
> - Started pipeline with DN1->DN2->DN3
> - DN1 is re-reg and update pipeline is called
> - Update pipeline will success with DN1->DN3->DN4
> - Again update pipeline is called,which will fail with NPE.
> In step3 updatepipeline will set the storages as null since DN1 re-reg(which
> will remove and add the storages)
> {{FSNamesystem#updatePipelineInternal}}
> {code}
>lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
> storages, lastBlock.getBlockType())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
