[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Status: Patch Available (was: Open) > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11609.trunk.patch > > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609.trunk.patch > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11609.trunk.patch > > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954245#comment-15954245 ] Kihwal Lee commented on HDFS-11609: --- bq. I have a patch with 1) and a unit test, but want to hear from others before posting. I thought it is acceptable to simply let admins know something is wrong (instead of pretending everything is fine), but then the recovery would be hard for many people, especially if it involves many nodes and blocks. So it might be better to automatically recover as much as possible. The version of patch I am attaching now considers a decommissioned node as a replication source if there is no live node. I.e. implementing the option 3). 1) is not done here, as it makes replication not happen. > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[
https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951587#comment-15951587
]
Kihwal Lee commented on HDFS-11609:
---
h3. Inability to correctly guess the previous replication priority
Guessing the previous replication priority level of a block works most of
times, but is not perfect. Different orders of events can lead to the identical
current state, but the previous priority levels can differ. We can improve the
priority update method so that the guessing logic still provides benefit in
majority of cases, yet its correctness is not strictly necessary.
Following shows problems I encountered and solutions.
In {{UnderReplicatedBlocks}},
{code}
private int getPriority(int curReplicas,
int readOnlyReplicas,
int decommissionedReplicas,
int expectedReplicas) {
assert curReplicas >= 0 : "Negative replicas!";
{code}
This is called from {{update()}}, which calls it with {{curReplicas}} set to
{{curReplicas-curReplicasDelta}}. When all replica-containing nodes are dead
({{curReplicas}} is 0), but a decommissioned node having a replica joins,
{{update()}} is called with {{curReplicas}} of -1, which sets off the assert.
This causes initial block report processing to stop in the middle. This node
is live and decommissioned and the block will appear missing because the block
report wasn't processed due to the assertion failure.
This can be avoided if {{curReplicasDelta}} is not set to 1 if this replica is
decommissioned. This value originates from {{BlockManager}}'s
{{addStoredBlock()}}.
{code}
if (result == AddBlockResult.ADDED) {
- curReplicaDelta = 1;
+ curReplicaDelta = (node.isDecommissioned()) ? 0 : 1;
{code}
This fixes this particular issue.
The assert is removed in the real build, so it acts differently in production
runtime. Instead block report processing blowing up, "-1" will cause it to
return the level, {{QUEUE_VERY_UNDER_REPLICATED}} without the above fix, which
is incorrect.
If the previous priority level is guessed incorrectly and it happens to be
identical to the current level, the old entry won't be removed, resulting in
duplicate entries. The {{remove()}} method is already robust so if a block is
not found in the specified level, it tries to remove it from other priority
levels too. So we can simply call {{remove()}} unconditionally. Guessing the
old priority is not functionally necessary with this change, but is still
useful, since the guess is normally correct which makes it visit only one
priority level for removal in most of cases.
{code}
-if(oldPri != curPri) {
- remove(block, oldPri);
-}
+// oldPri is mostly correct, but not always. If not found with oldPri,
+// other levels will be searched until the block is found & removed.
+remove(block, oldPri);
{code}
h3. Replication priority level of a block with only decommissioned replicas
With the surrounding bugs fixed, now we can address the real issue.
{{getPriority()}} explicitly does this:
{code}
} else if (curReplicas == 0) {
// If there are zero non-decommissioned replicas but there are
// some decommissioned replicas, then assign them highest priority
if (decommissionedReplicas > 0) {
return QUEUE_HIGHEST_PRIORITY;
}
{code}
This does not make any sense. Since decommissioned nodes are never chose as a
replication source, the block cannot be re-replicated. Being at this priority,
the block won't be recognized as "missing" either. It will appear that the
cluster is healthy until the decommissioned nodes are taken down, at which
point it might be too late to recover the data.
There are several possible approaches to this.
1) If all it has is decommissioned replicas, show it as missing. I.e. priority
level of {{QUEUE_WITH_CORRUPT_BLOCKS}}. {{fsck}} will show the decommissioned
locations and the admin can recommission/decommission or manually copy the data
out.
2) Re-evaluate all replicas when a decommissioned node rejoins. The simplest
way is to start decommissioning the node again.
3) Allow a decommissioned replica to be picked as a replication source in this
special case. 1) might still be needed.
I have a patch with 1) and a unit test, but want to hear from others before
posting.
> Some blocks can be permanently lost if nodes are decommissioned while dead
> --
>
> Key: HDFS-11609
> URL: https://issues.apache.org/jira/browse/HDFS-11609
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
>
> When all the nodes containing a replica of a block are decommissioned
[jira] [Assigned] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11609: - Assignee: Kihwal Lee > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
Kihwal Lee created HDFS-11609: - Summary: Some blocks can be permanently lost if nodes are decommissioned while dead Key: HDFS-11609 URL: https://issues.apache.org/jira/browse/HDFS-11609 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.0 Reporter: Kihwal Lee Priority: Critical When all the nodes containing a replica of a block are decommissioned while they are dead, they get decommissioned right away even if there are missing blocks. This behavior was introduced by HDFS-7374. The problem starts when those decommissioned nodes are brought back online. The namenode no longer shows missing blocks, which creates a false sense of cluster health. When the decommissioned nodes are removed and reformatted, the block data is permanently lost. The namenode will report missing blocks after the heartbeat recheck interval (e.g. 10 minutes) from the moment the last node is taken down. There are multiple issues in the code. As some cause different behaviors in testing vs. production, it took a while to reproduce it in a unit test. I will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11592) Closing a file has a wasteful preconditions
[
https://issues.apache.org/jira/browse/HDFS-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11592:
--
Target Version/s: 2.8.1
> Closing a file has a wasteful preconditions
> ---
>
> Key: HDFS-11592
> URL: https://issues.apache.org/jira/browse/HDFS-11592
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-11592.001.patch
>
>
> When a file is closed, the NN checks if all the blocks are complete. Instead
> of a simple 'if (!complete) throw new IllegalState(expensive-err-string)" it
> invokes "Preconditions.checkStatus(complete, expensive-err-string)". The
> check is done in a loop for all blocks, so more blocks = more penalty. The
> expensive string should only be computed when an error actually occurs. A
> telltale sign is seeing this in a stacktrace:
> {noformat}
>at java.lang.Class.getEnclosingMethod0(Native Method)
> at java.lang.Class.getEnclosingMethodInfo(Class.java:1072)
> at java.lang.Class.getEnclosingClass(Class.java:1272)
> at java.lang.Class.getSimpleBinaryName(Class.java:1443)
> at java.lang.Class.getSimpleName(Class.java:1309)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.assertAllBlocksComplete(INodeFile.java:246)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10601) Improve log message to include hostname when the NameNode is in safemode
[ https://issues.apache.org/jira/browse/HDFS-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10601: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.8.0 Status: Resolved (was: Patch Available) Committed from trunk to branch-2.8. Thanks for working on this Kuhu. > Improve log message to include hostname when the NameNode is in safemode > > > Key: HDFS-10601 > URL: https://issues.apache.org/jira/browse/HDFS-10601 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-10601.001.patch, HDFS-10601.002.patch, > HDFS-10601.003.patch, HDFS-10601.004.patch > > > When remote NN operations are involved, it would be nice to have the Namenode > hostname in safemode notification log. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10601) Improve log message to include hostname when the NameNode is in safemode
[ https://issues.apache.org/jira/browse/HDFS-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928313#comment-15928313 ] Kihwal Lee commented on HDFS-10601: --- +1 looks good. > Improve log message to include hostname when the NameNode is in safemode > > > Key: HDFS-10601 > URL: https://issues.apache.org/jira/browse/HDFS-10601 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Minor > Attachments: HDFS-10601.001.patch, HDFS-10601.002.patch, > HDFS-10601.003.patch, HDFS-10601.004.patch > > > When remote NN operations are involved, it would be nice to have the Namenode > hostname in safemode notification log. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11417) Add datanode admin command to get the storage info.
[ https://issues.apache.org/jira/browse/HDFS-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868417#comment-15868417 ] Kihwal Lee commented on HDFS-11417: --- Sorry, it is actually you who are working on both issues! > Add datanode admin command to get the storage info. > --- > > Key: HDFS-11417 > URL: https://issues.apache.org/jira/browse/HDFS-11417 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.3 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > > It is good to add one admin command for datanode to get the data directory > info like storage type, directory path, number of block, capacity, used > space. This will be help full in large cluster where DN has multiple data > directory configured. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11417) Add datanode admin command to get the storage info.
[ https://issues.apache.org/jira/browse/HDFS-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868413#comment-15868413 ] Kihwal Lee commented on HDFS-11417: --- More info was added to the datanoed UI. HDFS-11375 is improving it. You might want to give them input if something you think is important is missing. > Add datanode admin command to get the storage info. > --- > > Key: HDFS-11417 > URL: https://issues.apache.org/jira/browse/HDFS-11417 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.3 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > > It is good to add one admin command for datanode to get the data directory > info like storage type, directory path, number of block, capacity, used > space. This will be help full in large cluster where DN has multiple data > directory configured. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[
https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863850#comment-15863850
]
Kihwal Lee commented on HDFS-11379:
---
The impact of this bug can vary depending on how wide the job is, but it
essentially becomes DDOS using {{getBlockLocations()}}. In one incident, the NN
could deal with the rpc load, but it ran out of space for logging.
> DFSInputStream may infinite loop requesting block locations
> ---
>
> Key: HDFS-11379
> URL: https://issues.apache.org/jira/browse/HDFS-11379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.8.0, 3.0.0-alpha3
>
> Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch
>
>
> DFSInputStream creation caches file size and initial range of locations. If
> the file is truncated (or replaced) and the client attempts to read outside
> the initial range, the client goes into a tight infinite looping requesting
> locations for the nonexistent range.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863840#comment-15863840 ] Kihwal Lee commented on HDFS-11379: --- Thanks, I just cherry-picked it to branch-2.8.0. > DFSInputStream may infinite loop requesting block locations > --- > > Key: HDFS-11379 > URL: https://issues.apache.org/jira/browse/HDFS-11379 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch > > > DFSInputStream creation caches file size and initial range of locations. If > the file is truncated (or replaced) and the client attempts to read outside > the initial range, the client goes into a tight infinite looping requesting > locations for the nonexistent range. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11379: -- Fix Version/s: (was: 2.8.1) 2.8.0 > DFSInputStream may infinite loop requesting block locations > --- > > Key: HDFS-11379 > URL: https://issues.apache.org/jira/browse/HDFS-11379 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch > > > DFSInputStream creation caches file size and initial range of locations. If > the file is truncated (or replaced) and the client attempts to read outside > the initial range, the client goes into a tight infinite looping requesting > locations for the nonexistent range. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11379: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1 3.0.0-alpha3 Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-2.8. > DFSInputStream may infinite loop requesting block locations > --- > > Key: HDFS-11379 > URL: https://issues.apache.org/jira/browse/HDFS-11379 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 3.0.0-alpha3, 2.8.1 > > Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch > > > DFSInputStream creation caches file size and initial range of locations. If > the file is truncated (or replaced) and the client attempts to read outside > the initial range, the client goes into a tight infinite looping requesting > locations for the nonexistent range. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[
https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861596#comment-15861596
]
Kihwal Lee commented on HDFS-11379:
---
Nicely done. Although the retry in {{readWithStrategy()}} makes it try one more
time before throwing an exception back to the reader, it is far better than
going into an infinite loop.
+1
[~djp], this looks to be important enough to put in 2.8.0. Is it okay to commit
this to the release branch?
> DFSInputStream may infinite loop requesting block locations
> ---
>
> Key: HDFS-11379
> URL: https://issues.apache.org/jira/browse/HDFS-11379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch
>
>
> DFSInputStream creation caches file size and initial range of locations. If
> the file is truncated (or replaced) and the client attempts to read outside
> the initial range, the client goes into a tight infinite looping requesting
> locations for the nonexistent range.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859820#comment-15859820 ] Kihwal Lee commented on HDFS-11379: --- I will review it today. > DFSInputStream may infinite loop requesting block locations > --- > > Key: HDFS-11379 > URL: https://issues.apache.org/jira/browse/HDFS-11379 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch > > > DFSInputStream creation caches file size and initial range of locations. If > the file is truncated (or replaced) and the client attempts to read outside > the initial range, the client goes into a tight infinite looping requesting > locations for the nonexistent range. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang
[
https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832211#comment-15832211
]
Kihwal Lee commented on HDFS-11192:
---
Looking at the java source, {{Thread.start0()}} is mapped to a JVM native
method, {{JVM_StartThread()}}. Thread's uncaught exception handling is wired at
the Thread class so that it is called on exit. So setting the handler for
worker threads won't do any good.
But the exception does bubble up and is handled by the ThredGroup's hander. In
this case, it's the namenode's main thread. So we could set the
UncaughtExceptionHandler in caller thread so that it terminates on OOM. It
might be better to set it before a quota init and set it back to the default
after to avoid other subsystems (e.g. jetty) causing exit.
{{ForkJoinPool}} is also used by jdk itself in jdk8 (e.g.
{{ConcurrentHashMap}}), so even with this fix, you can run into strange
problems if the process cannot create more threads.
> OOM during Quota Initialization lead to Namenode hang
> -
>
> Key: HDFS-11192
> URL: https://issues.apache.org/jira/browse/HDFS-11192
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: namenodeThreadDump.out
>
>
> AFAIK ,In RecurisveTask Execution, When ForkjoinThreadpool's thread dies or
> not able to create,it will not notify the parent.Parent still waiting for the
> notify call..that's not timed waiting also.
> *Trace from Namenode log*
> {noformat}
> Exception in thread "ForkJoinPool-1-worker-2" Exception in thread
> "ForkJoinPool-1-worker-3" java.lang.OutOfMemoryError: unable to create new
> native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang
[
https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831904#comment-15831904
]
Kihwal Lee commented on HDFS-11192:
---
We can create the quota init ForkJoinPool with a UncaughtExceptionHandler. Need
to check whether it will work in this case.
> OOM during Quota Initialization lead to Namenode hang
> -
>
> Key: HDFS-11192
> URL: https://issues.apache.org/jira/browse/HDFS-11192
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: namenodeThreadDump.out
>
>
> AFAIK ,In RecurisveTask Execution, When ForkjoinThreadpool's thread dies or
> not able to create,it will not notify the parent.Parent still waiting for the
> notify call..that's not timed waiting also.
> *Trace from Namenode log*
> {noformat}
> Exception in thread "ForkJoinPool-1-worker-2" Exception in thread
> "ForkJoinPool-1-worker-3" java.lang.OutOfMemoryError: unable to create new
> native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828340#comment-15828340 ] Kihwal Lee commented on HDFS-9574: -- [~arpitagarwal], please elaborate the reason for marking it incompatible. If valid, we need to update the release note as well. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, > HDFS-9574.v3.br26.patch, HDFS-9574.v3.br27.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9409) DataNode shutdown does not guarantee full shutdown of all threads due to race condition.
[
https://issues.apache.org/jira/browse/HDFS-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819152#comment-15819152
]
Kihwal Lee commented on HDFS-9409:
--
{{BlockScanner}} is the same way. It removes a thread from its tracking data
structure after a timed join. This is probably less problematic because the
timeout in each {{join()}} is 5 minutes. Although this wait may be good for
unit tests, waiting up to 5 minutes per each volume scanner thread is
unreasonable when the datanode needs to be restarted quickly. We actually
internally reduced it down to 100ms and the rolling upgrades works a lot better.
Perhaps we need a flag in datanode that tells whether it should wait until
everything is terminated or quickly shutdown without waiting a long time for
daemon threads to terminate. The shutdown code of each subsystem would then
check this flag and behave differently. We would turn it on for unit testing
and off for production. How does it sound?
> DataNode shutdown does not guarantee full shutdown of all threads due to race
> condition.
>
>
> Key: HDFS-9409
> URL: https://issues.apache.org/jira/browse/HDFS-9409
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Chris Nauroth
>
> {{DataNode#shutdown}} is documented to return "only after shutdown is
> complete". Even after completion of this method, it's possible that threads
> started by the DataNode are still running. Race conditions in the shutdown
> sequence may cause it to skip stopping and joining the {{BPServiceActor}}
> threads.
> This is likely not a big problem in normal operations, because these are
> daemon threads that won't block overall process exit. It is more of a
> problem for tests, because it makes it impossible to write reliable
> assertions that these threads exited cleanly. For large test suites, it can
> also cause an accumulation of unneeded threads, which might harm test
> performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818861#comment-15818861 ] Kihwal Lee commented on HDFS-7967: -- The 2.8 patch doesn't seem right. > Reduce the performance impact of the balancer > - > > Key: HDFS-7967 > URL: https://issues.apache.org/jira/browse/HDFS-7967 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-7967-branch-2.8.patch, HDFS-7967-branch-2.patch, > HDFS-7967.branch-2-1.patch, HDFS-7967.branch-2.001.patch, > HDFS-7967.branch-2.002.patch, HDFS-7967.branch-2.8-1.patch, > HDFS-7967.branch-2.8.001.patch, HDFS-7967.branch-2.8.002.patch > > > The balancer needs to query for blocks to move from overly full DNs. The > block lookup is extremely inefficient. An iterator of the node's blocks is > created from the iterators of its storages' blocks. A random number is > chosen corresponding to how many blocks will be skipped via the iterator. > Each skip requires costly scanning of triplets. > The current design also only considers node imbalances while ignoring > imbalances within the nodes's storages. A more efficient and intelligent > design may eliminate the costly skipping of blocks via round-robin selection > of blocks from the storages based on remaining capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3618) SSH fencing option may incorrectly succeed if nc (netcat) command not present
[
https://issues.apache.org/jira/browse/HDFS-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816250#comment-15816250
]
Kihwal Lee commented on HDFS-3618:
--
The "-z" option is supported by the plain old BSD implementation of netcat. It
looks like newer distros are moving to NMAP's netcat replacement, which does
not support "-z".
Someone suggested an equivalent of "-z" for nmap-ncat.
http://unix.stackexchange.com/questions/296023/whats-the-gnu-netcats-z-equivalent-option-in-nmap-ncat
A lazy way out is simply making the entire command configurable. One could
also make it detect which variant is available. {{nc -z 127.0.0.1 1234}} will
return 2 if the host has nmap-ncat and 0 or 1 if bsd nc.
> SSH fencing option may incorrectly succeed if nc (netcat) command not present
> -
>
> Key: HDFS-3618
> URL: https://issues.apache.org/jira/browse/HDFS-3618
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: auto-failover
>Affects Versions: 2.0.0-alpha
>Reporter: Brahma Reddy Battula
>Assignee: Vinayakumar B
> Labels: BB2015-05-TBR
> Attachments: HDFS-3618-04.patch, HDFS-3618.patch, HDFS-3618.patch,
> HDFS-3618.patch, zkfc.txt, zkfc_threaddump.out
>
>
> Started NN's and zkfc's in Suse11.
> Suse11 will have netcat installation and netcat -z will work(but nc -z wn't
> work)..
> While executing following command, got command not found hence rc will be
> other than zero and assuming that server was down..Here we are ending up
> without checking whether service is down or not..
> {code}
> LOG.info(
> "Indeterminate response from trying to kill service. " +
> "Verifying whether it is running using nc...");
> rc = execCommand(session, "nc -z " + serviceAddr.getHostName() +
> " " + serviceAddr.getPort());
> if (rc == 0) {
> // the service is still listening - we are unable to fence
> LOG.warn("Unable to fence - it is running but we cannot kill it");
> return false;
> } else {
> LOG.info("Verified that the service is down.");
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815227#comment-15815227 ] Kihwal Lee commented on HDFS-7967: -- I have reviewed the patch and am fine with it. We have used a variant of this over a year and this is the latest improved version. We can commit this to branch-2 and 2.8 now, but then we will likely forget about the remaining work and move on. So I think we need to discuss what we are going to do for trunk. [~daryn], would you share your thoughts and concerns on the state of trunk and possible solutions? > Reduce the performance impact of the balancer > - > > Key: HDFS-7967 > URL: https://issues.apache.org/jira/browse/HDFS-7967 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-7967-branch-2.8.patch, HDFS-7967-branch-2.patch, > HDFS-7967.branch-2-1.patch, HDFS-7967.branch-2.001.patch, > HDFS-7967.branch-2.002.patch, HDFS-7967.branch-2.8-1.patch, > HDFS-7967.branch-2.8.001.patch, HDFS-7967.branch-2.8.002.patch > > > The balancer needs to query for blocks to move from overly full DNs. The > block lookup is extremely inefficient. An iterator of the node's blocks is > created from the iterators of its storages' blocks. A random number is > chosen corresponding to how many blocks will be skipped via the iterator. > Each skip requires costly scanning of triplets. > The current design also only considers node imbalances while ignoring > imbalances within the nodes's storages. A more efficient and intelligent > design may eliminate the costly skipping of blocks via round-robin selection > of blocks from the storages based on remaining capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802662#comment-15802662
]
Kihwal Lee commented on HDFS-9083:
--
[~djp], that's correct.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer
[
https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802511#comment-15802511
]
Kihwal Lee commented on HDFS-7967:
--
The 2.8 patch works fine with jdk8. The unit tests ran fine too.
{noformat}
Results :
Tests run: 4100, Failures: 0, Errors: 0, Skipped: 17
{noformat}
But it doesn't compile with jdk7.
{noformat}
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[47,25]
error: package java.util.function does not exist
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java:[23,25]
error: package java.util.function does not exist
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[3999,10]
error: cannot find symbol
[ERROR] symbol: class Predicate
location: class BlockManager
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java:[325,40]
error: cannot find symbol
[ERROR] symbol: class Predicate
location: class DatanodeStorageInfo
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java:[92,18]
error: cannot find symbol
[ERROR] symbol: class Predicate
location: class DatanodeStorageInfo.BlockIterator
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java:[96,18]
error: cannot find symbol
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[4000,10]
error: cannot find symbol
[ERROR] symbol: class Predicate
location: class BlockManager
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[4015,10]
error: BlockManager.BalancerBlockIterator is not abstract and does not
override abstract method remove() in Iterator
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[4023,11]
error: cannot find symbol
[ERROR] symbol: method sort(Comparator)
location: variable iters of type ArrayList
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:[4042,13]
error: cannot find symbol
{noformat}
> Reduce the performance impact of the balancer
> -
>
> Key: HDFS-7967
> URL: https://issues.apache.org/jira/browse/HDFS-7967
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-7967-branch-2.8.patch, HDFS-7967-branch-2.patch
>
>
> The balancer needs to query for blocks to move from overly full DNs. The
> block lookup is extremely inefficient. An iterator of the node's blocks is
> created from the iterators of its storages' blocks. A random number is
> chosen corresponding to how many blocks will be skipped via the iterator.
> Each skip requires costly scanning of triplets.
> The current design also only considers node imbalances while ignoring
> imbalances within the nodes's storages. A more efficient and intelligent
> design may eliminate the costly skipping of blocks via round-robin selection
> of blocks from the storages based on remaining capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801541#comment-15801541 ] Kihwal Lee commented on HDFS-11160: --- [~jojochuang], please make sure it is in 2.8. > VolumeScanner reports write-in-progress replicas as corrupt incorrectly > --- > > Key: HDFS-11160 > URL: https://issues.apache.org/jira/browse/HDFS-11160 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Environment: CDH5.7.4 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HDFS-11160.001.patch, HDFS-11160.002.patch, > HDFS-11160.003.patch, HDFS-11160.004.patch, HDFS-11160.005.patch, > HDFS-11160.006.patch, HDFS-11160.007.patch, HDFS-11160.008.patch, > HDFS-11160.branch-2.patch, HDFS-11160.reproduce.patch > > > Due to a race condition initially reported in HDFS-6804, VolumeScanner may > erroneously detect good replicas as corrupt. This is serious because in some > cases it results in data loss if all replicas are declared corrupt. This bug > is especially prominent when there are a lot of append requests via > HttpFs/WebHDFS. > We are investigating an incidence that caused very high block corruption rate > in a relatively small cluster. Initially, we thought HDFS-11056 is to blame. > However, after applying HDFS-11056, we are still seeing VolumeScanner > reporting corrupt replicas. > It turns out that if a replica is being appended while VolumeScanner is > scanning it, VolumeScanner may use the new checksum to compare against old > data, causing checksum mismatch. > I have a unit test to reproduce the error. Will attach later. A quick and > simple fix is to hold FsDatasetImpl lock and read from disk the checksum. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7784) load fsimage in parallel
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795396#comment-15795396 ] Kihwal Lee commented on HDFS-7784: -- [~xiegang112], when you have a chance to test the performance, please also share the jvm GC setting and the hardware spec (e.g. how many cores, as it affects the GC performance). It will be even better if you can measure the GC activities before and after. If everything looks positive, people will certainly be interested. > load fsimage in parallel > > > Key: HDFS-7784 > URL: https://issues.apache.org/jira/browse/HDFS-7784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Walter Su >Assignee: Walter Su >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-7784.001.patch, test-20150213.pdf > > > When single Namenode has huge amount of files, without using federation, the > startup/restart speed is slow. The fsimage loading step takes the most of the > time. fsimage loading can seperate to two parts, deserialization and object > construction(mostly map insertion). Deserialization takes the most of CPU > time. So we can do deserialization in parallel, and add to hashmap in serial. > It will significantly reduce the NN start time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751772#comment-15751772 ] Kihwal Lee commented on HDFS-11094: --- I am fine with the latest patch. > Send back HAState along with NamespaceInfo during a versionRequest as an > optional parameter > --- > > Key: HDFS-11094 > URL: https://issues.apache.org/jira/browse/HDFS-11094 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, > HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, > HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch, > HDFS-11094.009-b2.patch, HDFS-11094.009.patch, HDFS-11094.010-b2.patch, > HDFS-11094.010.patch, HDFS-11094.011.patch > > > The datanode should know which NN is active when it is connecting/registering > to the NN. Currently, it only figures this out during its first (and > subsequent) heartbeat(s) and so there is a period of time where the datanode > is alive and registered, but can't actually do anything because it doesn't > know which NN is active. A byproduct of this is that the MiniDFSCluster will > become active before it knows what NN is active, which can lead to NPEs when > calling getActiveNN(). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos
[
https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HDFS-11207.
---
Resolution: Done
> Unnecessary incompatible change of NNHAStatusHeartbeat.state in
> DatanodeProtocolProtos
> --
>
> Key: HDFS-11207
> URL: https://issues.apache.org/jira/browse/HDFS-11207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 3.0.0-alpha1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Critical
> Attachments: HDFS-11207.001.patch
>
>
> HDFS-5079 changed the meaning of state in {{NNHAStatusHeartbeat}} when it
> added in the {{INITIALIZING}} state via {{HAServiceStateProto}}.
> Before change:
> {noformat}
> enum State {
>ACTIVE = 0;
>STANDBY = 1;
> }
> {noformat}
> After change:
> {noformat}
> enum HAServiceStateProto {
> INITIALIZING = 0;
> ACTIVE = 1;
> STANDBY = 2;
> }
> {noformat}
> So the new {{INITIALIZING}} state will be interpreted as {{ACTIVE}}, new
> {{ACTIVE}} interpreted as {{STANDBY}} and new {{STANDBY}} interpreted as
> unknown. Any rolling upgrade to 3.0.0 will break because the datanodes that
> haven't been updated will misinterpret the NN state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos
[
https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733496#comment-15733496
]
Kihwal Lee commented on HDFS-11207:
---
Reverted HDFS-5079 and HDFS-11094.
> Unnecessary incompatible change of NNHAStatusHeartbeat.state in
> DatanodeProtocolProtos
> --
>
> Key: HDFS-11207
> URL: https://issues.apache.org/jira/browse/HDFS-11207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 3.0.0-alpha1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Critical
> Attachments: HDFS-11207.001.patch
>
>
> HDFS-5079 changed the meaning of state in {{NNHAStatusHeartbeat}} when it
> added in the {{INITIALIZING}} state via {{HAServiceStateProto}}.
> Before change:
> {noformat}
> enum State {
>ACTIVE = 0;
>STANDBY = 1;
> }
> {noformat}
> After change:
> {noformat}
> enum HAServiceStateProto {
> INITIALIZING = 0;
> ACTIVE = 1;
> STANDBY = 2;
> }
> {noformat}
> So the new {{INITIALIZING}} state will be interpreted as {{ACTIVE}}, new
> {{ACTIVE}} interpreted as {{STANDBY}} and new {{STANDBY}} interpreted as
> unknown. Any rolling upgrade to 3.0.0 will break because the datanodes that
> haven't been updated will misinterpret the NN state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-11094: --- I've reverted this, as HDFS-5079 was reverted. Please rework the patch to reflect that. > Send back HAState along with NamespaceInfo during a versionRequest as an > optional parameter > --- > > Key: HDFS-11094 > URL: https://issues.apache.org/jira/browse/HDFS-11094 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, > HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, > HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch, > HDFS-11094.009-b2.patch, HDFS-11094.009.patch > > > The datanode should know which NN is active when it is connecting/registering > to the NN. Currently, it only figures this out during its first (and > subsequent) heartbeat(s) and so there is a period of time where the datanode > is alive and registered, but can't actually do anything because it doesn't > know which NN is active. A byproduct of this is that the MiniDFSCluster will > become active before it knows what NN is active, which can lead to NPEs when > calling getActiveNN(). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Release Note: This change affects wire-compatibility of the NameNode/DataNode heartbeat protocol. Only present in 3.0.0-alpah1. It has been reverted before 3.0.0-alpha2 (was: This change affects wire-compatibility of the NameNode/DataNode heartbeat protocol.) > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. > - > > Key: HDFS-5079 > URL: https://issues.apache.org/jira/browse/HDFS-5079 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Konstantin Shvachko >Assignee: Tao Luo > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-5079.patch > > > NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective > class should also be removed from DatanodeProtocolProtos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733485#comment-15733485 ] Kihwal Lee commented on HDFS-5079: -- Reverted and updated the release note. > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. > - > > Key: HDFS-5079 > URL: https://issues.apache.org/jira/browse/HDFS-5079 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Konstantin Shvachko >Assignee: Tao Luo > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-5079.patch > > > NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective > class should also be removed from DatanodeProtocolProtos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Fix Version/s: 3.0.0-alpha1 > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. > - > > Key: HDFS-5079 > URL: https://issues.apache.org/jira/browse/HDFS-5079 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Konstantin Shvachko >Assignee: Tao Luo > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-5079.patch > > > NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective > class should also be removed from DatanodeProtocolProtos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Fix Version/s: (was: 3.0.0-alpha1) > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. > - > > Key: HDFS-5079 > URL: https://issues.apache.org/jira/browse/HDFS-5079 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Konstantin Shvachko >Assignee: Tao Luo > Attachments: HDFS-5079.patch > > > NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective > class should also be removed from DatanodeProtocolProtos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732634#comment-15732634 ] Kihwal Lee commented on HDFS-5079: -- This doesn't seem to much value while breaking compatibility during rolling upgrade. I propose revert. > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. > - > > Key: HDFS-5079 > URL: https://issues.apache.org/jira/browse/HDFS-5079 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Konstantin Shvachko >Assignee: Tao Luo > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-5079.patch > > > NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective > class should also be removed from DatanodeProtocolProtos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos
[
https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732628#comment-15732628
]
Kihwal Lee commented on HDFS-11207:
---
I am going to revert HDFS-5079 later today if there is no objections.
> Unnecessary incompatible change of NNHAStatusHeartbeat.state in
> DatanodeProtocolProtos
> --
>
> Key: HDFS-11207
> URL: https://issues.apache.org/jira/browse/HDFS-11207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 3.0.0-alpha1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Critical
> Attachments: HDFS-11207.001.patch
>
>
> HDFS-5079 changed the meaning of state in {{NNHAStatusHeartbeat}} when it
> added in the {{INITIALIZING}} state via {{HAServiceStateProto}}.
> Before change:
> {noformat}
> enum State {
>ACTIVE = 0;
>STANDBY = 1;
> }
> {noformat}
> After change:
> {noformat}
> enum HAServiceStateProto {
> INITIALIZING = 0;
> ACTIVE = 1;
> STANDBY = 2;
> }
> {noformat}
> So the new {{INITIALIZING}} state will be interpreted as {{ACTIVE}}, new
> {{ACTIVE}} interpreted as {{STANDBY}} and new {{STANDBY}} interpreted as
> unknown. Any rolling upgrade to 3.0.0 will break because the datanodes that
> haven't been updated will misinterpret the NN state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[
https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729073#comment-15729073
]
Kihwal Lee edited comment on HDFS-11160 at 12/7/16 3:53 PM:
+1 The patch looks good. About
{noformat}
// TODO: we only need to do this if the visibleLength is not
// at chunk boundary
{noformat}
Frequent appending will likely leave the end of the block unaligned. So I think
this optimization is not worth adding.
was (Author: kihwal):
The patch looks good. About
{noformat}
// TODO: we only need to do this if the visibleLength is not
// at chunk boundary
{noformat}
Frequent appending will likely leave the end of the block unaligned. So I think
this optimization is not worth adding.
> VolumeScanner reports write-in-progress replicas as corrupt incorrectly
> ---
>
> Key: HDFS-11160
> URL: https://issues.apache.org/jira/browse/HDFS-11160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Environment: CDH5.7.4
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-11160.001.patch, HDFS-11160.002.patch,
> HDFS-11160.003.patch, HDFS-11160.reproduce.patch
>
>
> Due to a race condition initially reported in HDFS-6804, VolumeScanner may
> erroneously detect good replicas as corrupt. This is serious because in some
> cases it results in data loss if all replicas are declared corrupt. This bug
> is especially prominent when there are a lot of append requests via
> HttpFs/WebHDFS.
> We are investigating an incidence that caused very high block corruption rate
> in a relatively small cluster. Initially, we thought HDFS-11056 is to blame.
> However, after applying HDFS-11056, we are still seeing VolumeScanner
> reporting corrupt replicas.
> It turns out that if a replica is being appended while VolumeScanner is
> scanning it, VolumeScanner may use the new checksum to compare against old
> data, causing checksum mismatch.
> I have a unit test to reproduce the error. Will attach later. A quick and
> simple fix is to hold FsDatasetImpl lock and read from disk the checksum.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[
https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729073#comment-15729073
]
Kihwal Lee commented on HDFS-11160:
---
The patch looks good. About
{noformat}
// TODO: we only need to do this if the visibleLength is not
// at chunk boundary
{noformat}
Frequent appending will likely leave the end of the block unaligned. So I think
this optimization is not worth adding.
> VolumeScanner reports write-in-progress replicas as corrupt incorrectly
> ---
>
> Key: HDFS-11160
> URL: https://issues.apache.org/jira/browse/HDFS-11160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Environment: CDH5.7.4
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-11160.001.patch, HDFS-11160.002.patch,
> HDFS-11160.003.patch, HDFS-11160.reproduce.patch
>
>
> Due to a race condition initially reported in HDFS-6804, VolumeScanner may
> erroneously detect good replicas as corrupt. This is serious because in some
> cases it results in data loss if all replicas are declared corrupt. This bug
> is especially prominent when there are a lot of append requests via
> HttpFs/WebHDFS.
> We are investigating an incidence that caused very high block corruption rate
> in a relatively small cluster. Initially, we thought HDFS-11056 is to blame.
> However, after applying HDFS-11056, we are still seeing VolumeScanner
> reporting corrupt replicas.
> It turns out that if a replica is being appended while VolumeScanner is
> scanning it, VolumeScanner may use the new checksum to compare against old
> data, causing checksum mismatch.
> I have a unit test to reproduce the error. Will attach later. A quick and
> simple fix is to hold FsDatasetImpl lock and read from disk the checksum.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos
[
https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726418#comment-15726418
]
Kihwal Lee commented on HDFS-11207:
---
I think we should revert HDFS-5079. It breaks compatibility while adding not
much value.
> Unnecessary incompatible change of NNHAStatusHeartbeat.state in
> DatanodeProtocolProtos
> --
>
> Key: HDFS-11207
> URL: https://issues.apache.org/jira/browse/HDFS-11207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 3.0.0-alpha1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Critical
> Attachments: HDFS-11207.001.patch
>
>
> HDFS-5079 changed the meaning of state in {{NNHAStatusHeartbeat}} when it
> added in the {{INITIALIZING}} state via {{HAServiceStateProto}}.
> Before change:
> {noformat}
> enum State {
>ACTIVE = 0;
>STANDBY = 1;
> }
> {noformat}
> After change:
> {noformat}
> enum HAServiceStateProto {
> INITIALIZING = 0;
> ACTIVE = 1;
> STANDBY = 2;
> }
> {noformat}
> So the new {{INITIALIZING}} state will be interpreted as {{ACTIVE}}, new
> {{ACTIVE}} interpreted as {{STANDBY}} and new {{STANDBY}} interpreted as
> unknown. Any rolling upgrade to 3.0.0 will break because the datanodes that
> haven't been updated will misinterpret the NN state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) I've committed this to trunk through branch-2.8. Thanks for working on this, [~cheersyang]. > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Assignee: Weiwei Yang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11198.01.patch, HDFS-11198.02.patch > > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725570#comment-15725570 ] Kihwal Lee commented on HDFS-11198: --- +1 > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11198.01.patch, HDFS-11198.02.patch > > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722401#comment-15722401 ] Kihwal Lee commented on HDFS-11198: --- The build master is down at this moment. > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11198.01.patch > > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716687#comment-15716687 ] Kihwal Lee commented on HDFS-11094: --- bq. For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()? I think the existing tests are quite adequate. I understand that a full-blown mini cluster is sometimes needed to test the distributed file system. However, we should avoid adding such end-to-end tests if it is possible to have reasonable unit tests. > Send back HAState along with NamespaceInfo during a versionRequest as an > optional parameter > --- > > Key: HDFS-11094 > URL: https://issues.apache.org/jira/browse/HDFS-11094 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, > HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, > HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch, > HDFS-11094.009.patch > > > The datanode should know which NN is active when it is connecting/registering > to the NN. Currently, it only figures this out during its first (and > subsequent) heartbeat(s) and so there is a period of time where the datanode > is alive and registered, but can't actually do anything because it doesn't > know which NN is active. A byproduct of this is that the MiniDFSCluster will > become active before it knows what NN is active, which can lead to NPEs when > calling getActiveNN(). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715608#comment-15715608 ] Kihwal Lee commented on HDFS-11146: --- Sorry, I am busy and won't be able to review it properly soon. I will probably get to it next week. I would pay close attention to compatibility, interactions with block report lease, etc. > Excess replicas will not be deleted until all storages's FBR received after > failover > > > Key: HDFS-11146 > URL: https://issues.apache.org/jira/browse/HDFS-11146 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-11146.patch > > > Excess replicas will not be deleted until all storages's FBR received after > failover. > Thinking following soultion can help. > *Solution:* > I think after failover, As DNs aware of failover ,so they can send another > block report (FBR) irrespective of interval.May be some shuffle can be done, > similar to initial delay. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Assignee: Weiwei Yang > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Assignee: Weiwei Yang >Priority: Critical > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715214#comment-15715214 ] Kihwal Lee commented on HDFS-11198: --- sure go ahead. > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Assignee: Weiwei Yang >Priority: Critical > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8674) Improve performance of postponed block scans
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8674: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-2.8. Thanks for working on this, Daryn. > Improve performance of postponed block scans > > > Key: HDFS-8674 > URL: https://issues.apache.org/jira/browse/HDFS-8674 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-8674.2.patch, HDFS-8674.branch-2.patch, > HDFS-8674.patch, HDFS-8674.patch, HDFS-8674.trunk.2.patch, > HDFS-8674.trunk.patch > > > When a standby goes active, it marks all nodes as "stale" which will cause > block invalidations for over-replicated blocks to be queued until full block > reports are received from the nodes with the block. The replication monitor > scans the queue with O(N) runtime. It picks a random offset and iterates > through the set to randomize blocks scanned. > The result is devastating when a cluster loses multiple nodes during a > rolling upgrade. Re-replication occurs, the nodes come back, the excess block > invalidations are postponed. Rescanning just 2k blocks out of millions of > postponed blocks may take multiple seconds. During the scan, the write lock > is held which stalls all other processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Description: The new NN UI shows links to DN web pages, but since the link is from the info address returned from jmx, it is in the IP address:port form. This breaks if users are using filters utilizing cookies. Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it does not break any existing functions. It just doesn't work properly in certain environments. was: The new NN UI shows link to DN web pages, but since the link is from the info address returned from jmx, it is in IP address:port form. This breaks if users are using filters utilizing cookies. Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it does not break any existing functions. It just doesn't work properly in certain environments. > NN UI should link DN web address using hostnames > > > Key: HDFS-11198 > URL: https://issues.apache.org/jira/browse/HDFS-11198 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Kihwal Lee >Priority: Critical > > The new NN UI shows links to DN web pages, but since the link is from the > info address returned from jmx, it is in the IP address:port form. This > breaks if users are using filters utilizing cookies. > Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it > does not break any existing functions. It just doesn't work properly in > certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712633#comment-15712633 ] Kihwal Lee commented on HDFS-8674: -- +1 The patch looks good. > Improve performance of postponed block scans > > > Key: HDFS-8674 > URL: https://issues.apache.org/jira/browse/HDFS-8674 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-8674.2.patch, HDFS-8674.branch-2.patch, > HDFS-8674.patch, HDFS-8674.patch, HDFS-8674.trunk.2.patch, > HDFS-8674.trunk.patch > > > When a standby goes active, it marks all nodes as "stale" which will cause > block invalidations for over-replicated blocks to be queued until full block > reports are received from the nodes with the block. The replication monitor > scans the queue with O(N) runtime. It picks a random offset and iterates > through the set to randomize blocks scanned. > The result is devastating when a cluster loses multiple nodes during a > rolling upgrade. Re-replication occurs, the nodes come back, the excess block > invalidations are postponed. Rescanning just 2k blocks out of millions of > postponed blocks may take multiple seconds. During the scan, the write lock > is held which stalls all other processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11198) NN UI should link DN web address using hostnames
Kihwal Lee created HDFS-11198: - Summary: NN UI should link DN web address using hostnames Key: HDFS-11198 URL: https://issues.apache.org/jira/browse/HDFS-11198 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Kihwal Lee Priority: Critical The new NN UI shows link to DN web pages, but since the link is from the info address returned from jmx, it is in IP address:port form. This breaks if users are using filters utilizing cookies. Since this is a new feature in 2.8, I didn't mark it as a blocker. I.e. it does not break any existing functions. It just doesn't work properly in certain environments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans
[
https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712103#comment-15712103
]
Kihwal Lee commented on HDFS-8674:
--
{{LightWeightLinkedSet}} does not support {{remove()}}.
> Improve performance of postponed block scans
>
>
> Key: HDFS-8674
> URL: https://issues.apache.org/jira/browse/HDFS-8674
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-8674.2.patch, HDFS-8674.patch, HDFS-8674.patch,
> HDFS-8674.trunk.patch
>
>
> When a standby goes active, it marks all nodes as "stale" which will cause
> block invalidations for over-replicated blocks to be queued until full block
> reports are received from the nodes with the block. The replication monitor
> scans the queue with O(N) runtime. It picks a random offset and iterates
> through the set to randomize blocks scanned.
> The result is devastating when a cluster loses multiple nodes during a
> rolling upgrade. Re-replication occurs, the nodes come back, the excess block
> invalidations are postponed. Rescanning just 2k blocks out of millions of
> postponed blocks may take multiple seconds. During the scan, the write lock
> is held which stalls all other processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11131) TestThrottledAsyncChecker#testContextIsPassed is flaky
[
https://issues.apache.org/jira/browse/HDFS-11131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712070#comment-15712070
]
Kihwal Lee commented on HDFS-11131:
---
{{testCancellation}} also fails. Failure doesn't seem to be rare.
{noformat}
---
Test set:
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
---
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 60.441 sec <<<
FAILURE!
- in org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
testCancellation(org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker)
Time elapsed: 60.01 sec <<< ERROR!
java.lang.Exception: test timed out after 6 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker.testCancellation(TestThrottledAsyncChecker.java:118)
{noformat}
This was the only entry in the log.
{noformat}
2016-12-01 08:09:15,390 [pool-5-thread-1] INFO
checker.TestThrottledAsyncChecker (TestThrottledAsyncChecker.java:check(245))
- LatchedCheckable
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable@74224991
waiting.
{noformat}
> TestThrottledAsyncChecker#testContextIsPassed is flaky
> --
>
> Key: HDFS-11131
> URL: https://issues.apache.org/jira/browse/HDFS-11131
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> This test failed in a few precommit runs. e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/17481/testReport/org.apache.hadoop.hdfs.server.datanode.checker/TestThrottledAsyncChecker/testContextIsPassed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId
[
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712026#comment-15712026
]
Kihwal Lee commented on HDFS-11179:
---
We have been running with HDFS-8674 internally for quite some time and don't
remember seeing the count going negative. It removes the separate counter and
simply uses the size of the data structure, so it is very likely that it fixes
what you saw.
> LightWeightHashSet can't remove blocks correctly which have a large number
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
> Our test cluster has faced a problem that
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}}
> recently. When {{LightWeightHashSet}} remove blocks which have a large number
> blockId, overflows happen and the blocks can't be removed correctly(, let
> alone ec blocks whose blockId starts with the minimum of long).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang
[
https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710722#comment-15710722
]
Kihwal Lee commented on HDFS-11192:
---
Was it hitting its ulimit on # of threads/processes? If that is the case,
other bad things can happen even if this stage doesn't cause any trouble. E.g.
replication queue init will be done by starting up a separate thread. If that
doesn't happen, the user will be in a bigger trouble. A potential improvement
will be to terminate NN if this happens.
> OOM during Quota Initialization lead to Namenode hang
> -
>
> Key: HDFS-11192
> URL: https://issues.apache.org/jira/browse/HDFS-11192
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: namenodeThreadDump.out
>
>
> AFAIK ,In RecurisveTask Execution, When ForkjoinThreadpool's thread dies or
> not able to create,it will not notify the parent.Parent still waiting for the
> notify call..that's not timed waiting also.
> *Trace from Namenode log*
> {noformat}
> Exception in thread "ForkJoinPool-1-worker-2" Exception in thread
> "ForkJoinPool-1-worker-3" java.lang.OutOfMemoryError: unable to create new
> native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans
[
https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709744#comment-15709744
]
Kihwal Lee commented on HDFS-8674:
--
Looks like {{HDFS-8674.2.patch}} is for branch-2. Can you post a trunk version?
> Improve performance of postponed block scans
>
>
> Key: HDFS-8674
> URL: https://issues.apache.org/jira/browse/HDFS-8674
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-8674.2.patch, HDFS-8674.patch, HDFS-8674.patch
>
>
> When a standby goes active, it marks all nodes as "stale" which will cause
> block invalidations for over-replicated blocks to be queued until full block
> reports are received from the nodes with the block. The replication monitor
> scans the queue with O(N) runtime. It picks a random offset and iterates
> through the set to randomize blocks scanned.
> The result is devastating when a cluster loses multiple nodes during a
> rolling upgrade. Re-replication occurs, the nodes come back, the excess block
> invalidations are postponed. Rescanning just 2k blocks out of millions of
> postponed blocks may take multiple seconds. During the scan, the write lock
> is held which stalls all other processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8674) Improve performance of postponed block scans
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8674: - Target Version/s: 2.8.0, 3.0.0-alpha2 (was: 2.8.0, 2.7.4, 3.0.0-alpha2) > Improve performance of postponed block scans > > > Key: HDFS-8674 > URL: https://issues.apache.org/jira/browse/HDFS-8674 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-8674.2.patch, HDFS-8674.patch, HDFS-8674.patch > > > When a standby goes active, it marks all nodes as "stale" which will cause > block invalidations for over-replicated blocks to be queued until full block > reports are received from the nodes with the block. The replication monitor > scans the queue with O(N) runtime. It picks a random offset and iterates > through the set to randomize blocks scanned. > The result is devastating when a cluster loses multiple nodes during a > rolling upgrade. Re-replication occurs, the nodes come back, the excess block > invalidations are postponed. Rescanning just 2k blocks out of millions of > postponed blocks may take multiple seconds. During the scan, the write lock > is held which stalls all other processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.
[ https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709732#comment-15709732 ] Kihwal Lee commented on HDFS-11180: --- +1 for the patch. I am not sure whether it is worth mentioning the correctness of the value in the release note. For most metrics uses, if not all, the values won't be practically no more "inaccurate" than before. > Intermittent deadlock in NameNode when failover happens. > > > Key: HDFS-11180 > URL: https://issues.apache.org/jira/browse/HDFS-11180 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Abhishek Modi >Assignee: Akira Ajisaka >Priority: Blocker > Labels: high-availability > Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, > HDFS-11180.02.patch, HDFS-11180.03.patch, HDFS-11180.04.patch, jstack.log > > > It is happening due to metrics getting updated at the same time when failover > is happening. Please find attached jstack at that point of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.
[ https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702354#comment-15702354 ] Kihwal Lee commented on HDFS-11180: --- bq. NameNode holds a lock of FSEditLog and requires a lock of MetricsSystemImpl when registering IPCLoggerChannel metrics. It looks like this deadlock can happen only when the QJM is used. Metrics update does not need a precise txid. We could introduce unsynchronized methods for metrics and perhaps use volatile for txid? > Intermittent deadlock in NameNode when failover happens. > > > Key: HDFS-11180 > URL: https://issues.apache.org/jira/browse/HDFS-11180 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Abhishek Modi > Labels: high-availability > Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, jstack.log > > > It is happening due to metrics getting updated at the same time when failover > is happening. Please find attached jstack at that point of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685014#comment-15685014
]
Kihwal Lee commented on HDFS-10966:
---
Committed to branch-2.7. Thanks for working on this [~mwagner] and finishing it
up, [~zhz].
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10966-branch-2.7.00.patch, HDFS-10966.00.patch,
> HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10966:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.7.4
Status: Resolved (was: Patch Available)
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10966-branch-2.7.00.patch, HDFS-10966.00.patch,
> HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685011#comment-15685011
]
Kihwal Lee commented on HDFS-10966:
---
The 2.7 patch looks good.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966-branch-2.7.00.patch, HDFS-10966.00.patch,
> HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684079#comment-15684079
]
Kihwal Lee commented on HDFS-10966:
---
I've committed this to trunk through branch-2.8. Please provide a 2.7 version
of the patch.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch, HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683978#comment-15683978
]
Kihwal Lee commented on HDFS-10966:
---
+1
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch, HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8824) Do not use small blocks for balancing the cluster
[ https://issues.apache.org/jira/browse/HDFS-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677540#comment-15677540 ] Kihwal Lee commented on HDFS-8824: -- While this will initially increase the efficiency of balancing, it is not without a negative side-effect. Older nodes in a cluster will slowly filled with smaller blocks as time goes on. This is accelerated if the cluster is heterogeneous. The smaller nodes will fill up more quickly/frequently and the balancer will move only big blocks out of those nodes. As more balacing happens, those nodes will contain more and more small blocks. If sufficient time passes, the blocks on those nodes will almost entirely small. This feature can be enabled for quickly resolving a storage balance issue, but long-term use can have unintended side-effect. Fortunately, we have not released any (other than alpha) with this feature. We can include more information in the release note and/or address the issue in the code/config. > Do not use small blocks for balancing the cluster > - > > Key: HDFS-8824 > URL: https://issues.apache.org/jira/browse/HDFS-8824 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8824_20150727b.patch, h8824_20150811b.patch > > > Balancer gets datanode block lists from NN and then move the blocks in order > to balance the cluster. It should not use the blocks with small size since > moving the small blocks generates a lot of overhead and the small blocks do > not help balancing the cluster much. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673862#comment-15673862 ] Kihwal Lee commented on HDFS-11146: --- NN could tell datanodes to send FBR in a heartbeat response. This way, NN can decide when to get a FBR from whom, instead of DNs randomly sending. > Excess replicas will not be deleted until all storages's FBR received after > failover > > > Key: HDFS-11146 > URL: https://issues.apache.org/jira/browse/HDFS-11146 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > > Excess replicas will not be deleted until all storages's FBR received after > failover. > Thinking following soultion can help. > *Solution:* > I think after failover, As DNs aware of failover ,so they can send another > block report (FBR) irrespective of interval.May be some shuffle can be done, > similar to initial delay. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11153) RPC Client detect address changed should reconnect immediately
[
https://issues.apache.org/jira/browse/HDFS-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673846#comment-15673846
]
Kihwal Lee commented on HDFS-11153:
---
You can enable this through config. Here is an example:
{noformat}
dfs.client.retry.policy.enabled=true
dfs.client.retry.policy.spec=1000,2
{noformat}
> RPC Client detect address changed should reconnect immediately
> --
>
> Key: HDFS-11153
> URL: https://issues.apache.org/jira/browse/HDFS-11153
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.0.0-alpha1
>Reporter: DENG FEI
>
> HA mode,the _*"ipc.client.connect.max.retries.on.timeouts"*_ and
> _*"ipc.client.connect.max.retries"*_ is set zero,but if met active NN's ip
> changed,it will detect the change,but won't reconnect because exceed the max
> retry times,after do 15 times failover and then throw connection or standby
> exception.
> maybe if found the address is changed,should reconnect immediately no matter
> the retry times limit.
>
> log is below:
> {noformat}
> 2016-11-16 17:00:20,844 (WARN org.apache.hadoop.ipc.Client 510): Address
> change detected. Old: *:9000 New: X:9000
> 2016-11-16 17:01:09,893 (WARN org.apache.hadoop.ipc.Client 510): Address
> change detected. Old: *::9000 New: X:9000
> 2016-11-16 17:01:09,893 (WARN
> org.apache.hadoop.io.retry.RetryInvocationHandler 118): Exception while
> invoking class
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo.
> Not retrying because failovers (15) exceeded maximum allowed (15)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671089#comment-15671089
]
Kihwal Lee commented on HDFS-10966:
---
the patch looks reasonable in general.
- It is replacing the logic introduced by HDFS-4261, so please double check
there is no regression. Does the run-time of related test cases change?
- Please document the new config variable in {{hdfs-default.xml}}.
- If {{chooseNextMove()}} keeps failing, the while loop can spin for up to one
minute. How about inserting a small delay, say, 100ms?
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-8870) Lease is leaked on write failure
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668304#comment-15668304 ] Kihwal Lee edited comment on HDFS-8870 at 11/15/16 9:02 PM: Committed this to trunk, branch-2 and branch-2.8. Do you have a patch for 2.7 and 2.6? was (Author: kihwal): Committed this to trunk, branch-2 and branch-2.8. > Lease is leaked on write failure > > > Key: HDFS-8870 > URL: https://issues.apache.org/jira/browse/HDFS-8870 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Kuhu Shukla > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-8870.001.patch > > > Creating this ticket on behalf of [~daryn] > We've seen this in our of our cluster. When a long running process has a > write failure, the lease is leaked and gets renewed until the token is > expired. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8870) Lease is leaked on write failure
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668304#comment-15668304 ] Kihwal Lee commented on HDFS-8870: -- Committed this to trunk, branch-2 and branch-2.8. > Lease is leaked on write failure > > > Key: HDFS-8870 > URL: https://issues.apache.org/jira/browse/HDFS-8870 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Kuhu Shukla > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-8870.001.patch > > > Creating this ticket on behalf of [~daryn] > We've seen this in our of our cluster. When a long running process has a > write failure, the lease is leaked and gets renewed until the token is > expired. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8870) Lease is leaked on write failure
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8870: - Fix Version/s: 3.0.0-alpha2 2.8.0 > Lease is leaked on write failure > > > Key: HDFS-8870 > URL: https://issues.apache.org/jira/browse/HDFS-8870 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Kuhu Shukla > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-8870.001.patch > > > Creating this ticket on behalf of [~daryn] > We've seen this in our of our cluster. When a long running process has a > write failure, the lease is leaked and gets renewed until the token is > expired. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8870) Lease is leaked on write failure
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8870: - Assignee: Kuhu Shukla (was: Daryn Sharp) > Lease is leaked on write failure > > > Key: HDFS-8870 > URL: https://issues.apache.org/jira/browse/HDFS-8870 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Kuhu Shukla > Attachments: HDFS-8870.001.patch > > > Creating this ticket on behalf of [~daryn] > We've seen this in our of our cluster. When a long running process has a > write failure, the lease is leaked and gets renewed until the token is > expired. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8870) Lease is leaked on write failure
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668242#comment-15668242 ] Kihwal Lee commented on HDFS-8870: -- +1 The patch looks good. > Lease is leaked on write failure > > > Key: HDFS-8870 > URL: https://issues.apache.org/jira/browse/HDFS-8870 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Kuhu Shukla > Attachments: HDFS-8870.001.patch > > > Creating this ticket on behalf of [~daryn] > We've seen this in our of our cluster. When a long running process has a > write failure, the lease is leaked and gets renewed until the token is > expired. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11056) Concurrent append and read operations lead to checksum error
[
https://issues.apache.org/jira/browse/HDFS-11056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657402#comment-15657402
]
Kihwal Lee commented on HDFS-11056:
---
+1 for the 2.7 patch. It looks to be a correct port. Thanks [~jojochuang].
> Concurrent append and read operations lead to checksum error
>
>
> Key: HDFS-11056
> URL: https://issues.apache.org/jira/browse/HDFS-11056
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, httpfs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11056.001.patch, HDFS-11056.002.patch,
> HDFS-11056.branch-2.7.patch, HDFS-11056.branch-2.patch,
> HDFS-11056.reproduce.patch
>
>
> If there are two clients, one of them open-append-close a file continuously,
> while the other open-read-close the same file continuously, the reader
> eventually gets a checksum error in the data read.
> On my local Mac, it takes a few minutes to produce the error. This happens to
> httpfs clients, but there's no reason not believe this happens to any append
> clients.
> I have a unit test that demonstrates the checksum error. Will attach later.
> Relevant log:
> {quote}
> 2016-10-25 15:34:45,153 INFO audit - allowed=trueugi=weichiu
> (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/tmp/bar.txt
> dst=nullperm=null proto=rpc
> 2016-10-25 15:34:45,155 INFO DataNode - Receiving
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 src:
> /127.0.0.1:51130 dest: /127.0.0.1:50131
> 2016-10-25 15:34:45,155 INFO FsDatasetImpl - Appending to FinalizedReplica,
> blk_1073741825_1182, FINALIZED
> getNumBytes() = 182
> getBytesOnDisk() = 182
> getVisibleLength()= 182
> getVolume() =
> /Users/weichiu/sandbox/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/dfs/data/data1
> getBlockURI() =
> file:/Users/weichiu/sandbox/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/dfs/data/data1/current/BP-837130339-172.16.1.88-1477434851452/current/finalized/subdir0/subdir0/blk_1073741825
> 2016-10-25 15:34:45,167 INFO DataNode - opReadBlock
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 received exception
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> 2016-10-25 15:34:45,167 WARN DataNode -
> DatanodeRegistration(127.0.0.1:50131,
> datanodeUuid=41c96335-5e4b-4950-ac22-3d21b353abb8, infoPort=50133,
> infoSecurePort=0, ipcPort=50134,
> storageInfo=lv=-57;cid=testClusterID;nsid=1472068852;c=1477434851452):Got
> exception while serving
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 to /127.0.0.1:51121
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockInputStream(FsDatasetImpl.java:773)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:400)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:581)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:150)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:102)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-25 15:34:45,168 INFO FSNamesystem -
> updatePipeline(blk_1073741825_1182, newGS=1183, newLength=182,
> newNodes=[127.0.0.1:50131], client=DFSClient_NONMAPREDUCE_-1743096965_197)
> 2016-10-25 15:34:45,168 ERROR DataNode - 127.0.0.1:50131:DataXceiver error
> processing READ_BLOCK operation src: /127.0.0.1:51121 dst: /127.0.0.1:50131
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockInputStream(FsDatasetImpl.java:773)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:400)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:581)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:150)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:102)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-25 15:34:45,168 INFO FSNamesystem -
> updatePipeline(blk_1073741825_1182 => blk_1073741825_1183) success
> 2016-10-25 15:34:45,170 WARN DFSClient - Found Checksum error for
>
[jira] [Commented] (HDFS-11114) Support for running async disk checks in DataNode
[
https://issues.apache.org/jira/browse/HDFS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651285#comment-15651285
]
Kihwal Lee commented on HDFS-4:
---
This failed recently.
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/220/testReport/junit/org.apache.hadoop.hdfs.server.datanode.checker/TestThrottledAsyncChecker/testContextIsPassed/
{noformat}
Stacktrace
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker.testContextIsPassed(TestThrottledAsyncChecker.java:174)
Standard Output
2016-11-09 09:25:11,786 [pool-2-thread-1] INFO
checker.TestThrottledAsyncChecker (TestThrottledAsyncChecker.java:check(245)) -
LatchedCheckable
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable@7aeb663a
waiting.
2016-11-09 09:25:11,790 [pool-2-thread-1] INFO
checker.TestThrottledAsyncChecker
(TestThrottledAsyncChecker.java:onFailure(272)) - onFailure callback invoked
for
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable@7aeb663a
with exception
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable.check(TestThrottledAsyncChecker.java:246)
at
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable.check(TestThrottledAsyncChecker.java:239)
at
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:130)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-09 09:25:11,817 [pool-5-thread-1] INFO
checker.TestThrottledAsyncChecker (TestThrottledAsyncChecker.java:check(245)) -
LatchedCheckable
org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker$LatchedCheckable@12aed2de
waiting.
{noformat}
> Support for running async disk checks in DataNode
> -
>
> Key: HDFS-4
> URL: https://issues.apache.org/jira/browse/HDFS-4
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0-alpha2
>
>
> Introduce support for running async checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11115) Remove bytes2Array and string2Bytes
[
https://issues.apache.org/jira/browse/HDFS-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648021#comment-15648021
]
Kihwal Lee commented on HDFS-5:
---
You are undoing this optimization.
{code}
// Using the charset canonical name for String/byte[] conversions is much
// more efficient due to use of cached encoders/decoders.
private static final String UTF8_CSN = StandardCharsets.UTF_8.name();
{code}
> Remove bytes2Array and string2Bytes
> ---
>
> Key: HDFS-5
> URL: https://issues.apache.org/jira/browse/HDFS-5
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Reporter: Sahil Kang
>Priority: Minor
>
> In DFSUtilClient.java we have something like:
> {code: language=java}
> public static byte[] string2Bytes(String str) {
> try {
> return str.getBytes("UTF-8");
> } catch (UnsupportedEncodingException e) {
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> static String bytes2String(byte[] bytes, int offset, int length) {
> try {
> return new String(bytes, offset, length, "UTF-8");
> } catch (UnsupportedEncodingException e) {
> throw new IllegalArgumentException("UTF8 encoding is not supported", e);
> }
> }
> {code}
> Using StandardCharsets, these methods become trivial:
> {code: language=java}
> public static byte[] string2Bytes(String str) {
> return str.getBytes(StandardCharsets.UTF_8);
> }
> static String bytes2String(byte[] bytes, int offset, int length) {
> return new String(bytes, offset, length, StandardCharsets.UTF_8);
> }
> {code}
> I think we should remove these methods and use StandardCharsets whenever we
> need to convert between bytes and strings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11114) Support for running async disk checks in DataNode
[ https://issues.apache.org/jira/browse/HDFS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647937#comment-15647937 ] Kihwal Lee commented on HDFS-4: --- It broke branch-2. [~steve_l] also noticed it. > Support for running async disk checks in DataNode > - > > Key: HDFS-4 > URL: https://issues.apache.org/jira/browse/HDFS-4 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.9.0 > > > Introduce support for running async checks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11056) Concurrent append and read operations lead to checksum error
[
https://issues.apache.org/jira/browse/HDFS-11056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633249#comment-15633249
]
Kihwal Lee commented on HDFS-11056:
---
I was looking at the patch since yesterday. It looks like the partial chunk sum
is loaded from disk and saved in memory before it is modified. That seems like
a correct approach. +1
> Concurrent append and read operations lead to checksum error
>
>
> Key: HDFS-11056
> URL: https://issues.apache.org/jira/browse/HDFS-11056
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, httpfs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-11056.001.patch, HDFS-11056.002.patch,
> HDFS-11056.reproduce.patch
>
>
> If there are two clients, one of them open-append-close a file continuously,
> while the other open-read-close the same file continuously, the reader
> eventually gets a checksum error in the data read.
> On my local Mac, it takes a few minutes to produce the error. This happens to
> httpfs clients, but there's no reason not believe this happens to any append
> clients.
> I have a unit test that demonstrates the checksum error. Will attach later.
> Relevant log:
> {quote}
> 2016-10-25 15:34:45,153 INFO audit - allowed=trueugi=weichiu
> (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/tmp/bar.txt
> dst=nullperm=null proto=rpc
> 2016-10-25 15:34:45,155 INFO DataNode - Receiving
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 src:
> /127.0.0.1:51130 dest: /127.0.0.1:50131
> 2016-10-25 15:34:45,155 INFO FsDatasetImpl - Appending to FinalizedReplica,
> blk_1073741825_1182, FINALIZED
> getNumBytes() = 182
> getBytesOnDisk() = 182
> getVisibleLength()= 182
> getVolume() =
> /Users/weichiu/sandbox/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/dfs/data/data1
> getBlockURI() =
> file:/Users/weichiu/sandbox/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/dfs/data/data1/current/BP-837130339-172.16.1.88-1477434851452/current/finalized/subdir0/subdir0/blk_1073741825
> 2016-10-25 15:34:45,167 INFO DataNode - opReadBlock
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 received exception
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> 2016-10-25 15:34:45,167 WARN DataNode -
> DatanodeRegistration(127.0.0.1:50131,
> datanodeUuid=41c96335-5e4b-4950-ac22-3d21b353abb8, infoPort=50133,
> infoSecurePort=0, ipcPort=50134,
> storageInfo=lv=-57;cid=testClusterID;nsid=1472068852;c=1477434851452):Got
> exception while serving
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182 to /127.0.0.1:51121
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockInputStream(FsDatasetImpl.java:773)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:400)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:581)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:150)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:102)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-25 15:34:45,168 INFO FSNamesystem -
> updatePipeline(blk_1073741825_1182, newGS=1183, newLength=182,
> newNodes=[127.0.0.1:50131], client=DFSClient_NONMAPREDUCE_-1743096965_197)
> 2016-10-25 15:34:45,168 ERROR DataNode - 127.0.0.1:50131:DataXceiver error
> processing READ_BLOCK operation src: /127.0.0.1:51121 dst: /127.0.0.1:50131
> java.io.IOException: No data exists for block
> BP-837130339-172.16.1.88-1477434851452:blk_1073741825_1182
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockInputStream(FsDatasetImpl.java:773)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:400)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:581)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:150)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:102)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-25 15:34:45,168 INFO FSNamesystem -
> updatePipeline(blk_1073741825_1182 => blk_1073741825_1183) success
> 2016-10-25 15:34:45,170 WARN DFSClient - Found Checksum error for
>
[jira] [Commented] (HDFS-11090) Leave safemode immediately if all blocks have reported in
[ https://issues.apache.org/jira/browse/HDFS-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630287#comment-15630287 ] Kihwal Lee commented on HDFS-11090: --- I agree with [~jingzhao]. If people don't want safe mode extension, it can be configured off. But something like this might make sense to standby namenodes, since they are not going to start replicating after leaving a safe mode. > Leave safemode immediately if all blocks have reported in > - > > Key: HDFS-11090 > URL: https://issues.apache.org/jira/browse/HDFS-11090 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.3 >Reporter: Andrew Wang >Assignee: Yiqun Lin > Attachments: HDFS-11090.001.patch > > > Startup safemode is triggered by two thresholds: % blocks reported in, and > min # datanodes. It's extended by an interval (default 30s) until these two > thresholds are met. > Safemode extension is helpful when the cluster has data, and the default % > blocks threshold (0.99) is used. It gives DNs a little extra time to report > in and thus avoid unnecessary replication work. > However, we can leave startup safemode early if 100% of blocks have reported > in. > Note that operators sometimes change the % blocks threshold to > 1 to never > automatically leave safemode. We should maintain this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11078) NPE in LazyPersistFileScrubber
[ https://issues.apache.org/jira/browse/HDFS-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625742#comment-15625742 ] Kihwal Lee commented on HDFS-11078: --- Since this can happen legitimately and does not require any further attention from users, logging at debug or info at most seems preferable. > NPE in LazyPersistFileScrubber > -- > > Key: HDFS-11078 > URL: https://issues.apache.org/jira/browse/HDFS-11078 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: HDFS-11078.000.patch > > > If a block is removed, it will be removed from the block map. When the > clearCorruptLazyPersistFiles() tries to delete the block, it may already be > deleted and generate a null pointer exception. > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.clearCorruptLazyPersistFiles(FSNamesystem.java:3820) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:3851) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10997) Reduce number of path resolving methods
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10997: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed to branch-2 and branch-2.8. Thanks for working on this, Daryn. > Reduce number of path resolving methods > --- > > Key: HDFS-10997 > URL: https://issues.apache.org/jira/browse/HDFS-10997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10997.1.patch, HDFS-10997.2.patch, > HDFS-10997.branch-2.patch, HDFS-10997.patch > > > FSDirectory contains many methods for resolving paths to an IIP and/or inode. > These should be unified into a couple methods that will consistently do the > basics of resolving reserved paths, blocking write ops from snapshot paths, > verifying ancestors as directories, and throwing if symlinks are encountered. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10997) Reduce number of path resolving methods
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625365#comment-15625365 ] Kihwal Lee commented on HDFS-10997: --- The branch-2 patch looks good. I will commit it shortly. > Reduce number of path resolving methods > --- > > Key: HDFS-10997 > URL: https://issues.apache.org/jira/browse/HDFS-10997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10997.1.patch, HDFS-10997.2.patch, > HDFS-10997.branch-2.patch, HDFS-10997.patch > > > FSDirectory contains many methods for resolving paths to an IIP and/or inode. > These should be unified into a couple methods that will consistently do the > basics of resolving reserved paths, blocking write ops from snapshot paths, > verifying ancestors as directories, and throwing if symlinks are encountered. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11069:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.7.4
Status: Resolved (was: Patch Available)
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612896#comment-15612896
]
Kihwal Lee commented on HDFS-11069:
---
Committed to trunk, branch-2, branch-2.8 and branch-2.7.
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612863#comment-15612863
]
Kihwal Lee commented on HDFS-11069:
---
Thanks for the review, Daryn. I've verified it working as expected (allow
local, deny remote) on a secure cluster.
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee reopened HDFS-10455:
---
Reverted the commits.
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Attachments: HDFS-10455.000.patch, HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10455:
--
Fix Version/s: (was: 3.0.0-alpha2)
(was: 2.8.0)
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Attachments: HDFS-10455.000.patch, HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612831#comment-15612831
]
Kihwal Lee commented on HDFS-11069:
---
{{TestPermission}} is broken by HDFS-10455.
The other test passes.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hdfs.TestFileCorruption
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.826 sec - in
org.apache.hadoop.hdfs.TestFileCorruption
Results :
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612820#comment-15612820
]
Kihwal Lee commented on HDFS-10455:
---
The precommit was from May. You should have re-run it.
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10455.000.patch, HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612810#comment-15612810
]
Kihwal Lee commented on HDFS-10455:
---
It broke {{TestPermission}}
{noformat}
Running org.apache.hadoop.security.TestPermission
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.996 sec <<<
FAILURE! - in org.apache.hadoop.security.TestPermission
testFilePermission(org.apache.hadoop.security.TestPermission) Time elapsed:
5.926 sec <<< FAILURE!
java.lang.AssertionError:
Expected: a string starting with "User does not belong to"
but: was "User user-1375012529 does not belong to nogroup
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:92)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1727)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:834)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:488)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2535)
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:865)
at org.junit.Assert.assertThat(Assert.java:832)
at
org.apache.hadoop.security.TestPermission.testNonSuperCannotChangeToOtherGroup(TestPermission.java:340)
at
org.apache.hadoop.security.TestPermission.testFilePermission(TestPermission.java:305)
{noformat}
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10455.000.patch, HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11063) Set NameNode RPC server handler thread name with more descriptive information about the RPC call.
[
https://issues.apache.org/jira/browse/HDFS-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612485#comment-15612485
]
Kihwal Lee commented on HDFS-11063:
---
Once side-effect is that the namenode log can get bigger and contain redundant
information.
> Set NameNode RPC server handler thread name with more descriptive information
> about the RPC call.
> -
>
> Key: HDFS-11063
> URL: https://issues.apache.org/jira/browse/HDFS-11063
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chris Nauroth
>
> We often run {{jstack}} on a NameNode process as a troubleshooting step if it
> is suffering high load or appears to be hanging. By reading the stack trace,
> we can identify if a caller is blocked inside an expensive operation. This
> would be even more helpful if we updated the RPC server handler thread name
> with more descriptive information about the RPC call. This could include the
> calling user, the called RPC method, and the most significant argument to
> that method (most likely the path).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11069:
--
Attachment: HDFS-11069.patch
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11069:
--
Status: Patch Available (was: Open)
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-11069.patch
>
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11069:
--
Description:
The current implementation of {{checkSuperuserPrivilege()}} allows the datanode
user from any node to be recognized as a super user. If one datanode is
compromised, the intruder can issue {{shutdownDatanode()}}, {{evictWriters()}},
{{triggerBlockReport()}}, etc. against all other datanodes. Although this does
not expose stored data, it can cause service disruptions.
This needs to be tightened to allow only the local datanode user.
was:
The current implementation of {{checkSuperuserPrivilege()}} allows the datanode
user from any node to be recognized as a super user. If one datanode is
compromised, the intruder can issue {{shutdownDatanode()}}, {{evictWriters()}},
{{triggerBlockReport()}}, etc. against all other datanodes.
This needs to be tightened to allow only the local datanode user.
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes. Although this does not expose stored data, it can cause service
> disruptions.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612304#comment-15612304
]
Kihwal Lee commented on HDFS-11069:
---
{code}
private void checkSuperuserPrivilege() {
...
if (callerUgi.getShortUserName().equals(dnUserName)) {
return;
}
...
{code}
Instead of checking only the short name, the full name should be checked. E.g.
{{dn_user/datanode01.yourdomain@yourdomain.com}} instead of simply
{{dn_user}}.
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee reassigned HDFS-11069:
-
Assignee: Kihwal Lee
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC
[
https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11069:
--
Issue Type: Improvement (was: Bug)
> Tighten the authorization of datanode RPC
> -
>
> Key: HDFS-11069
> URL: https://issues.apache.org/jira/browse/HDFS-11069
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Reporter: Kihwal Lee
>
> The current implementation of {{checkSuperuserPrivilege()}} allows the
> datanode user from any node to be recognized as a super user. If one
> datanode is compromised, the intruder can issue {{shutdownDatanode()}},
> {{evictWriters()}}, {{triggerBlockReport()}}, etc. against all other
> datanodes.
> This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11069) Tighten the authorization of datanode RPC
Kihwal Lee created HDFS-11069:
-
Summary: Tighten the authorization of datanode RPC
Key: HDFS-11069
URL: https://issues.apache.org/jira/browse/HDFS-11069
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Kihwal Lee
The current implementation of {{checkSuperuserPrivilege()}} allows the datanode
user from any node to be recognized as a super user. If one datanode is
compromised, the intruder can issue {{shutdownDatanode()}}, {{evictWriters()}},
{{triggerBlockReport()}}, etc. against all other datanodes.
This needs to be tightened to allow only the local datanode user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11053) Unnecessary superuser check in versionRequest()
[
https://issues.apache.org/jira/browse/HDFS-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-11053:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.7.4
Status: Resolved (was: Patch Available)
Thanks for the reviews. I've committed it to trunk through branch-2.7.
> Unnecessary superuser check in versionRequest()
> ---
>
> Key: HDFS-11053
> URL: https://issues.apache.org/jira/browse/HDFS-11053
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, security
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-11053.patch
>
>
> The {{versionRequest()}} call does not return any sensitive information. It
> is mainly used for sanity checks. The presence of
> {{checkSuperuserPrivilege()}} forces users to run datanode as a hdfs
> superuser.
> In secure setup, a keytab obtained from a compromised datanode can allow the
> intruder to gain hdfs superuser privilege. We should allow datanodes to be
> run as non-hdfs-superuser by removing {{checkSuperuserPrivilege()}} from
> {{versionRequest()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
