[jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941654#comment-16941654
]
Xiaoqiao He commented on HDFS-14305:
Thanks [~shv],[~vagarychen] for your explains. it makes sense to me.
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, security
>Reporter: Chao Sun
>Assignee: Konstantin Shvachko
>Priority: Major
> Labels: multi-sbnn, release-blocker
> Fix For: 2.10.0, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14305-007.patch, HDFS-14305-008.patch,
> HDFS-14305.001.patch, HDFS-14305.002.patch, HDFS-14305.003.patch,
> HDFS-14305.004.patch, HDFS-14305.005.patch, HDFS-14305.006.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941748#comment-16941748
]
Xiaoqiao He commented on HDFS-14882:
[~lukmajercak], Thanks for your comments. This ticket is plan to adjust and
sort location considering nodes' load whose distance to client is same.
{quote}I only wish we could also use some sort of an estimate of load that
we've already scheduled on each DN, not just xceivers reported by them.{quote}
Sorry I don't understand that clearly. IIUC, we are also estimating load of
node using its xceiver count when schedule write ops currently. Please correct
me if i am wrong. Thanks.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14498) LeaseManager can loop forever on the file for which create has failed
[
https://issues.apache.org/jira/browse/HDFS-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941859#comment-16941859
]
Xiaoqiao He commented on HDFS-14498:
Thank [~leigh.jo...@ripjar.com] for your report. Would you like offer some log,
jstack information, it will be more helpful to find the root cause.
> LeaseManager can loop forever on the file for which create has failed
> --
>
> Key: HDFS-14498
> URL: https://issues.apache.org/jira/browse/HDFS-14498
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Sergey Shelukhin
>Priority: Major
>
> The logs from file creation are long gone due to infinite lease logging,
> however it presumably failed... the client who was trying to write this file
> is definitely long dead.
> The version includes HDFS-4882.
> We get this log pattern repeating infinitely:
> {noformat}
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: [Lease. Holder:
> DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1] has expired hard
> limit
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.
> Holder: DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1], src=
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.internalReleaseLease:
> Failed to release lease for file . Committed blocks are waiting to be
> minimally replicated. Try again later.
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Cannot release the path
> in the lease [Lease. Holder: DFSClient_NONMAPREDUCE_-20898906_61,
> pending creates: 1]. It will be retried.
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR*
> NameSystem.internalReleaseLease: Failed to release lease for file .
> Committed blocks are waiting to be minimally replicated. Try again later.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3357)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:573)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:509)
> at java.lang.Thread.run(Thread.java:745)
> $ grep -c "Recovering.*DFSClient_NONMAPREDUCE_-20898906_61, pending creates:
> 1" hdfs_nn*
> hdfs_nn.log:1068035
> hdfs_nn.log.2019-05-16-14:1516179
> hdfs_nn.log.2019-05-16-15:1538350
> {noformat}
> Aside from an actual bug fix, it might make sense to make LeaseManager not
> log so much, in case if there are more bugs like this...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940643#comment-16940643
]
Xiaoqiao He commented on HDFS-14090:
Thanks [~crh] for your works and sorry for late response.
1. compile RBF sub-module and run rbf related unit tests, all passed.
2. check isolation logic with test env (use cases: high load of one subcluster
+ unavailability of one subcluster, lightweight load of all subclusters, high
lad of all subclusters) and it meets expectation for me.
3. check code style, it is clean as Jenkins reports above.
+1 for [^HDFS-14090.014.patch] from my side overall. hope to step forward.
Thanks [~crh] again.
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940652#comment-16940652
]
Xiaoqiao He commented on HDFS-14305:
Thanks [~shv],[~arp] very much for your feedback and works. it seems that all
works(include [^HDFS-14305.006.patch] ) we did is just reducing conflict
probability rather than avoid it completely.
{quote}If you start the NNs in arbitrary order, you can get block token
collisions because the ranges will change in 3.2.1 compared to 3.2.0.{quote}
This case seems not eliminate with/without [^HDFS-14305-007.patch] changes.
Please correct me if something wrong. Thanks [~shv] again.
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, security
>Reporter: Chao Sun
>Assignee: Xiaoqiao He
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14305-007.patch, HDFS-14305.001.patch,
> HDFS-14305.002.patch, HDFS-14305.003.patch, HDFS-14305.004.patch,
> HDFS-14305.005.patch, HDFS-14305.006.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14554) Avoid to process duplicate blockreport of datanode when namenode in startup safemode
[ https://issues.apache.org/jira/browse/HDFS-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940646#comment-16940646 ] Xiaoqiao He commented on HDFS-14554: Thanks [~lindongdong] for your comments. This ticket is not valid. I just suggest to try lifeline feature or update safemode checking mechanism/HDFS-14559 is one demo implementation or turn off safemode auto leave when namenode startup if you also meet bottleneck about namenode restart. My own opinion FYI. > Avoid to process duplicate blockreport of datanode when namenode in startup > safemode > > > Key: HDFS-14554 > URL: https://issues.apache.org/jira/browse/HDFS-14554 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14554.001.patch > > > When NameNode restart and enter stage of startup safemode, load of NameNode > could be very high since blockreport request storm from datanodes together. > Sometimes datanode may request blockreport times because NameNode doesn't > return in time and DataNode meet timeoutexception then retry, Thus it > sharpens load of NameNode. > So we should check and filter duplicate blockreport request from one > datannode and reduce load of Namenode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14775) Add Timestamp for longest FSN write/read lock held log
[
https://issues.apache.org/jira/browse/HDFS-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940674#comment-16940674
]
Xiaoqiao He commented on HDFS-14775:
Thanks [~zhangchen] for your works. This improvement is useful for me. I prefer
[^HDFS-14775.002.patch] than the last one, I guess that you remote
AtomicReference for `longestWriteLockHeldInfo` based on only one thread could
hold WriteLock, right? maybe we could keep AtomicReference and it will not take
any performance overhead in my opinion. some other code style comment but not
important.
1. it should be better to follow other alignment type.
{code:java}
+final long currentTimeMs =
+TimeUnit.NANOSECONDS.toMillis(currentTimeStampNanos);
{code}
2. const in FSNamesystemLock#hashCode is just a little confused. is it better
with {{Objects.hashCode}}?
Ping [~xkrogen],[~linyiqun],[~elgoiri] would you help take a look?
> Add Timestamp for longest FSN write/read lock held log
> --
>
> Key: HDFS-14775
> URL: https://issues.apache.org/jira/browse/HDFS-14775
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14775.001.patch, HDFS-14775.002.patch,
> HDFS-14775.003.patch
>
>
> HDFS-13946 improved the log for longest read/write lock held time, it's very
> useful improvement.
> In some condition, we need to locate the detailed call information(user, ip,
> path, etc.) for longest lock holder, but the default throttle interval(10s)
> is too long to find the corresponding audit log. I think we should add the
> timestamp for the {{longestWriteLockHeldStackTrace}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14810) review FSNameSystem editlog sync
[ https://issues.apache.org/jira/browse/HDFS-14810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940660#comment-16940660 ] Xiaoqiao He commented on HDFS-14810: [~jojochuang] gentle reminder again. any update here? > review FSNameSystem editlog sync > > > Key: HDFS-14810 > URL: https://issues.apache.org/jira/browse/HDFS-14810 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14810.001.patch, HDFS-14810.002.patch, > HDFS-14810.003.patch, HDFS-14810.004.patch > > > refactor and unified type of edit log sync in FSNamesystem as HDFS-11246 > mentioned. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14775) Add Timestamp for longest FSN write/read lock held log
[
https://issues.apache.org/jira/browse/HDFS-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940674#comment-16940674
]
Xiaoqiao He edited comment on HDFS-14775 at 9/30/19 6:34 AM:
-
Thanks [~zhangchen] for your works. This improvement is useful for me. I prefer
[^HDFS-14775.002.patch] than the last one, I guess that you remote
AtomicReference for `longestWriteLockHeldInfo` based on only one thread could
hold WriteLock, right? maybe we could keep AtomicReference safety and it will
not take any performance overhead in my opinion. some other code style comment
but not important.
1. it should be better to follow other alignment type.
{code:java}
+final long currentTimeMs =
+TimeUnit.NANOSECONDS.toMillis(currentTimeStampNanos);
{code}
2. const in FSNamesystemLock#hashCode is just a little confused. is it better
with {{Objects.hashCode}}?
Ping [~xkrogen],[~linyiqun],[~elgoiri] would you help take a look?
was (Author: hexiaoqiao):
Thanks [~zhangchen] for your works. This improvement is useful for me. I prefer
[^HDFS-14775.002.patch] than the last one, I guess that you remote
AtomicReference for `longestWriteLockHeldInfo` based on only one thread could
hold WriteLock, right? maybe we could keep AtomicReference and it will not take
any performance overhead in my opinion. some other code style comment but not
important.
1. it should be better to follow other alignment type.
{code:java}
+final long currentTimeMs =
+TimeUnit.NANOSECONDS.toMillis(currentTimeStampNanos);
{code}
2. const in FSNamesystemLock#hashCode is just a little confused. is it better
with {{Objects.hashCode}}?
Ping [~xkrogen],[~linyiqun],[~elgoiri] would you help take a look?
> Add Timestamp for longest FSN write/read lock held log
> --
>
> Key: HDFS-14775
> URL: https://issues.apache.org/jira/browse/HDFS-14775
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14775.001.patch, HDFS-14775.002.patch,
> HDFS-14775.003.patch
>
>
> HDFS-13946 improved the log for longest read/write lock held time, it's very
> useful improvement.
> In some condition, we need to locate the detailed call information(user, ip,
> path, etc.) for longest lock holder, but the default throttle interval(10s)
> is too long to find the corresponding audit log. I think we should add the
> timestamp for the {{longestWriteLockHeldStackTrace}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14882) Consider DataNode load when #getBlockLocation
Xiaoqiao He created HDFS-14882: -- Summary: Consider DataNode load when #getBlockLocation Key: HDFS-14882 URL: https://issues.apache.org/jira/browse/HDFS-14882 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Xiaoqiao He Assignee: Xiaoqiao He Currently, we consider load of datanode when #chooseTarget for writer, however not consider it for reader. Thus, the process slot of datanode could be occupied by #BlockSender for reader, and disk/network will be busy workload, then meet some slow node exception. IIRC same case is reported times. Based on the fact, I propose to consider load for reader same as it did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.001.patch Status: Patch Available (was: Open) submit demo patch and trigger Jenkins. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14952) Skip safemode if blockSafe is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-14952:
---
Attachment: HDFS-14952.001.patch
> Skip safemode if blockSafe is 0 in new NN
> -
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14952) Skip safemode if blockSafe is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966781#comment-16966781
]
Xiaoqiao He commented on HDFS-14952:
Thanks [~rajesh.balamohan] for your report. It speeds extension period (~30s by
default) to leave safemode automatically when block thresholds are met. Agree
that we should not wait extension period when initialize new cluster.
> Skip safemode if blockSafe is 0 in new NN
> -
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Priority: Trivial
> Labels: performance
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14952) Skip safemode if blockSafe is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966788#comment-16966788
]
Xiaoqiao He commented on HDFS-14952:
[^HDFS-14952.001.patch] try to correct the extension period for new cluster
setup, FYI.
> Skip safemode if blockSafe is 0 in new NN
> -
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962758#comment-16962758
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~pifta] for your suggestions,
{quote}I would suggest two more things to do, we might deprecate the old sorter
methods, as we most likely won't need them on the long run, as their use
effectively overrides the new setting, and an update would be nice to the
APIDoc of these methods.{quote}
It make sense for me. I agree that the following methods should be removed.
BTW, this common interface is invoke by different classes. I would like to
update that later.
{code:java}
public void sortByDistance(Node reader, Node[] nodes, int activeLen)
public void sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes,
int activeLen)
{code}
cc [~ayushtkn],[~elgoiri] any other comments?
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch,
> HDFS-14882.009.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962758#comment-16962758
]
Xiaoqiao He edited comment on HDFS-14882 at 10/30/19 7:08 AM:
--
Thanks [~pifta] for your suggestions,
{quote}I would suggest two more things to do, we might deprecate the old sorter
methods, as we most likely won't need them on the long run, as their use
effectively overrides the new setting, and an update would be nice to the
APIDoc of these methods.{quote}
It make sense for me. I agree that the following methods should be removed.
BTW, this common interface is invoke by different classes. I would like to
update that later.
{code:java}
public void sortByDistance(Node reader, Node[] nodes, int activeLen)
public void sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes,
int activeLen)
{code}
Check the failed unit test {{TestNetworkTopology}} is related to this changes.
Will fix that later.
cc [~ayushtkn],[~elgoiri] any other comments?
was (Author: hexiaoqiao):
Thanks [~pifta] for your suggestions,
{quote}I would suggest two more things to do, we might deprecate the old sorter
methods, as we most likely won't need them on the long run, as their use
effectively overrides the new setting, and an update would be nice to the
APIDoc of these methods.{quote}
It make sense for me. I agree that the following methods should be removed.
BTW, this common interface is invoke by different classes. I would like to
update that later.
{code:java}
public void sortByDistance(Node reader, Node[] nodes, int activeLen)
public void sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes,
int activeLen)
{code}
cc [~ayushtkn],[~elgoiri] any other comments?
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch,
> HDFS-14882.009.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.010.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965313#comment-16965313 ] Xiaoqiao He commented on HDFS-14882: [^HDFS-14882.010.patch] try to fix bug about NetworkTopology.java#sortByDistanceUsingNetworkLocation invokes. Update APIDoc of new methods, And correct unit test TestNetworkTopology#testSortByDistance, a. uncertain nodes sorted result if distance to reader is equal. b. we should invoke #sortByDistanceUsingNetworkLocation to test if reader is not a datanode. [^HDFS-14882.010.patch] do not deprecate or remove the old sorter methods since there are many unit tests and subclass depend on that. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.010.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: (was: HDFS-14882.010.patch) > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14720) DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.
[
https://issues.apache.org/jira/browse/HDFS-14720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965347#comment-16965347
]
Xiaoqiao He commented on HDFS-14720:
[~surendrasingh] Thanks for your report. Would you like to share scenario about
this case? Do you open Erasure Coding feature?
> DataNode shouldn't report block as bad block if the block length is
> Long.MAX_VALUE.
> ---
>
> Key: HDFS-14720
> URL: https://issues.apache.org/jira/browse/HDFS-14720
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14720.001.patch
>
>
> {noformat}
> 2019-08-11 09:15:58,092 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Can't replicate block
> BP-725378529-10.0.0.8-1410027444173:blk_13276745777_1112363330268 because
> on-disk length 175085 is shorter than NameNode recorded length
> 9223372036854775807.{noformat}
> If the block length is Long.MAX_VALUE, means file belongs to this block is
> deleted from the namenode and DN got the command after deletion of file. In
> this case command should be ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14775) Add Timestamp for longest FSN write/read lock held log
[
https://issues.apache.org/jira/browse/HDFS-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967196#comment-16967196
]
Xiaoqiao He commented on HDFS-14775:
[^HDFS-14775.005.patch] LGTM, +1. Thanks [~zhangchen].
> Add Timestamp for longest FSN write/read lock held log
> --
>
> Key: HDFS-14775
> URL: https://issues.apache.org/jira/browse/HDFS-14775
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14775.001.patch, HDFS-14775.002.patch,
> HDFS-14775.003.patch, HDFS-14775.004.patch, HDFS-14775.005.patch
>
>
> HDFS-13946 improved the log for longest read/write lock held time, it's very
> useful improvement.
> In some condition, we need to locate the detailed call information(user, ip,
> path, etc.) for longest lock holder, but the default throttle interval(10s)
> is too long to find the corresponding audit log. I think we should add the
> timestamp for the {{longestWriteLockHeldStackTrace}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index
[
https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973224#comment-16973224
]
Xiaoqiao He commented on HDFS-14617:
Thanks [~Feng Yuan] for your pings. IIRC, it is exactly invoke #loadRootINode
only once to update root directory INode attributions. So we could avoid to
synchronize FSImageFormatPBINode#Loader. Consider that run only one time, and
no any more performance impact, so it is fine for me to keep synchronize.
> Improve fsimage load time by writing sub-sections to the fsimage index
> --
>
> Key: HDFS-14617
> URL: https://issues.apache.org/jira/browse/HDFS-14617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 2.10.0, 3.3.0
>
> Attachments: HDFS-14617.001.patch, ParallelLoading.svg,
> SerialLoading.svg, dirs-single.svg, flamegraph.parallel.svg,
> flamegraph.serial.svg, inodes.svg
>
>
> Loading an fsimage is basically a single threaded process. The current
> fsimage is written out in sections, eg iNode, iNode_Directory, Snapshots,
> Snapshot_Diff etc. Then at the end of the file, an index is written that
> contains the offset and length of each section. The image loader code uses
> this index to initialize an input stream to read and process each section. It
> is important that one section is fully loaded before another is started, as
> the next section depends on the results of the previous one.
> What I would like to propose is the following:
> 1. When writing the image, we can optionally output sub_sections to the
> index. That way, a given section would effectively be split into several
> sections, eg:
> {code:java}
>inode_section offset 10 length 1000
> inode_sub_section offset 10 length 500
> inode_sub_section offset 510 length 500
>
>inode_dir_section offset 1010 length 1000
> inode_dir_sub_section offset 1010 length 500
> inode_dir_sub_section offset 1010 length 500
> {code}
> Here you can see we still have the original section index, but then we also
> have sub-section entries that cover the entire section. Then a processor can
> either read the full section in serial, or read each sub-section in parallel.
> 2. In the Image Writer code, we should set a target number of sub-sections,
> and then based on the total inodes in memory, it will create that many
> sub-sections per major image section. I think the only sections worth doing
> this for are inode, inode_reference, inode_dir and snapshot_diff. All others
> tend to be fairly small in practice.
> 3. If there are under some threshold of inodes (eg 10M) then don't bother
> with the sub-sections as a serial load only takes a few seconds at that scale.
> 4. The image loading code can then have a switch to enable 'parallel loading'
> and a 'number of threads' where it uses the sub-sections, or if not enabled
> falls back to the existing logic to read the entire section in serial.
> Working with a large image of 316M inodes and 35GB on disk, I have a proof of
> concept of this change working, allowing just inode and inode_dir to be
> loaded in parallel, but I believe inode_reference and snapshot_diff can be
> make parallel with the same technique.
> Some benchmarks I have are as follows:
> {code:java}
> Threads 1 2 3 4
>
> inodes448 290 226 189
> inode_dir 326 211 170 161
> Total 927 651 535 488 (MD5 calculation about 100 seconds)
> {code}
> The above table shows the time in seconds to load the inode section and the
> inode_directory section, and then the total load time of the image.
> With 4 threads using the above technique, we are able to better than half the
> load time of the two sections. With the patch in HDFS-13694 it would take a
> further 100 seconds off the run time, going from 927 seconds to 388, which is
> a significant improvement. Adding more threads beyond 4 has diminishing
> returns as there are some synchronized points in the loading code to protect
> the in memory structures.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14952) Skip safemode if blockTotal is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-14952:
---
Attachment: HDFS-14952.002.patch
Assignee: Xiaoqiao He
Status: Patch Available (was: Open)
upload new patch and try to trigger Jenkins.
> Skip safemode if blockTotal is 0 in new NN
> --
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Assignee: Xiaoqiao He
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch, HDFS-14952.002.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14986) ReplicaCachingGetSpaceUsed throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973305#comment-16973305
]
Xiaoqiao He commented on HDFS-14986:
we meet very similar case (ConcurrentModificationException or DeadLock) when
backport HDFS-14313 to branch-2.7, and I believe there is still some space to
improve current implementation.
[~Aiphag0] may have some more information. Would you like to offer our patch
here?
> ReplicaCachingGetSpaceUsed throws ConcurrentModificationException
> --
>
> Key: HDFS-14986
> URL: https://issues.apache.org/jira/browse/HDFS-14986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, performance
>Reporter: Ryan Wu
>Assignee: Ryan Wu
>Priority: Major
>
> Running DU across lots of disks is very expensive . We applied the patch
> HDFS-14313 to get used space from ReplicaInfo in memory.However, new du
> threads throw the exception
> {code:java}
> // 2019-11-08 18:07:13,858 ERROR
> [refreshUsed-/home/vipshop/hard_disk/7/dfs/dn/current/BP-1203969992--1450855658517]
>
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed:
> ReplicaCachingGetSpaceUsed refresh
> errorjava.util.ConcurrentModificationException: Tree has been modified
> outside of iteratorat
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.checkForModification(FoldedTreeSet.java:311)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.hasNext(FoldedTreeSet.java:256)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:343)at
> java.util.HashSet.(HashSet.java:120)at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.deepCopyReplica(FsDatasetImpl.java:1052)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed.refresh(ReplicaCachingGetSpaceUsed.java:73)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:178)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14986) ReplicaCachingGetSpaceUsed throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973339#comment-16973339
]
Xiaoqiao He commented on HDFS-14986:
[~linyiqun], we did some modify when backport HDFS-14313 since there are very
diffs between branch-2.7 and trunk.
As [~Aiphag0] said above, if no synchronous control, it will throw CME, but
meet deadlock if add sync when restart DataNode. The root cause is
`BlockPoolSlice#loadDfsUsed() may return -1. And the
ReplicaCachingGetSpaceUsed#init() will blocks` when start as mentioned above.
> ReplicaCachingGetSpaceUsed throws ConcurrentModificationException
> --
>
> Key: HDFS-14986
> URL: https://issues.apache.org/jira/browse/HDFS-14986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, performance
>Reporter: Ryan Wu
>Assignee: Ryan Wu
>Priority: Major
>
> Running DU across lots of disks is very expensive . We applied the patch
> HDFS-14313 to get used space from ReplicaInfo in memory.However, new du
> threads throw the exception
> {code:java}
> // 2019-11-08 18:07:13,858 ERROR
> [refreshUsed-/home/vipshop/hard_disk/7/dfs/dn/current/BP-1203969992--1450855658517]
>
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed:
> ReplicaCachingGetSpaceUsed refresh error
> java.util.ConcurrentModificationException: Tree has been modified outside of
> iterator
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.checkForModification(FoldedTreeSet.java:311)
>
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.hasNext(FoldedTreeSet.java:256)
>
> at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
> at java.util.HashSet.(HashSet.java:120)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.deepCopyReplica(FsDatasetImpl.java:1052)
>
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed.refresh(ReplicaCachingGetSpaceUsed.java:73)
>
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:178)
>
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973364#comment-16973364 ] Xiaoqiao He commented on HDFS-14882: Ping [~elgoiri],[~ayushtkn] any furthermore comments? > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14952) Skip safemode if blockTotal is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974868#comment-16974868
]
Xiaoqiao He commented on HDFS-14952:
[^HDFS-14952.003.patch] try to fix failed unit test #TestHASafeMode when
blockTotal=0. Others seems unrelated.
> Skip safemode if blockTotal is 0 in new NN
> --
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Assignee: Xiaoqiao He
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch, HDFS-14952.002.patch,
> HDFS-14952.003.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14952) Skip safemode if blockTotal is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-14952:
---
Attachment: HDFS-14952.003.patch
> Skip safemode if blockTotal is 0 in new NN
> --
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Assignee: Xiaoqiao He
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch, HDFS-14952.002.patch,
> HDFS-14952.003.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969041#comment-16969041 ] Xiaoqiao He commented on HDFS-14882: Thanks [~elgoiri] for your feedback. Ping [~pifta], would like to give another check about [^HDFS-14882.010.patch]. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969039#comment-16969039
]
Xiaoqiao He commented on HDFS-14117:
Hi [~ramkumar],[~elgoiri],[~ayushtkn], any progress here?
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ramkumar Ramalingam
>Assignee: Ramkumar Ramalingam
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14529) NPE while Loading the Editlogs
[
https://issues.apache.org/jira/browse/HDFS-14529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970116#comment-16970116
]
Xiaoqiao He commented on HDFS-14529:
Thanks [~sodonnell] for your quick response. I have noticed HDFS-12369.
However, in my case, we do not use SNAPSHOT feature, and I also check file
mentioned above log and its all parent path, all of them are not snapshot path.
I have to state that the case I mentioned is appear with old version. I would
like to share some more information if any progress.
> NPE while Loading the Editlogs
> --
>
> Key: HDFS-14529
> URL: https://issues.apache.org/jira/browse/HDFS-14529
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Ayush Saxena
>Priority: Major
>
> {noformat}
> 2019-05-31 15:15:42,397 ERROR namenode.FSEditLogLoader: Encountered exception
> on operation TimesOp [length=0,
> path=/testLoadSpace/dir0/dir0/dir0/dir2/_file_9096763, mtime=-1,
> atime=1559294343288, opCode=OP_TIMES, txid=18927893]
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:490)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:711)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:286)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:181)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:924)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:771)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:331)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1105)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:726)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1558)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1725){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14529) NPE while Loading the Editlogs
[
https://issues.apache.org/jira/browse/HDFS-14529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970092#comment-16970092
]
Xiaoqiao He commented on HDFS-14529:
I also meet another NPE at StandbyNN (build with hadoop-2.7.1) to replay
editlog, it seems some corner case to trigger FSEditLogLoader throw null
pointer.
{code:java}
2019-11-06 18:30:25,948 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
on operation CloseOp [length=0, inodeId=0, path=$path, replication=3,
mtime=1573034707723, atime=1571949218729, blockSize=67108864,
blocks=[blk_2870262427_1841120265], permissions=*:*:rw-r--r--, aclEntries=null,
clientName=, clientMachine=, overwrite=false, storagePolicyId=0,
opCode=OP_CLOSE, txid=21246238494]
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.setGenerationStampAndVerifyReplicas(BlockInfoContiguousUnderConstruction.java:259)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.commitBlock(BlockInfoContiguousUnderConstruction.java:279)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.forceCompleteBlock(BlockManager.java:1199)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.updateBlocks(FSEditLogLoader.java:1022)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:438)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:844)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:825)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:232)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:331)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:284)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:426)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:297)
{code}
> NPE while Loading the Editlogs
> --
>
> Key: HDFS-14529
> URL: https://issues.apache.org/jira/browse/HDFS-14529
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Ayush Saxena
>Priority: Major
>
> {noformat}
> 2019-05-31 15:15:42,397 ERROR namenode.FSEditLogLoader: Encountered exception
> on operation TimesOp [length=0,
> path=/testLoadSpace/dir0/dir0/dir0/dir2/_file_9096763, mtime=-1,
> atime=1559294343288, opCode=OP_TIMES, txid=18927893]
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:490)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:711)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:286)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:181)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:924)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:771)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:331)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1105)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:726)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1558)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1725){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14720) DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.
[
https://issues.apache.org/jira/browse/HDFS-14720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970059#comment-16970059
]
Xiaoqiao He commented on HDFS-14720:
Thanks [~surendrasingh] and [~hemanthboyina],
{quote}It just not log the warn message , even it report badblock to namenode
and increase the work load for namenode.{quote}
It is true. One minor suggestion,
{code:java}
+ if (getBlock().getNumBytes() != BlockCommand.NO_ACK)
{code}
`BlockCommand.NO_ACK` is not very clear express here, it is better to add some
comments, perhaps nice with JIRA id. FYI. Thanks.
> DataNode shouldn't report block as bad block if the block length is
> Long.MAX_VALUE.
> ---
>
> Key: HDFS-14720
> URL: https://issues.apache.org/jira/browse/HDFS-14720
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14720.001.patch, HDFS-14720.002.patch
>
>
> {noformat}
> 2019-08-11 09:15:58,092 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Can't replicate block
> BP-725378529-10.0.0.8-1410027444173:blk_13276745777_1112363330268 because
> on-disk length 175085 is shorter than NameNode recorded length
> 9223372036854775807.{noformat}
> If the block length is Long.MAX_VALUE, means file belongs to this block is
> deleted from the namenode and DN got the command after deletion of file. In
> this case command should be ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14720) DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.
[
https://issues.apache.org/jira/browse/HDFS-14720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970332#comment-16970332
]
Xiaoqiao He commented on HDFS-14720:
[^HDFS-14720.003.patch] LGTM, +1. Thanks [~hemanthboyina].
> DataNode shouldn't report block as bad block if the block length is
> Long.MAX_VALUE.
> ---
>
> Key: HDFS-14720
> URL: https://issues.apache.org/jira/browse/HDFS-14720
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14720.001.patch, HDFS-14720.002.patch,
> HDFS-14720.003.patch
>
>
> {noformat}
> 2019-08-11 09:15:58,092 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Can't replicate block
> BP-725378529-10.0.0.8-1410027444173:blk_13276745777_1112363330268 because
> on-disk length 175085 is shorter than NameNode recorded length
> 9223372036854775807.{noformat}
> If the block length is Long.MAX_VALUE, means file belongs to this block is
> deleted from the namenode and DN got the command after deletion of file. In
> this case command should be ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15000) Improve FsDatasetImpl to avoid IO operation in datasetLock
Xiaoqiao He created HDFS-15000: -- Summary: Improve FsDatasetImpl to avoid IO operation in datasetLock Key: HDFS-15000 URL: https://issues.apache.org/jira/browse/HDFS-15000 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Xiaoqiao He Assignee: Aiphago As HDFS-14997 mentioned, some methods in #FsDatasetImpl such as #finalizeBlock, #finalizeReplica, #createRbw includes IO operation in the datasetLock, It will block some logic when IO load is very high. We should reduce grain fineness or move IO operation out of datasetLock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: HDFS-14997.002.patch > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979379#comment-16979379 ] Xiaoqiao He commented on HDFS-14997: [~sodonnell], Try to upload v002 patch and add unit test. Please help to take a review if have time. cc [~weichiu], [~elgoiri] and other watcher guys. > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15053) RBF: Add permission check for safemode operation
[ https://issues.apache.org/jira/browse/HDFS-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15053: --- Attachment: HDFS-15053.002.patch > RBF: Add permission check for safemode operation > > > Key: HDFS-15053 > URL: https://issues.apache.org/jira/browse/HDFS-15053 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15053.001.patch, HDFS-15053.002.patch > > > Propose to add superuser permission check for safemode operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15053) RBF: Add permission check for safemode operation
[
https://issues.apache.org/jira/browse/HDFS-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995438#comment-16995438
]
Xiaoqiao He commented on HDFS-15053:
Hi [~ayushtkn], Thanks for your suggestion, [^HDFS-15053.002.patch] try to
unify {{checkSuperuserPrivilege}} to single method and fix the unit test.
Please take another review if have time.
> RBF: Add permission check for safemode operation
>
>
> Key: HDFS-15053
> URL: https://issues.apache.org/jira/browse/HDFS-15053
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15053.001.patch, HDFS-15053.002.patch
>
>
> Propose to add superuser permission check for safemode operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[
https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993275#comment-16993275
]
Xiaoqiao He commented on HDFS-14997:
Thanks [~weichiu],
{quote}Question: I can imagine DNA_INVALIDATE taking a long time to process. Do
we already have metrics or log messages when the block invalidation command
taking too long?
{quote}
It looks no metrics or log to trace performance of IO operation (include some
very long lock waiting times) in trunk according to my observation. I add them
in my internal version. I also suggest to add them and next JIRA will work it
if necessary. Thanks.
> BPServiceActor process command from NameNode asynchronously
> ---
>
> Key: HDFS-14997
> URL: https://issues.apache.org/jira/browse/HDFS-14997
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch,
> HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch
>
>
> There are two core functions, report(#sendHeartbeat, #blockReport,
> #cacheReport) and #processCommand in #BPServiceActor main process flow. If
> processCommand cost long time it will block send report flow. Meanwhile
> processCommand could cost long time(over 1000s the worst case I meet) when IO
> load of DataNode is very high. Since some IO operations are under
> #datasetLock, So it has to wait to acquire #datasetLock long time when
> process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat
> will not send to NameNode in-time, and trigger other disasters.
> I propose to improve #processCommand asynchronously and not block
> #BPServiceActor to send heartbeat back to NameNode when meet high IO load.
> Notes:
> 1. Lifeline could be one effective solution, however some old branches are
> not support this feature.
> 2. IO operations under #datasetLock is another issue, I think we should solve
> it at another JIRA.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995751#comment-16995751
]
Xiaoqiao He commented on HDFS-15051:
Hi [~elgoiri], Thanks for your use case. [^HDFS-15051.002.patch] try to fix the
permission check as following rules with unit test to cover
#updateMountTableEntry method.
a. For #addMountTableEntry, try to check parent mount point write permission
then add mount point if check pass.
b. For #updateMountTableEntry, try to check the current mount point write
permission then update mount point if check pass.
c. For #removeMountTableEntry, I believe it is correct and without any changes.
BTW, I want to state that I prefer to grant the MountTable operation privilege
to superuser/admin only, even attach patch and try to fix this issue.If that,
it will be more simple and controllable in my opinion. Welcome more suggestions
and discussion. Thanks.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15051:
---
Attachment: HDFS-15051.002.patch
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15068) DataNode could meet deadlock if invoke refreshVolumes when register
Xiaoqiao He created HDFS-15068:
--
Summary: DataNode could meet deadlock if invoke refreshVolumes
when register
Key: HDFS-15068
URL: https://issues.apache.org/jira/browse/HDFS-15068
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Xiaoqiao He
Assignee: Aiphago
DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host
start` to trigger #refreshVolumes.
1. DataNod#refreshVolumes hold datanode instance ownable {{synchronizer}} when
enter this method first, then try to hold BPOfferService {{readlock}} when
`bpos.getNamespaceInfo()` in following code segment.
{code:java}
for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
nsInfos.add(bpos.getNamespaceInfo());
}
{code}
2. BPOfferService#registrationSucceeded (which is invoked by #register when
DataNode start or #reregister when processCommandFromActor) hold BPOfferService
{{writelock}} first, then try to hold datanode instance ownable
{{synchronizer}} in following method.
{code:java}
synchronized void bpRegistrationSucceeded(DatanodeRegistration bpRegistration,
String blockPoolId) throws IOException {
id = bpRegistration;
if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
throw new IOException("Inconsistent Datanode IDs. Name-node returned "
+ bpRegistration.getDatanodeUuid()
+ ". Expecting " + storage.getDatanodeUuid());
}
registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15068) DataNode could meet deadlock if invoke refreshVolumes when register
[
https://issues.apache.org/jira/browse/HDFS-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997936#comment-16997936
]
Xiaoqiao He commented on HDFS-15068:
Thanks [~Aiphag0] for your contribution. I agree that it is safe to move `get
namespace information` logic out from `synchronized`. It will be better to add
unit test to cover this corner case.
> DataNode could meet deadlock if invoke refreshVolumes when register
> ---
>
> Key: HDFS-15068
> URL: https://issues.apache.org/jira/browse/HDFS-15068
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Aiphago
>Priority: Critical
> Attachments: HDFS-15068.001.patch
>
>
> DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host
> start` to trigger #refreshVolumes.
> 1. DataNod#refreshVolumes hold datanode instance ownable {{synchronizer}}
> when enter this method first, then try to hold BPOfferService {{readlock}}
> when `bpos.getNamespaceInfo()` in following code segment.
> {code:java}
> for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
> nsInfos.add(bpos.getNamespaceInfo());
> }
> {code}
> 2. BPOfferService#registrationSucceeded (which is invoked by #register when
> DataNode start or #reregister when processCommandFromActor) hold
> BPOfferService {{writelock}} first, then try to hold datanode instance
> ownable {{synchronizer}} in following method.
> {code:java}
> synchronized void bpRegistrationSucceeded(DatanodeRegistration
> bpRegistration,
> String blockPoolId) throws IOException {
> id = bpRegistration;
> if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
> throw new IOException("Inconsistent Datanode IDs. Name-node returned "
> + bpRegistration.getDatanodeUuid()
> + ". Expecting " + storage.getDatanodeUuid());
> }
>
> registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15068) DataNode could meet deadlock if invoke refreshVolumes when register
[
https://issues.apache.org/jira/browse/HDFS-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15068:
---
Status: Patch Available (was: Open)
> DataNode could meet deadlock if invoke refreshVolumes when register
> ---
>
> Key: HDFS-15068
> URL: https://issues.apache.org/jira/browse/HDFS-15068
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Aiphago
>Priority: Critical
> Attachments: HDFS-15068.001.patch
>
>
> DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host
> start` to trigger #refreshVolumes.
> 1. DataNod#refreshVolumes hold datanode instance ownable {{synchronizer}}
> when enter this method first, then try to hold BPOfferService {{readlock}}
> when `bpos.getNamespaceInfo()` in following code segment.
> {code:java}
> for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
> nsInfos.add(bpos.getNamespaceInfo());
> }
> {code}
> 2. BPOfferService#registrationSucceeded (which is invoked by #register when
> DataNode start or #reregister when processCommandFromActor) hold
> BPOfferService {{writelock}} first, then try to hold datanode instance
> ownable {{synchronizer}} in following method.
> {code:java}
> synchronized void bpRegistrationSucceeded(DatanodeRegistration
> bpRegistration,
> String blockPoolId) throws IOException {
> id = bpRegistration;
> if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
> throw new IOException("Inconsistent Datanode IDs. Name-node returned "
> + bpRegistration.getDatanodeUuid()
> + ". Expecting " + storage.getDatanodeUuid());
> }
>
> registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15051) Propose to revoke WRITE MountTableEntry privilege to super user only
Xiaoqiao He created HDFS-15051:
--
Summary: Propose to revoke WRITE MountTableEntry privilege to
super user only
Key: HDFS-15051
URL: https://issues.apache.org/jira/browse/HDFS-15051
Project: Hadoop HDFS
Issue Type: Sub-task
Components: rbf
Reporter: Xiaoqiao He
Assignee: Xiaoqiao He
The current permission checker of #MountTableStoreImpl is not very restrict. In
some case, any user could add/update/remove MountTableEntry without the
expected permission checking.
The following code segment try to check permission when operate
MountTableEntry, however mountTable object is from Client/RouterAdmin
{{MountTable mountTable = request.getEntry();}}, and user could pass any mode
which could bypass the permission checker.
{code:java}
public void checkPermission(MountTable mountTable, FsAction access)
throws AccessControlException {
if (isSuperUser()) {
return;
}
FsPermission mode = mountTable.getMode();
if (getUser().equals(mountTable.getOwnerName())
&& mode.getUserAction().implies(access)) {
return;
}
if (isMemberOfGroup(mountTable.getGroupName())
&& mode.getGroupAction().implies(access)) {
return;
}
if (!getUser().equals(mountTable.getOwnerName())
&& !isMemberOfGroup(mountTable.getGroupName())
&& mode.getOtherAction().implies(access)) {
return;
}
throw new AccessControlException(
"Permission denied while accessing mount table "
+ mountTable.getSourcePath()
+ ": user " + getUser() + " does not have " + access.toString()
+ " permissions.");
}
{code}
I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15050) Optimize log information when DFSInputStream meet CannotObtainBlockLengthException
[
https://issues.apache.org/jira/browse/HDFS-15050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15050:
---
Attachment: HDFS-15050.001.patch
Status: Patch Available (was: Open)
submit init patch with minor exception changes.
> Optimize log information when DFSInputStream meet
> CannotObtainBlockLengthException
> --
>
> Key: HDFS-15050
> URL: https://issues.apache.org/jira/browse/HDFS-15050
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15050.001.patch
>
>
> We could not identify which file it belongs easily when DFSInputStream meet
> CannotObtainBlockLengthException, as the following exception log. Just
> suggest to log file path string when we meet CannotObtainBlockLengthException.
> {code:java}
> Caused by: java.io.IOException: Cannot obtain block length for
> LocatedBlock{BP-***:blk_***_***; getBlockSize()=690504; corrupt=false;
> offset=1811939328;
> locs=[DatanodeInfoWithStorage[*:50010,DS-2bcadcc4-458a-45c6-a91b-8461bf7cdd71,DISK],
>
> DatanodeInfoWithStorage[*:50010,DS-8f2bb259-ecb2-4839-8769-4a0523360d58,DISK],
>
> DatanodeInfoWithStorage[*:50010,DS-69f4de6f-2428-42ff-9486-98c2544b1ada,DISK]]}
> at
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:402)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:345)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:280)
> at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:272)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1664)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:266)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:481)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:828)
> at
> org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:109)
> at
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:65)
> ... 16 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15050) Optimize log information when DFSInputStream meet CannotObtainBlockLengthException
Xiaoqiao He created HDFS-15050:
--
Summary: Optimize log information when DFSInputStream meet
CannotObtainBlockLengthException
Key: HDFS-15050
URL: https://issues.apache.org/jira/browse/HDFS-15050
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Reporter: Xiaoqiao He
Assignee: Xiaoqiao He
We could not identify which file it belongs easily when DFSInputStream meet
CannotObtainBlockLengthException, as the following exception log. Just suggest
to log file path string when we meet CannotObtainBlockLengthException.
{code:java}
Caused by: java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-***:blk_***_***; getBlockSize()=690504; corrupt=false;
offset=1811939328;
locs=[DatanodeInfoWithStorage[*:50010,DS-2bcadcc4-458a-45c6-a91b-8461bf7cdd71,DISK],
DatanodeInfoWithStorage[*:50010,DS-8f2bb259-ecb2-4839-8769-4a0523360d58,DISK],
DatanodeInfoWithStorage[*:50010,DS-69f4de6f-2428-42ff-9486-98c2544b1ada,DISK]]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:402)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:345)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:280)
at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:272)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1664)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:266)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:481)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:828)
at
org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:109)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:65)
... 16 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994393#comment-16994393
]
Xiaoqiao He commented on HDFS-15051:
+ RouterStateManager which is without any access control.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15053) RBF: Add permission check for safemode operation
[ https://issues.apache.org/jira/browse/HDFS-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15053: --- Attachment: HDFS-15053.001.patch Status: Patch Available (was: Open) submit the init patch with permission check when operate safemode, only super user have this privilege. > RBF: Add permission check for safemode operation > > > Key: HDFS-15053 > URL: https://issues.apache.org/jira/browse/HDFS-15053 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15053.001.patch > > > Propose to add superuser permission check for safemode operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994693#comment-16994693
]
Xiaoqiao He commented on HDFS-15051:
It seems that no related with other module to add permission check and set only
super user have privilege to operate safemode (at interface
#RouterStateManager), So I file another separate JIRA(HDFS-15053) to work on
this.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15053) RBF: Add permission check for safemode operation
Xiaoqiao He created HDFS-15053: -- Summary: RBF: Add permission check for safemode operation Key: HDFS-15053 URL: https://issues.apache.org/jira/browse/HDFS-15053 Project: Hadoop HDFS Issue Type: Sub-task Components: rbf Reporter: Xiaoqiao He Assignee: Xiaoqiao He Propose to add superuser permission check for safemode operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994281#comment-16994281
]
Xiaoqiao He commented on HDFS-15051:
{quote}We can change to have just superuser access, if only the admin is
supposed to handle the mount points and we don't expect any user to be directly
operating on the Router Mount points. {quote}
I agree with that one hundred percent.
Another side, if we open the update mount point privilege to all end user,
there will be some other issues. Such as, end user change someone mount point
from one namespace to another, but do not transfer data meanwhile, we could not
keep the data consistency guarantee.
{quote}User isn't suppose to know there is a router, then why he should have
rights on Router Operations
{quote}
+1. I believe we should keep mount points transparently to end user. So it is
reasonable to revoke this privilege in my own opinion. Any more information or
consideration about the initial design will be better to determine what we
should step forward.
cc [~linyiqun] would you mind to offer some more suggestions?
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15051) Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15051:
---
Attachment: HDFS-15051.001.patch
Status: Patch Available (was: Open)
submit demo patch and revoke MountTableEntry operation privilege.
> Propose to revoke WRITE MountTableEntry privilege to super user only
>
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997311#comment-16997311
]
Xiaoqiao He commented on HDFS-15051:
if no object, [^HDFS-15051.003.patch] try to fix permission check logic based
on current implementation.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15051:
---
Attachment: HDFS-15051.003.patch
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15051:
---
Attachment: HDFS-15051.004.patch
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch, HDFS-15051.004.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997423#comment-16997423
]
Xiaoqiao He commented on HDFS-15051:
[^HDFS-15051.004.patch] fix checkstyle.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch, HDFS-15051.004.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994154#comment-16994154
]
Xiaoqiao He commented on HDFS-15051:
Thanks [~elgoiri] for offering the design consideration, However it seems that
goals is not met in current implementation, and there are still security
vulnerabilities, I mean MountTableEntry operation privilege. Consider the
following case, when one mount entry attribution as following, it seems that
any user such as `anonymous` could update it use command as `bin/hdfs
dfsrouteradmin -update /user nameservice /user2 -owner hdfs -group hadoop -mode
700` since permission checker of #RouterAdminServer only rely on the
information(user,group,mode) which supply from client. Please correct me if
something i missed.
{code:java}
/user nameservice /user hdfshadoop rwxr-xr-x
{code}
So, I propose we could enhance privilege control and avoid malicious updates,
even revoke update MountTableEntry privilege to super user only.
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961123#comment-16961123
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~pifta] for your suggestions, [^HDFS-14882.suggestion] is more graceful
changes in my opinion and we could reduce the double sort overhead. Just one
concerned, I am not sure if we have another overhead about
{{Collections.shuffle(list)}}. BTW, {{KeyInputStream.java}} seems a class of
ozone project, may be an unexpected change.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch,
> HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961096#comment-16961096
]
Xiaoqiao He commented on HDFS-13736:
[^HDFS-13736.006.patch] LGTM, check the failed unit tests, it is unrelated with
this changes I think.
It will be better if we keep the following code format same with community
style.
1. keep 4 spaces when start with newlines.
{code:java}
+ targets = chooseTarget(2, dataNodes[3], null,
+ favouredNodes);
+ assertEquals(targets.length, 2);
+ for (int i = 0; i < targets.length; i++) {
+assertTrue("Target should be a part of Expected Targets",
+expectedTargets.contains(targets[i].getDatanodeDescriptor()));
{code}
2. annotation start with upper case letter at most times.
{code:java}
* choose storage of local or favored node.
* @param localOrFavoredNode local or favored node
* @param isFavoredNode if target node is favored node
{code}
non-essentials but nice to have. Thanks [~xiaodong.hu].
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
> --
>
> Key: HDFS-13736
> URL: https://issues.apache.org/jira/browse/HDFS-13736
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Major
> Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch,
> HDFS-13736.003.patch, HDFS-13736.004.patch, HDFS-13736.005.patch,
> HDFS-13736.006.patch
>
>
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955815#comment-16955815 ] Xiaoqiao He commented on HDFS-14915: [~ayushtkn] Thanks for your works. it seems that double check operations and superuser check distributed on different method (such as #reencryptEncryptionZone + #reencryptEncryptionZoneInt), some of them are duplicated. Would you like to refactor it for once? +1 for other changes. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955804#comment-16955804
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~ayushtkn] for your comments.
{quote}Though things are quiet similar but not for same thing, I think we
should have a separate config.
{quote}
It makes sense to me. [^HDFS-14882.004.patch] add a separate config to govern
this changes. And also extend test case with decommissioned node. Please take
another reviews. Thanks.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955911#comment-16955911
]
Xiaoqiao He commented on HDFS-13736:
Thanks [~xiaodong.hu] for interesting catching. [^HDFS-13736.003.patch] almost
LGTM. Some checkstyle nit comment.
1. line length should be better less than 80 characters. such as
#BlockPlacementPolicyDefault L244, L505, L555.
2. Please following alignment code style, such as the following
segment(#BlockPlacementPolicyDefault L590-594) may be better. Please also check
TestReplicationPolicy#testChooseFromFavoredNodesWhenPreferLocalSetToFalse or
let's wait what Jenkins says.
{code:java}
protected DatanodeStorageInfo chooseLocalStorage(Node localMachine,
boolean isFavoredNode, Set excludedNodes, long blocksize,
int maxNodesPerRack, List results,
boolean avoidStaleNodes, EnumMap storageTypes,
boolean fallbackToLocalRack) throws NotEnoughReplicasException {
{code}
3. It is better to add annotation for
BlockPlacementPolicyDefault#chooseLocalStorage, especially for new parameter
#isFavoredNode in order to avoid it's readability getting worse and worse.
Thanks [~xiaodong.hu] for your contributions again.
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
> --
>
> Key: HDFS-13736
> URL: https://issues.apache.org/jira/browse/HDFS-13736
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Major
> Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch,
> HDFS-13736.003.patch
>
>
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.005.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955918#comment-16955918 ] Xiaoqiao He commented on HDFS-14882: [^HDFS-14882.005.patch] fix checkstyle and try to trigger Jenkins again. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.004.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955919#comment-16955919 ] Xiaoqiao He commented on HDFS-14915: +1 for [^HDFS-14915-02.patch] from my side. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955887#comment-16955887 ] Xiaoqiao He commented on HDFS-14915: Thanks [~ayushtkn] for your quick update. We need double #checkOperation both before and after hold lock. For #reencryptEncryptionZone, there are three times to do #checkOperation(WRITE) and distributed on three method. Some of them are duplicated. What I means is that we should delete some duplicated checking and it is better to embrace in one method (or file another JIRA to track). FYI. LGTM about superuser privilege checking improvement. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956085#comment-16956085 ] Xiaoqiao He edited comment on HDFS-14882 at 10/21/19 1:31 PM: -- Check failed unit tests, most of them failed due to OOM, Please help to double check. Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. was (Author: hexiaoqiao): Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956027#comment-16956027
]
Xiaoqiao He commented on HDFS-13736:
[^HDFS-13736.004.patch] looks much better otherwise the following unnecessary
print line.
{quote}+ System.err.println(Arrays.toString(targets));
{quote}
Let's wait what Jenkins report {Test Results, checkstyle}, please check if the
failed unit test (failed unit test #TestRenameWithSnapshots last reported
looks not related) is related with changes and if any other checkstyle issues.
Please fix checkstyle first. Thanks.
Ping [~ayushtkn], would like to take another reviews?
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
> --
>
> Key: HDFS-13736
> URL: https://issues.apache.org/jira/browse/HDFS-13736
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Major
> Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch,
> HDFS-13736.003.patch, HDFS-13736.004.patch
>
>
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.006.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956085#comment-16956085 ] Xiaoqiao He commented on HDFS-14882: Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14908) LeaseManager should check parent-child relationship when filter open files.
[ https://issues.apache.org/jira/browse/HDFS-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956922#comment-16956922 ] Xiaoqiao He commented on HDFS-14908: Thanks [~LiJinglun] for your catch and works, IIUC this issue only acts on DFSAdmin#listOpenFiles, right? another side, it may be not real result when we use static string(ignore some JVM overhead? not very sure please help to double check) to test, should we test it with some random string? Based on the benchmark result, it seems they are little different between #isParent and #startWith, almost no diff if we consider per-execution and very short time near nano-second. So my opinion is that keep this somewhat readable may be more important, such as #String.startsWith. Just my own opinion please correct me if something wrong. > LeaseManager should check parent-child relationship when filter open files. > --- > > Key: HDFS-14908 > URL: https://issues.apache.org/jira/browse/HDFS-14908 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.1.0, 3.0.1 >Reporter: Jinglun >Assignee: Jinglun >Priority: Minor > Attachments: HDFS-14908.001.patch, HDFS-14908.002.patch, > HDFS-14908.003.patch, Test.java, TestV2.java > > > Now when doing listOpenFiles(), LeaseManager only checks whether the filter > path is the prefix of the open files. We should check whether the filter path > is the parent/ancestor of the open files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14908) LeaseManager should check parent-child relationship when filter open files.
[ https://issues.apache.org/jira/browse/HDFS-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959520#comment-16959520 ] Xiaoqiao He commented on HDFS-14908: Thanks [~LiJinglun] for your works and strict benchmark. IMO, we should keep code simple and readable since performance cost diff is small and only used by DFSAdmin. I prefer to v1 patch. Would like to hear some more suggestions from [~elgoiri],[~weichiu]. Thanks [~LiJinglun]. > LeaseManager should check parent-child relationship when filter open files. > --- > > Key: HDFS-14908 > URL: https://issues.apache.org/jira/browse/HDFS-14908 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.1.0, 3.0.1 >Reporter: Jinglun >Assignee: Jinglun >Priority: Minor > Attachments: HDFS-14908.001.patch, HDFS-14908.002.patch, > HDFS-14908.003.patch, Test.java, TestV2.java, TestV3.java > > > Now when doing listOpenFiles(), LeaseManager only checks whether the filter > path is the prefix of the open files. We should check whether the filter path > is the parent/ancestor of the open files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.009.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961746#comment-16961746 ] Xiaoqiao He commented on HDFS-14882: [^HDFS-14882.009.patch] originate from [~pifta]'s suggestion and demo patch, and try to remove unrelated class and delete some useless import. [~ayushtkn],[~pifta],[~elgoiri] Please help to take a review. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, > HDFS-14882.009.patch, HDFS-14882.suggestion > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961747#comment-16961747 ] Xiaoqiao He commented on HDFS-13736: +1 for [^HDFS-13736.007.patch]. > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch, HDFS-13736.004.patch, HDFS-13736.005.patch, > HDFS-13736.006.patch, HDFS-13736.007.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955423#comment-16955423 ] Xiaoqiao He commented on HDFS-14882: [^HDFS-14882.002.patch] fix checkstyle and try to trigger Jenkins again. Hi watchers, anyone would like to help take another review? > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.002.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955426#comment-16955426
]
Xiaoqiao He commented on HDFS-14283:
Thanks [~leosun08] for your works.
{quote}But i have a problem that current block.getLocations() which gets a list
of DataNodes in priority order does not consider choosed DN LOAD, bandwidth
etc. I think it is necessary to add this logic later.{quote}
HDFS-14882 is working now, it is very pleasure if you are interested to review
it？
For this ticket, I am concerned about which one should be given priority
between distance and cached. Or leave the option to user?
Consider the following case, 3 replicas (names ra, rb, rc respectively) of one
block, and set cache replicas number to 2 which combine with rb and rc. then
another client which topology distance is more near to host which ra located at
(one corner case is that the client from the same host which ra located at)
rather than hosts rb/rc located. Then which one host should the client request
to firstly. I believe both ra or rb/rc is reasonable. [^HDFS-14283.003.patch]
seems to choose cache priority policy, right? I just suggest maybe it is better
to leave the choice to user.
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.003.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955487#comment-16955487
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~ayushtkn] for your reviews. I have addressed all the latest comments,
I think.
{quote}You even need to add the new config in Hdfs-defaults.xml
{quote}
I try to reused the config which has used by
BlockPlacementPolicyDefault#isGoodDatanode. IMO, they both follow the same
semantic, so I think we do not need to add another one.
I would like to give a brief introduction for this changes.
Generally, this patch try to re-order nodes who have same topology distance to
client and based on Load descend order.
Firstly, calculate all distances from nodes to client.
Then, try to use #start and #end to embrace nodes with the same distances.
Thirdly, re-sort by load descend order. (Note: Skip it when section length is
less than 2 since it is unnecessary to sort one node.)
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.007.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956667#comment-16956667
]
Xiaoqiao He commented on HDFS-14882:
[~elgoiri],[~ayushtkn] Thanks for your nice reviews and comments.
[^HDFS-14882.007.patch] could address the latest comments, Please have another
look.
{quote}We may also want to tune the description for
"dfs.namenode.redundancy.considerLoad" to be more precise.{quote}
+1, IMO we should change that in another JIRA, FYI. Thanks.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956673#comment-16956673 ] Xiaoqiao He commented on HDFS-13736: Thanks [~xiaodong.hu] for your works and contributions. [^HDFS-13736.005.patch] LGTM. There is another checkstyle issue Jenkins reported (refer to https://builds.apache.org/job/PreCommit-HDFS-Build/28148/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt). It seems to related with #ParameterNumber, I have no idea if it is tolerable issue. [~ayushtkn],[~weichiu] would you like to share some other suggestions? > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch, HDFS-13736.004.patch, HDFS-13736.005.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957592#comment-16957592
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~ayushtkn],[~elgoiri] for your suggestions, [^HDFS-14882.008.patch]
update descriptions of {{dfs.namenode.redundancy.considerLoad}} +
{{dfs.namenode.read.considerLoad}} and note explicitly that used for read or
write. Please help to give another reviews. Thanks again.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14908) LeaseManager should check parent-child relationship when filter open files.
[
https://issues.apache.org/jira/browse/HDFS-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957619#comment-16957619
]
Xiaoqiao He commented on HDFS-14908:
Thanks [~LiJinglun] for your feedback. What I mean that it could be more
simpler as following demo shows consider this is a minor bugfix and only act on
DFSAdmin#listOpenFiles, AND not more performance changes even if we add many
codes. Welcome to some more suggestions cc [~elgoiri].
{code:java}
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -1872,7 +1872,8 @@ private void metaSave(PrintWriter out) {
String fullPathName = inodeFile.getFullPathName();
if (org.apache.commons.lang3.StringUtils.isEmpty(path)
-|| fullPathName.startsWith(path)) {
+|| (fullPathName.startsWith(path) && (fullPathName.equals(path)
+|| fullPathName.charAt(path.length() - 1) ==
Path.SEPARATOR_CHAR))) {
openFileEntries.add(new OpenFileEntry(inodeFile.getId(),
inodeFile.getFullPathName(),
inodeFile.getFileUnderConstructionFeature().getClientName(),
{code}
> LeaseManager should check parent-child relationship when filter open files.
> ---
>
> Key: HDFS-14908
> URL: https://issues.apache.org/jira/browse/HDFS-14908
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Minor
> Attachments: HDFS-14908.001.patch, HDFS-14908.002.patch,
> HDFS-14908.003.patch, Test.java, TestV2.java
>
>
> Now when doing listOpenFiles(), LeaseManager only checks whether the filter
> path is the prefix of the open files. We should check whether the filter path
> is the parent/ancestor of the open files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.008.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: HDFS-14997.001.patch Assignee: Xiaoqiao He (was: Aiphago) Status: Patch Available (was: Open) submit demo patch and change to process commands asynchronously. > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: (was: HDFS-14997.001.patch) > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: HDFS-14997.001.patch > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
Xiaoqiao He created HDFS-14997: -- Summary: BPServiceActor process command from NameNode asynchronously Key: HDFS-14997 URL: https://issues.apache.org/jira/browse/HDFS-14997 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Xiaoqiao He Assignee: Aiphago There are two core functions, report(#sendHeartbeat, #blockReport, #cacheReport) and #processCommand in #BPServiceActor main process flow. If processCommand cost long time it will block send report flow. Meanwhile processCommand could cost long time(over 1000s the worst case I meet) when IO load of DataNode is very high. Since some IO operations are under #datasetLock, So it has to wait to acquire #datasetLock long time when process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat will not send to NameNode in-time, and trigger other disasters. I propose to improve #processCommand asynchronously and not block #BPServiceActor to send heartbeat back to NameNode when meet high IO load. Notes: 1. Lifeline could be one effective solution, however some old branches are not support this feature. 2. IO operations under #datasetLock is another issue, I think we should solve it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[
https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978517#comment-16978517
]
Xiaoqiao He commented on HDFS-14997:
Thanks [~sodonnell] for your comments.
{quote}We have seen a lot of occurrences when a client gets the error like
"failed to close file as the last block has insufficient number of
replicas"{quote}
suggest to tune up the parameter
`dfs.client.block.write.locateFollowingBlock.retries` (10 in my practice). This
case could be also related to high IO load in my opinion.
{quote}there are many places in the datanode where the FsDatasetImpl lock is
held for IO operations, and I suspect there are times we could potentially lock
on a volume rather than DN wide.
{quote}
What I noticed that three methods include FsDatasetImpl#{finalizeBlock,
finalizeReplica, createRbw} are very time consuming operation. Both of them
include IO operation in the global lock FsDatasetImpl#datasetLock. I believe we
can improve the implementation. My colleague [~Aiphag0] is working on this now,
he will file JIRA to trace it.
I am not sure if all 14 different type commands as following could be processed
async. For instance, if datanode process command not in-time, NameNode will
return DNA_REGISTER many times in some case. I think it is OK since
DNA_REGISTER is idempotent in both NameNode and DataNode sides. But I am not
sure if others are same idempotent and not have to keep the process order.
{DNA_UNKNOWN,DNA_TRANSFER,DNA_INVALIDATE,DNA_SHUTDOWN,DNA_REGISTER,DNA_FINALIZE,DNA_RECOVERBLOCK,DNA_ACCESSKEYUPDATE,DNA_BALANCERBANDWIDTHUPDATE,DNA_CACHE,DNA_UNCACHE,DNA_ERASURE_CODING_RECONSTRUCTION,DNA_BLOCK_STORAGE_MOVEMENT,DNA_DROP_SPS_WORK_COMMAND}
Any furthermore suggestions? Thanks again.
> BPServiceActor process command from NameNode asynchronously
> ---
>
> Key: HDFS-14997
> URL: https://issues.apache.org/jira/browse/HDFS-14997
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14997.001.patch
>
>
> There are two core functions, report(#sendHeartbeat, #blockReport,
> #cacheReport) and #processCommand in #BPServiceActor main process flow. If
> processCommand cost long time it will block send report flow. Meanwhile
> processCommand could cost long time(over 1000s the worst case I meet) when IO
> load of DataNode is very high. Since some IO operations are under
> #datasetLock, So it has to wait to acquire #datasetLock long time when
> process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat
> will not send to NameNode in-time, and trigger other disasters.
> I propose to improve #processCommand asynchronously and not block
> #BPServiceActor to send heartbeat back to NameNode when meet high IO load.
> Notes:
> 1. Lifeline could be one effective solution, however some old branches are
> not support this feature.
> 2. IO operations under #datasetLock is another issue, I think we should solve
> it at another JIRA.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: HDFS-14997.004.patch > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982686#comment-16982686 ] Xiaoqiao He commented on HDFS-14997: Thanks [~elgoiri] for your comments. v004 try to fix concerns mentioned above. Please take another reviews. Thanks. > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: HDFS-14997.005.patch > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983253#comment-16983253 ] Xiaoqiao He commented on HDFS-14997: [~elgoiri] v005 try to fix checkstyle and run all failed unit tests Jenkins reported above, It looks both of them are passed. Please help to take another reviews and double check. Thanks. > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor process command from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Attachment: (was: HDFS-14997.005.patch) > BPServiceActor process command from NameNode asynchronously > --- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
