[
https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425805#comment-17425805
]
Bryan Beaudreault commented on HDFS-16261:
------------------------------------------
I'm looking at this now. I don't have much experience in this area, but am
looking into two possibilities high level: handling this on the namenode or
handling in the datanode.
h2. Handling in the NameNode
When a DataNode receives a block, it notifies the namenode via
notifyNamenodeReceivedBlock. This sends a RECEIVED_BLOCK to the namenode along
with a "delHint", which tells the namenode to invalidate that block on the old
host.
Tracing that delHint through on the namenode side through a bunch of layers,
you eventually land in BlockManager.processExtraRedundancyBlock which
eventually lands in processChosenExcessRedundancy.
processChosenExcessRedundancy adds the block to a excessRedundancyMap and to a
nodeToBlocks map in InvalidateBlocks.
There is a RedundancyChore which periodically checks InvalidateBlocks, pulling
a configurable amount of blocks and adding them to the DatanodeDescriptor's
invalidateBlocks map. One quick option might be to configure the
RedundancyChore with dfs.namenode.redundancy.interval.seconds, though that's
not exactly what we want which is a per-block grace period.
Next time a DataNode sends a heartbeat, the Namenode processes various state
for that datanode and sends back a series of commands. Here the NameNode pulls
a configurable number of blocks from the DatanodeDescriptor's invalidateBlocks
and sends them to the DataNode as part of a DNA_INVALIDATE command.
If we were to handle this in the NameNode, we could potentially hook in a
couple places:
* When adding to nodeToBlocks, we could include a timestamp. The
RedundancyChore could only add blocks to the Descriptor's invalidateBlocks map
if older than a threshold.
* When adding to Descriptor's invalidateBlocks, we could add a timestamp. When
processing heartbeats, we could only send blocks via DNA_INVALIDATE which have
been in invalidateBlocks for more than a threshold
* As mentioned above, we could try tuning
dfs.namenode.redundancy.interval.seconds, though that isn't perfect because a
block could be added right before the chore runs and thus get immediately
invalidated.
h2. Handling in the DataNode
When a DataNode gets a request for a block, it looks that up in its
FsDatasetImpl volumeMap. If the block does not exist, a
ReplicaNotFoundException is thrown.
The DataNode receives the list of blocks to invalidate from the DNA_INVALIDATE
command, which is processed by BPOfferService. This is immediately handed off
to FsDatasetImpl.invalidate, which validates the request and immediately
removes the block from volumeMap. At this point, the data still exists on disk
but requests for the block would throw a ReplicaNotFoundException per above.
Once removed from volumeMap, the deletion of data is handled by the
FsDatasetAsyncDiskService. The processing is done async, but is immediately
handed off to a ThreadPoolExecutor which should execute fairly quickly.
A couple options:
* Defer the call to FsDatasetImpl.invalidate, at the highest level. This could
be passed off to a thread pool to be executed after a delay. In this case, the
block would remain in the volumeMap until the task is executed.
* Execute invalidate immediately, but defer the data deletion. We're already
using a thread pool here, so it might be easier to execute after a delay. It's
worth noting that there are other actions taken around the volumeMap removal.
We'd need to verify whether those need to be synchronized with removal from
volumeMap. In this case we'd need to either:
** relocate the volumeMap.remove call to within the FsDatasetAsyncDiskService.
This seems like somewhat of a leaky abstraction.
** Add a pendingDeletion map and add to that when removing from volumeMap. The
FsDatasetAsyncDiskService would remove from pendingDeletion once completed.
We'd need to update our block fetch code to check volumeMap _or_
pendingDeletion. This separation might give us opportunities in the future,
such as including a flag in the response that instructs the DFSClient "this
block may go away soon".
----
I'm doing more investigation and specifically want to look into what would
happen if the handling service died before invalidating blocks. I'm assuming
this is already handled since this process is very async already, but it will
be good to know. I also want to do a bit more thinking of the pros and cons of
each option above, and some experimenting with the easiest option of tuning the
redundancy chore. I'll report back when I have some more information, and also
open to other opinions or suggestions.
> Configurable grace period around deletion of invalidated blocks
> ---------------------------------------------------------------
>
> Key: HDFS-16261
> URL: https://issues.apache.org/jira/browse/HDFS-16261
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Bryan Beaudreault
> Assignee: Bryan Beaudreault
> Priority: Major
>
> When a block is moved with REPLACE_BLOCK, the new location is recorded in the
> NameNode and the NameNode instructs the old host to in invalidate the block
> using DNA_INVALIDATE. As it stands today, this invalidation is async but
> tends to happen relatively quickly.
> I'm working on a feature for HBase which enables efficient healing of
> locality through Balancer-style low level block moves (HBASE-26250). One
> issue is that HBase tends to keep open long running DFSInputStreams and
> moving blocks from under them causes lots of warns in the RegionServer and
> increases long tail latencies due to the necessary retries in the DFSClient.
> One way I'd like to fix this is to provide a configurable grace period on
> async invalidations. This would give the DFSClient enough time to refresh
> block locations before hitting any errors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
