[ https://issues.apache.org/jira/browse/HDFS-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056712#comment-16056712 ]
Ewan Higgs commented on HDFS-11125: ----------------------------------- The Provided Storage system is looking to use the SPS for asynchronously writing files to a remote system ({{hdfs://}}, {{s3a://}}, {{wasb://}}). The way the SPS currently works is file by file which suits the Provided Storage system very well since it works on a file by file basis as well. Moving to use smaller batches unlinked to the actual files would break current plans discussed offline with [~umamaheswararao]. This could be worked through by offering the interface as {{BlockStorageMovementCommandSchedulerSpi}} (or something less of a mouthful) and implementing it one way for normal SPS and another way for Provided Storage. > [SPS]: Use smaller batches of BlockMovingInfo into the block storage movement > command > ------------------------------------------------------------------------------------- > > Key: HDFS-11125 > URL: https://issues.apache.org/jira/browse/HDFS-11125 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode > Reporter: Rakesh R > Assignee: Rakesh R > > This is a follow-up task of HDFS-11068, where it sends all the blocks under a > trackID over single heartbeat response(DNA_BLOCK_STORAGE_MOVEMENT command). > If blocks are many under a given trackID(For example: a file contains many > blocks) then those requests go across a network and come with a lot of > overhead. In this jira, we will discuss and implement a mechanism to limit > the list of items into smaller batches with in trackID. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org