[
https://issues.apache.org/jira/browse/HADOOP-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-894:
---------------------------------------
Attachment: partialBlockList3.patch
In this patch:
- open takes three parameters open(src, offset, length)
- there is an undocumented config parameter "dfs.read.prefetch.size" that
defines the range within which
we want all block locations to be fetch from the name-node during the open call.
- I kept 10*defaultBlockSize as the default, because 2 vs 10 does not improve
much communication or name-node
performance, but in most cases 10 will be ALL blocks for the majority of files.
- Implemented block location caching for reads and preads.
- Included more test cases in TestPread
> dfs client protocol should allow asking for parts of the block map
> ------------------------------------------------------------------
>
> Key: HADOOP-894
> URL: https://issues.apache.org/jira/browse/HADOOP-894
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Konstantin Shvachko
> Attachments: partialBlockList.patch, partialBlockList2.patch,
> partialBlockList3.patch
>
>
> I think that the HDFS client protocol should change like:
> /** The meta-data about a file that was opened. */
> class OpenFileInfo {
> /** the info for the first block */
> public LocatedBlockInfo getBlockInfo();
> public long getBlockSize();
> public long getLength();
> }
> interface ClientProtocol extends VersionedProtocol {
> public OpenFileInfo open(String name) throws IOException;
> /** get block info for any range of blocks */
> public LocatedBlockInfo[] getBlockInfo(String name, int blockOffset, int
> blockLength) throws IOException;
> }
> so that the client can decide how much block info to request and when.
> Currently, when the file is opened or an error occurs, the entire block list
> is requested and sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
