In the current code, details about block locations of a file are
cached on the client when the file is opened. This cache remains with
the client until the file is closed. If the same file is re-opened by
the same DFSClient, it re-contacts the namenode and refetches the
block locations. This works ok for most map-reduce apps because it is
rare that the same DSClient re-opens the same file again.
Can you pl explain your use-case?
thanks,
dhruba
On Sun, Nov 2, 2008 at 10:57 PM, Taeho Kang <[EMAIL PROTECTED]> wrote:
> Dear Hadoop Users and Developers,
>
> I was wondering if there's a plan to add "file info cache" in DFSClient?
>
> It could eliminate network travelling cost for contacting Namenode and I
> think it would greatly improve the DFSClient's performance.
> The code I was looking at was this
>
> -----------------------
> DFSClient.java
>
> /**
> * Grab the open-file info from namenode
> */
> synchronized void openInfo() throws IOException {
> /* Maybe, we could add a file info cache here! */
> LocatedBlocks newInfo = callGetBlockLocations(src, 0, prefetchSize);
> if (newInfo == null) {
> throw new IOException("Cannot open filename " + src);
> }
> if (locatedBlocks != null) {
> Iterator<LocatedBlock> oldIter =
> locatedBlocks.getLocatedBlocks().iterator();
> Iterator<LocatedBlock> newIter =
> newInfo.getLocatedBlocks().iterator();
> while (oldIter.hasNext() && newIter.hasNext()) {
> if (! oldIter.next().getBlock().equals(newIter.next().getBlock()))
> {
> throw new IOException("Blocklist for " + src + " has changed!");
> }
> }
> }
> this.locatedBlocks = newInfo;
> this.currentNode = null;
> }
> -----------------------
>
> Does anybody have an opinion on this matter?
>
> Thank you in advance,
>
> Taeho
>
