Yes, you can solve the bottleneck by starting a webdav server on each
client. But this would include the burden to manage the servers etc. and
it may not be the intended use case for webdav. But we can further
discuss the architecture in the relevant issue.
Alban Chevignard wrote:
Thanks for the clarification. I agree that running a single WebDAV
server for all clients would make it a bottleneck. But I can't see
anything in the current WebDAV server implementation that precludes
running an instance of it on each client. It seems to me that would
solve any bottleneck issue.
-Alban
On Jan 23, 2008 2:53 AM, Enis Soztutar <[EMAIL PROTECTED]> wrote:
As you know, dfs client connects to the individual datanodes to
read/write data and has a minimal interaction with the Namenode, which
improves the io rate linearly(theoretically 1:1). However current
implementation of webdav interface, is just a server working on a single
machine, which translates the webdav requests to namenode. Thus the
whole traffic passes through this webdav server, which makes it a
bottleneck. I was planning to integrate webdav server with
namenode/datanode, and forward the requests to the other datanodes, so
that we can do io in parallel, but my focus on webdav has faded for now.
Alban Chevignard wrote:
What are the scalability issues associated with the current WebDAV interface?
Thanks,
-Alban
On Jan 22, 2008 7:27 AM, Enis Soztutar <[EMAIL PROTECTED]> wrote:
Webdav interface for hadoop works as it is, but it needs a major
redesign to be scalable, however it is still useful. It has even been
used with windows explorer defining the webdav server as a remote service.
Ted Dunning wrote:
There has been significant work on building a web-DAV interface for HDFS. I
haven't heard any news for some time, however.
On 1/21/08 11:32 AM, "Dawid Weiss" <[EMAIL PROTECTED]> wrote:
The Eclipse plug-in also features a DFS browser.
Yep. That's all true, I don't mean to self-promote, because there really isn't
that much to advertise ;) I was just quite attached to file manager-like user
interface; the mucommander clone I posted served me as a browser, but also for
rudimentary file operations (copying to/from, deleting folders etc.). In my
experience it's been quite handy.
It would be probably a good idea to implement a commons-vfs plugin for Hadoop
so
that HDFS filesystem is transparent to use for other apps.
Dawid
