On Sun, 22 Jan 2006, PS wrote:
Indexing is the key; observe how Google accesses millions of files in split
seconds; this could easily be achieved in a PC file system.
I think that you mean the right thing, but you're saying it in a very
confusing way.
1) Google doesn't access millions of files in a split second, it AFAIK
accesses relatively few files that are hashes (on its "index server")
that lead to URLs in a split second WITHOUT actually traversing millions
of alternatives (as you say, indexing is the key:-). File access
latency on a physical disk makes the former all but impossible without
highly specialized kernel hacks/hooks, ramdisks, caches, disk arrays,
and so on. Even bandwidth would be a limitation if one assumes block
I/O with a minimum block size of 4K -- 4K x 1M -> 4 Gigabytes/second
(note BYTES, not bits) exceeds the bandwidth of pretty much any physical
medium except maybe memory.
2) It cannot "easily" be achieved in a PC file system, if by that you
mean building an actual filesystem (at the kernel level) that supports
this sort of access. There is a lot more to a scalable, robust,
journalizeable filesystem than directory lookup capabilities. A lot of
Google's speed comes from being able to use substantial parallelism on a
distributed server environment with lots of data replication and
redundancy, a thing that is impossible for a PC filesystem with a number
of latency and bandwidth bottlenecks at different points in the dataflow
pathways towards what is typically a single physical disk on a single
e.g. PCI-whatever channel.
I think that what you mean (correctly) is that this is something that
"most" user/programmers would be better off trying to do in userspace on
top of any general purpose, known reliable/robust/efficient PC
filesystem, using hashes customized to the application. When I first
read your reply, though, I read it very differently as saying that it
would be easy to build a linux filesystem that actually permits millions
of files per second to be accessed and that this is what Google does
operationally.
rgb
Paul
Joe Landman wrote:
Methinks I lost lots of folks with my points ...
Major thesis is that on well designed hardware/software/filesystems, 50000
files is not a problem for accesses (though from a management point of view
it is a nightmare). For poorly designed/implemented file systems it is a
nightmare.
Way back when in the glory days of SGI, I seem to remember xfs being tested
with millions of files per directory (though don't hold me to that old
memory). Call this hearsay at this moment.
A well designed and implemented file system shouldn't bog you down as you
scale out in size, even if you shouldn't. Its sort of like your car. If
you go beyond 70 MPH somewhere in the US that supports such speeds, your
transmission shouldn't just drop out because you hit 71 MPH.
Graceful degradation is a good thing.
Joe
--
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:[EMAIL PROTECTED]
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
