Well I have set of interesting resources related to distributed fs:
http://www.coda.cs.cmu.edu/doc/html/index.html http://pl.atyp.us/pm/comments.php?id=552_0_1_0_C16
WebStore: Efficient Storage and Access of the Web
Daniel Golovin, Sandeep Pandey Computer Science Department Carnegie Mellon University Pittsburgh, PA 15213 {dgolovin, [EMAIL PROTECTED]
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
Doug already post it: http://www.lustre.org/whatislustre.html
Sorrento: A Self-Organizing Storage Cluster for Parallel Data-Intensive Applications Hong Tang, Aziz Gulbeden, Jingyu Zhou, Lingkun Chu, and Tao Yang Department of Computer Science University of California, Santa Barbara, CA 93106 {htang, gulbeden, jzhou, lkchu, [EMAIL PROTECTED]
I found some more but that wasn't that interesting from a first look over.
HTH Stefan
Am 15.06.2004 um 00:14 schrieb Abdul Chaudhry:
From what I have noticed , implementing your own buffer manager, disk cache or using NIO or whatever you want to call it, does improve performance in "some" situations. It avoids using the Operating System Disk Cache which has some overheads. In the database world it can sometimes yield higher than 30% performance gains - assuming your index'es are much larger than available ram. This would make sense for a smaller implementation of nutch , say, for intranet/enterprise search rather than internet search.
Anyway, someone posted that a distributed filesystem would be better and I guess that's the reason google uses its own distributed filesystem. Here is a paper that I'm sure someone else posted here at some point. http://www.cs.rochester.edu/sosp2003/papers/p125-ghemawat.pdf
-- Ab's
--- Byron Miller <[EMAIL PROTECTED]> wrote:-------------------------------------------------------Has anyone done any testing of converting read/writes & io to NIO or do you think that wouldn't offer much gain?
https://lists.sourceforge.net/lists/listinfo/nutch-developersThis SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED]
-------------------------------------------------------
This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference
Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer
Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA
REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
--------------------------------------------------------------- enterprise information technology consulting open technology: http://www.media-style.com open source: http://www.weta-group.net open discussion: http://www.text-mining.org
------------------------------------------------------- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
