Re: [Fedora-commons-users] Storing objects on a distributed filesystem or a local SAN

[Daniel Davis]

I am concerned about using NFS for Fedora's mount point.  I need to check with Chris Wilper because I seem to recall some problems when the network goes flaky (not to mention that NFS is a security mess unless it is behind a well controlled firewall).  Plus NFS is not really fast.  NFS is slowing dying off in most data centers except for backup or very fault tolerant cheap NAS storage. My tests were with high performance connections such as dedicated direct connection storage networks, iSCSI, and Fibrechannel.  Unless you use a connection that runs close to direct connection speeds you will really slow down disseminations (and ingest). The new Akubra project will likely take up how to store over less reliable networks. ALL the bitstreams and all the FOXML are stored as files.  This lets you rebuild if the DBMS or Triplestore fails.  The DBMS (and the Triplestore if you use one) is there to speed up operations or enable queries.  Only a very limited amount of data is stored in the DBMS (metadata kind of stuff extracted from the FOXML). It is perfectly OK to have the DBMS and FC run on the same server in all but the very largest installations.  Most installations run in your proposed configuration.  In a very large repository you may consider running the DBMS on its own server which is tuned just for that purpose. While you can store just the bitstreams on your SAN/Fabric, I would keep the objects there too.  If you have a well performing SAN/Fabric connection you can store everything on it except I would keep the operating system on a locally attached disk for bootstrap convenience and the swap partition. -- Dan Phil Cryer wrote: On Wed, 2008-08-20 at 14:57 -0400, Daniel Davis wrote: It should work fine. I have set the mount point within a Brocade Fabric partition and it worked perfectly. The one thing to remember is that only one Fedora instance should control the files. It is OK if the mount point is shared (though you may lose performance) but the specific storage directories must not be shared between two Fedora instances. I have had no problems using a well configured SAN/Fabric for DBMS or Triplestore persistence. It is generally better to dedicate the partition to a specific server and not share it when using it for Fedora, DBMS or Triplestore persistence. Thanks for the reply - so would I be able to run the mount point over a NFS connection to the SAN, or would it have to be hard connected? I was thinking of having the DBMS on the same server as FC, and then utilize the NFS/SAN mount as the storage for the digital objects in place of the way I currently do in FEDORA_HOME/data - is this right? Are the objects (jpgs, sid images in this case) stored within the RDMS or are they just on the filesystem in the data/objects directory, or am I missing the basics on how this functions? Thanks P I you dedicate the partition you still have a great failover mechanism. Should the server die you can fail over to a second server and let that server grab the SAN/Fabric partition. You can run Fedora without a rebuild if the failure of the first server was clean with respect to the file store, database, and triplestore. Otherwise you can run the rebuilder but still gain an advantage in that the content bitstreams and foxml files need not be moved. Finally, you can cluster the DBMS if you are using one with that capability and/or use Fedora Journaling if you want a more loosely coupled installation. -- Dan Davis Phil Cryer wrote: Is there any documentation about having Fedora-commons store objects to a distributed filesystem - HDFS or the like, or just a local SAN? I spoke with Sandy last week at RIRI about an S3 module, but I'm wondering if there's something to map to a local DFS/SAN for now. Would it be a case of just remapping FEDORA_HOME/data to mount the remote storage? Does FC even have to know about it, or is it system level at that point? Does it have something to do with: <param name="file_system" value="fedora.server.storage.lowlevel.GenericFileSystem"> Thanks P -- Daniel W. Davis Chief Software Architect, Fedora Commons Researcher, Cornell Information Science http://www.fedora-commons.org [EMAIL PROTECTED] [EMAIL PROTECTED] (607) 255-6090 (Office)

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users