On 5/13/10 6:30 AM, Jason Wood wrote: >> This is #467 and #573. In my mind these tickets are super-important: > > That would be extremely useful, if there is anything I can do to help > with this in testing or working out use cases please let me know.
The most useful thing I can think of right now is to add your desired use case to the #467 ticket. Something like "I have three datacenters with 5 servers each and want to use 3-of-10 encoding and make sure that I can still retrieve my files even when two datacenters are offline". The second most useful thing would be for you to use Tahoe (probably in a testing environment) for a couple of weeks, get used to the tahoe.cfg format and the overall architecture, and then make some suggestions on #467 about how you'd like to be able to configure your desired use case. For example, "I'd like to have a section in tahoe.cfg where I describe the properties of each server, and then another section where I describe the rules of share placement", and some potential syntaxes for each. Or "I want the server's tahoe.cfg to describe its properties (like 'datacenter=XYZ'), and then the client's tahoe.cfg can describe the rules", or something like that. (I think that explicit server lists are fairly easy to implement, but not very flexible.. what should the client do when a new server is added that their tahoe.cfg doesn't mention? (should it use it anyways?). Or if one of the explicitly-listed servers is unavailable? (should it declare failure?). On the other hand, describing a rule like "prioritize uniform distribution of the 'server.datacenter' property" might be hard to express clearly in the tahoe.cfg language, so we might resort to a python plugin scheme of some sort). > Does this negate the advantage of having the storage nodes use RAID-5/6? > Would it make sense to just use RAID-0 and let Tahoe-LAFS deal with the > redundancy? The Allmydata grid didn't bother with RAID at all: each Tahoe storage server node used a single spindle. The "RAID and/or Tahoe" question depends upon how much you trust RAID vs how much you trust Tahoe, and how expensive the different forms of repair would be. Tahoe can correctly be thought of as a form of "application-level RAID", with more flexibility than the usual RAID0/4/5 styles (I think RAID-0 is equivalent to 1-of-2 encoding, and RAID-5 is like 2-of-3). Using RAID to achieve your redundancy gets you fairly fast repair, because it's all being handled by a controller that sits right on top of the raw drive. Tahoe's repair is a lot slower, because it is driven by a client that's examining one file at a time, and since there are a lot of network roundtrips for each file. Doing a repair of a 1TB RAID-5 drive can easily be finished in a day. If that 1TB drive is filled with a million Tahoe files, the repair could take a month. On the other hand, many RAID configurations degrade significantly when a drive is lost, and Tahoe's read performance is nearly unaffected. So repair events may be infrequent enough to just let them happen quietly in the background and not care much about how long they take. The optimal choice is a complicated one. Given inputs of: * how much data will be stored, how it changes over time (inlet rate, churn) * expected drive failure rate (both single sector errors and complete fail) * server/datacenter layout, inter/intra-colo bandwidth, costs * drive/hardware costs it becomes a tradeoff between money (number of tahoe storage nodes, what sort of RAID [if any] you use for them, how many disks that means, how much those disks cost, how many computers you need to host them, how much bandwidth you spend doing upload/download/repair), bandwidth costs, read/write performance, and probability of file loss due to failures happening faster than repair. In addition, Tahoe's current repair code is not particularly clever: it doesn't put the new shares in exactly the right places, so you can easily get shares doubled up and not distributed as evenly as if you'd done a single upload. This is being tracked in ticket #610. > More questions: > > Are links stored in the same way that files are? So if a storage node > containing a link to a file goes down, will that link exist on another > node? Directories (which are tables of name/filecap pairs) are stored as regular files, with the same encoding technique. So they are just as safe/vulnerable to storage server failures as regular files are. The usual way people use Tahoe is to remember a single "rootcap" that points at their top-level directory, and not keep track of anything else. In this mode, if you have a directory A which contains filecap B, then to get the file back, you need to be able to read both A and B. So the chances of getting the file back are reduced by the number of links you must follow to get to the data. But we use so much redundancy that even fairly deep directory structures are practically as reliable as single files. > Can there be more than one introducer node? The documentation seems to > suggest there can be only one but this would be a single point of > failure wouldn't it? Not yet (see ticket #68, which M O Faruque Sarker will be implementing this summer, again thanks to GSoC). Currently, the Introducer is indeed a SPOF, but a fairly mild one, because the introducer is only needed briefly when each node is started and joins the grid. After it connects to the Introducer and downloads the current list of storage servers, it doesn't need to talk to the Introducer again until a new server is added. So, if the Introducer goes offline for an hour, the only impact would be on clients or servers which join during that hour: all previously-connected nodes will keep on talking to each other directly. > Can there be more than one storage folder on a storage node? So if a > storage server contains 3 drives without RAID, can it use all 3 for > storage? Not directly. Each storage server has a single "base directory" which we abbreviate as $BASEDIR. The server keeps all of its shares in a subdirectory named $BASEDIR/storage/shares/ . (Note that you can symlink this to whatever you want: you can run most of the node from one place, and store all the shares somewhere else). Since there's only one such subdirectory, you can only use one filesystem per node. On the other hand, shares are stored in a set of 1024 subdirectories of that one, named $BASEDIR/storage/shares/aa/, $BASEDIR/storage/shares/ab/, etc. If you were to symlink the first third of these to one filesystem, the next third to a second filesystem, etc, (hopefully with a script!), then you'd get about 1/3rd of the shares stored on each disk. The "how much space is available" and space-reservation tools would be confused, but basically everything else should work normally. > And finally, are there any large companies relying on Tahoe-LAFS at > present? I'm trying to sell this to the powers that be and if I can drop > some names I would stand a much better chance ! Can't help you there :). Maybe some of the users are hanging out on this list and can jump in with their experiences. cheers, -Brian _______________________________________________ tahoe-dev mailing list tahoe-dev@allmydata.org http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev