Re: [tahoe-dev] configuring sqlite efficiency and durability

2012-12-03 Thread David-Sarah Hopwood
On 04/12/12 04:22, Brian wrote: > On 12/3/12 6:13 PM, Zooko Wilcox-O'Hearn wrote: > >> In fact, as David-Sarah pointed out today, since we update the leasedb >> and then write out the full share data before acking on file upload, >> the window of opportunity for this failure is probably zero on fi

Re: [tahoe-dev] configuring sqlite efficiency and durability

2012-12-03 Thread David-Sarah Hopwood
On 04/12/12 02:13, Zooko Wilcox-O'Hearn wrote: > Write-Ahead Logging: > > http://www.sqlite.org/wal.html > > the "synchronous" setting: > > http://www.sqlite.org/pragma.html#pragma_synchronous > > The results are interesting. All numbers here are the total wall-clock > time to run that unit tes

[tahoe-dev] Mutable directory update performance

2012-12-03 Thread Kyle Markley
Hi, I've been tinkering around with some code that would like to grow up to be a parallelized flavor of "tahoe backup" so I could do backup, and eventually deep-check, on trees with tens of thousands of files without waiting many hours for the standard tools to serially walk through every fil

Re: [tahoe-dev] configuring sqlite efficiency and durability

2012-12-03 Thread Brian
On 12/3/12 6:13 PM, Zooko Wilcox-O'Hearn wrote: In fact, as David-Sarah pointed out today, since we update the leasedb and then write out the full share data before acking on file upload, the window of opportunity for this failure is probably zero on file upload. Hm, we should probably analyze

Re: [tahoe-dev] configuring sqlite efficiency and durability

2012-12-03 Thread Greg Troxel
I know you don't want to use postgresql for real, probably, but it would be interesting to have a choice of databases to see what the effects are. So in short, I'm in favor of using synchronous=NORMAL, journal_mode=WAL for all current uses of sqlite in Tahoe-LAFS. Let me know if you see a f

Re: [tahoe-dev] Public web interface

2012-12-03 Thread Greg Troxel
Nathan writes: > Just by counting arrows, it's obvious that 4, the built-in web server > would be the "leanest" approach in terms of fewest "hops", so this > might be most efficient. I think this is the key point, but I don't think hop count is the right metric of lean. The real issue is total

[tahoe-dev] configuring sqlite efficiency and durability

2012-12-03 Thread Zooko Wilcox-O'Hearn
Folks: One of the unit tests (allmydata.test.test_cli.Cp.test_copy_using_filecap) takes more than 1000 seconds to complete on the leasedb branch, which uses sqlite for tracking leases instead of putting leases as metadata appended to the ends of share container files. The same test takes about 13

Re: [tahoe-dev] Public web interface

2012-12-03 Thread Greg Troxel
I see the whole approach to "dropbox like functionality" and "web server" as basically broken, and coming from a world view where tahoe-lafs is the primary thing, rather than viewing it as one of many filesystems (albeit with different, interesting and useful properties than many). So I think th

Re: [tahoe-dev] Public web interface

2012-12-03 Thread Uncle Zzzen
> IMO, architecture 2, a fuse proxy, is less attractive than 1, the http proxy. I agree that the "fuse proxy" option is extremely unattractive and can be removed from the list. The "static file server" option is classic for simple use-cases (e.g. single publisher), but once we have dropbox-like fu

Re: [tahoe-dev] Public web interface

2012-12-03 Thread Nathan
On Mon, Dec 3, 2012 at 3:35 PM, Uncle Zzzen wrote: > Hi. > I'm busy with work lately, but there's a discussion Zooko and I were having > on a closed ticket, and I agree with him it actually belongs here, so here > goes: > > Sometimes there's a need to expose a [partial view of a] Tahoe-LAFS storag

[tahoe-dev] Public web interface

2012-12-03 Thread Uncle Zzzen
Hi. I'm busy with work lately, but there's a discussion Zooko and I were having on a closed ticket, and I agree with him it actually belongs here, so here goes: Sometimes there's a need to expose a [partial view of a] Tahoe-LAFS storage as a public web service. As far as I understand, there are 3