Re: question about how write_header works

2010-09-23 Thread Robert Dionne
On Sep 23, 2010, at 12:25 PM, Robert Newson wrote: > The idea also doesn't account for the waste in obsolete b+tree nodes. > Basically, it's more complicated than that. > > Compaction is unavoidable with an append-only strategy. One idea I've > pitched (and frankly stolen from Berkeley JE) is

Re: question about how write_header works

2010-09-23 Thread Robert Newson
The idea also doesn't account for the waste in obsolete b+tree nodes. Basically, it's more complicated than that. Compaction is unavoidable with an append-only strategy. One idea I've pitched (and frankly stolen from Berkeley JE) is for the database file to be a series of files instead of a single

Re: question about how write_header works

2010-09-23 Thread Paul Davis
On Thu, Sep 23, 2010 at 12:00 PM, chongqing xiao wrote: > Hi, Paul: > > Thanks for the clarification. > > I am not sure why this is designed this way but here is one approach I > think might work better > > Instead of appending the header to the data file, why not just moving > the header to a dif

Re: question about how write_header works

2010-09-23 Thread chongqing xiao
Hi, Paul: Thanks for the clarification. I am not sure why this is designed this way but here is one approach I think might work better Instead of appending the header to the data file, why not just moving the header to a different file. The header file can be implmented as before - 2 duplicate h

Re: question about how write_header works

2010-09-23 Thread Paul Davis
On Thu, Sep 23, 2010 at 11:37 AM, Randall Leeds wrote: > On Thu, Sep 23, 2010 at 15:44, Paul Davis wrote: >> As to the header2 or header1 problem, if header2 appears to be >> corrupted or is otherwise discarded, the header search just continues >> through the file looking for the next valid heade

Re: question about how write_header works

2010-09-23 Thread Randall Leeds
On Thu, Sep 23, 2010 at 15:44, Paul Davis wrote: > As to the header2 or header1 problem, if header2 appears to be > corrupted or is otherwise discarded, the header search just continues > through the file looking for the next valid header. In this case that > would mean that newData2 would not be

Re: question about how write_header works

2010-09-23 Thread Paul Davis
Its not appended each time data is written necessarily. There are optimizations to batch as many writes to the database together as possible as well as delayed commits which will write the header out every N seconds. Remember that *any* write to the database is going to look like wasted space. Eve

Re: question about how write_header works

2010-09-22 Thread chongqing xiao
Hi, Adam: Thanks for the answer. If that is how it works, that seems create a lot of wasted space assuming a new header has to be appended each time new data is saved. Also, assuming here is the data layout newData1 ->start header1 newData2 header2 -> end If header 2 is partially writte

Re: question about how write_header works

2010-09-22 Thread Adam Kocoloski
Hi Chong, that's exactly right. Regards, Adam On Sep 22, 2010, at 10:18 PM, chongqing xiao wrote: > Hi, > > Could anyone explain how write_header (or header) in works in couchdb? > > When appending new header, I am assuming the new header will be > appended to the end of the DB file and the o

question about how write_header works

2010-09-22 Thread chongqing xiao
Hi, Could anyone explain how write_header (or header) in works in couchdb? When appending new header, I am assuming the new header will be appended to the end of the DB file and the old header will be kept around? If that is the case, what will happen if the header is partially written? I am ass