Re: Replication vs. Compaction

2014-02-18 Thread Paul Davis
On Tue, Feb 18, 2014 at 2:28 PM, Jan Lehnardt wrote: > > On 18 Feb 2014, at 17:09 , Paul Davis wrote: > >> On Tue, Feb 18, 2014 at 9:30 AM, Adam Kocoloski wrote: >>> On Feb 18, 2014, at 11:38 AM, Jan Lehnardt wrote: >>> On 31 Jan 2014, at 20:08 , Jason Smith wrote: > However ther

Re: Replication vs. Compaction

2014-02-18 Thread Jan Lehnardt
On 18 Feb 2014, at 17:09 , Paul Davis wrote: > On Tue, Feb 18, 2014 at 9:30 AM, Adam Kocoloski wrote: >> On Feb 18, 2014, at 11:38 AM, Jan Lehnardt wrote: >> >>> On 31 Jan 2014, at 20:08 , Jason Smith wrote: >>> However there is a pathological situation where you are updating fast

Re: Replication vs. Compaction

2014-02-18 Thread Paul Davis
On Tue, Feb 18, 2014 at 9:30 AM, Adam Kocoloski wrote: > On Feb 18, 2014, at 11:38 AM, Jan Lehnardt wrote: > >> On 31 Jan 2014, at 20:08 , Jason Smith wrote: >> >>> However there is a pathological situation where you are >>> updating faster than the compactor can run, and you will get an infinit

Re: Replication vs. Compaction

2014-02-18 Thread Adam Kocoloski
On Feb 18, 2014, at 11:38 AM, Jan Lehnardt wrote: > On 31 Jan 2014, at 20:08 , Jason Smith wrote: > >> However there is a pathological situation where you are >> updating faster than the compactor can run, and you will get an infinite >> loop (plus very heavy i/o and filesystem waste as the com

Re: Replication vs. Compaction

2014-02-18 Thread Jan Lehnardt
On 31 Jan 2014, at 20:08 , Jason Smith wrote: > Anyway I think the broader point is, compaction is for compacting databases > (removing old document revisions), and replication is for making a copy of > a database (or subset). If compaction is causing downtime then that is a > different bug to t

Re: Replication vs. Compaction

2014-02-17 Thread Paul Davis
Typo'ed: "so it's unbalancing as much" should read: "so it's *not* unbalancing as much" On Mon, Feb 17, 2014 at 12:49 PM, Robert Samuel Newson wrote: > Replication will not rebalance the tree, no. It's just adding to the end (and > unbalancing the tree). > > The updates are happening in batch

Re: Replication vs. Compaction

2014-02-17 Thread Robert Samuel Newson
Replication will not rebalance the tree, no. It’s just adding to the end (and unbalancing the tree). The updates are happening in batches, though, so it’s unbalancing as much as the original individual updates did. B. On 17 Feb 2014, at 16:16, Boaz Citrin wrote: > Did some more testing. seem

Re: Replication vs. Compaction

2014-02-17 Thread Boaz Citrin
Did some more testing. seems like indeed compaction is faster than replication. One thing I observe though is that replication doesn't result the same as compaction; While it only copies the leaves, I suspect it doesn't produce a balanced tree, so subsequent compaction is needed anyway (and indeed

Re: Replication vs. Compaction

2014-02-03 Thread Adam Kocoloski
On Jan 31, 2014, at 3:43 PM, Jens Alfke wrote: > On Jan 31, 2014, at 12:07 PM, Boaz Citrin wrote: > >> But if replication only copies the leaf then it makes sense that it is >> fatser, at least on the same machine. Instead of balancing a tree it just >> copies a single revision. > > Um, no. Th

Re: Replication vs. Compaction

2014-01-31 Thread Jens Alfke
On Jan 31, 2014, at 12:07 PM, Boaz Citrin wrote: > But if replication only copies the leaf then it makes sense that it is > fatser, at least on the same machine. Instead of balancing a tree it just > copies a single revision. Um, no. The copied revision has to be inserted into the tree on the t

Re: Replication vs. Compaction

2014-01-31 Thread Jason Smith
Boaz, I encourage you to benchmark both techniques. I think you will discover that compaction is faster. However if not, then you will at least have peace of mind that your approach is the best. On Sat, Feb 1, 2014 at 3:07 AM, Boaz Citrin wrote: > But if replication only copies the leaf then it

Re: Replication vs. Compaction

2014-01-31 Thread Jason Smith
On Sat, Feb 1, 2014 at 2:27 AM, Boaz Citrin wrote: > @Jason - the purpose is that compaction will not catch up with heavy > write rate and will affect performance. > If compaction will not catch up with your writes, then you do not have sufficient hardware resources. You are operating in a very

Re: Replication vs. Compaction

2014-01-31 Thread Boaz Citrin
But if replication only copies the leaf then it makes sense that it is fatser, at least on the same machine. Instead of balancing a tree it just copies a single revision. Also I undestand from Jason that compaction tries to catch up, unlike a non continuos replication. So it seems that if I don't n

Re: Replication vs. Compaction

2014-01-31 Thread Jens Alfke
On Jan 31, 2014, at 11:27 AM, Boaz Citrin wrote: > @Jason - the purpose is that compaction will not catch up with heavy > write rate and will affect performance. The same will be true of replication. > I wonder if I can omit the compaction if replication gives the same result. > Will replicati

Re: Replication vs. Compaction

2014-01-31 Thread Boaz Citrin
@Jason - the purpose is that compaction will not catch up with heavy write rate and will affect performance. So I replicate/copy to a side database, compact it, then replicate the new changes from the original and switchover. I wonder if I can omit the compaction if replication gives the same resul

Re: Replication vs. Compaction

2014-01-31 Thread Jason Smith
Anyway I think the broader point is, compaction is for compacting databases (removing old document revisions), and replication is for making a copy of a database (or subset). If compaction is causing downtime then that is a different bug to talk about, but it should be totally transparent. Jens (i

Re: Replication vs. Compaction

2014-01-31 Thread Jens Alfke
On Jan 31, 2014, at 9:46 AM, Mark Hahn wrote: > It wouldn't matter if it did. Within the same server linux short-circuits > http to make it the same as unix sockets, i.e. very little overhead. I think you mean it short-circuits TCP :) There's extra work involved in HTTP generation & parsing no

Re: Replication vs. Compaction

2014-01-31 Thread Mark Hahn
> replication within the same CouchDB server does not use HTTP It wouldn't matter if it did. Within the same server linux short-circuits http to make it the same as unix sockets, i.e. very little overhead. On Fri, Jan 31, 2014 at 9:43 AM, Jason Smith wrote: > On Sat, Feb 1, 2014 at 12:30 AM,

Re: Replication vs. Compaction

2014-01-31 Thread Jason Smith
On Sat, Feb 1, 2014 at 12:30 AM, Jens Alfke wrote: > > On Jan 31, 2014, at 8:39 AM, Boaz Citrin wrote: > > > Instead we could replicate the database into a new instance and > > switchover, as replication is much faster. > > That seems sort of strange to me — I would expect replication to an empt

Re: Replication vs. Compaction

2014-01-31 Thread Jens Alfke
On Jan 31, 2014, at 8:39 AM, Boaz Citrin wrote: > Instead we could replicate the database into a new instance and > switchover, as replication is much faster. That seems sort of strange to me — I would expect replication to an empty db to be slower, as it's doing basically the same work as com

Replication vs. Compaction

2014-01-31 Thread Boaz Citrin
Hello, We need to periodically need to compact our database. The problem is that this takes many hours and we cannot stay offline for so long. Instead we could replicate the database into a new instance and switchover, as replication is much faster. The question however is if both produces the sam