Hello!
Just a random thought but would this also work by replicating the database
and deleting the old one?
That way the new database should stay available and not be bothered with
compaction. Of course the database then needs to be switched and then
replicated once again to capture all the new changes
On February 6, 2016 00:06:08 Anik Das <[email protected]> wrote:
Thanks Dave,
That was a very comprehensive article. We are currently running compaction
nightly. That's helping for now.
Regards,
On Feb 5 2016, at 7:36 pm, Dave Cottlehuber <[email protected]> wrote:
On Tue, 2 Feb 2016, at 10:08 PM, Anik Das wrote:
> Hello All,
>
> We were developing an application where we had to insert approximately
> 600,000
> documents into a database. The database had only one view (value emitted
> as
> null).
>
>
>
> It was not a batch insertion. After the insertion the database took up
> 3.5GB
> to our wonder. I googled around and did a compact query. After the
> compact
> query the size reduced to 350MB.
>
>
>
> I am new to couchdb and I'm unable to figure out what exactly is
> happening/happened.
>
>
>
> Anik Das
Welcome Anik :-)
Some quick points:
\- we use a B tree in CouchDB
\- its append only
\- to find a doc we walk down the tree from the root node
\- the root node is always the last node in the .couch btree file
\- adding or updating a doc requires appending (in order) the doc, and
intermediary levels, and finally the new root node of the tree
\- thus a single doc update needs to rewrite at least 2 nodes, itself +
the new root
\- as the tree gets wider (more leaf node documents) it also grows slowly
and increases levels
\- this adds more intermediate nodes to be updated as we go along
<http://horicky.blogspot.co.at/2008/10/couchdb-implementation.html> is a
very nice but old picture of this.
You should always plan to compact after a big upload or replication, but
a couple of things will ease the pain:
\- use _bulk_docs (and do some testing for optimum chunk size)
\- upload docs in uuid order (don't rely on couch generated uuids)
both of these reduce the number of interim updates to the tree, the
first simply by only rewriting at the end of each bulk update, the the
last by adding data in sorted order, less intermediary nodes need
updating.
Most people run compaction through a cron job or similar out of hours
scheduling tool.
A+
Dave
