[
https://issues.apache.org/jira/browse/COUCHDB-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007423#comment-13007423
]
Filipe Manana commented on COUCHDB-1092:
----------------------------------------
Paul,
I made slight changes so that the metadata is calculated via EJSON encoding.
Then I prepend it to the body (removing its trailing } and the leading { from
the body) :
https://github.com/fdmanana/couchdb/blob/230bd82e6d712e80666082765ebb8d706e016386/src/couchdb/couch_doc.erl#L571
This doesn't seem to decrease performance compared to the previous tests I
presented.
I think this JSON binary manipulation is very little. In an ideal world we
wouldn't have it of course, but in practice software engineering is far from
perfect and I don't think we should wait eternally for some silver bullet.
People are, and have been, complaining about our huge database sizes and slow
view index generation.
Adam,
Right, the example is unfortunate since it has many floats. I updated the
branch to experiment with compressing the JSON binaries:
https://github.com/fdmanana/couchdb/commit/f44ab20876073a74f635cb9cd9ed5f8bb686dc05
I also replaced the floats with integers (simply erased all the '.') and
created a 25 000 documents database with that new template.
After compaction:
my branch database size: 63Mb
trunk database size: 224Mb
This is very significant. Running a relaximation test:
http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef63400cbc1
The writes are clearly better compared to trunk. The reads are however hard to
interpret, and I would say that there doesn't seem be significant difference
(for the worst or the best).
As for keeping metadata in headers, while it might be ok for single documents,
for case like _bulk_docs I'm clueless about how it would work. I also think
that in practice proxies impose limits for the maximum length of header values
(I think none of the HTTP RFCs impose maximum limits however). Nevertheless,
that would be a very big change and would pretty much break the entire API.
> Storing documents bodies as raw JSON binaries instead of serialized JSON terms
> ------------------------------------------------------------------------------
>
> Key: COUCHDB-1092
> URL: https://issues.apache.org/jira/browse/COUCHDB-1092
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Currently we store documents as Erlang serialized (via the term_to_binary/1
> BIF) EJSON.
> The proposed patch changes the database file format so that instead of
> storing serialized
> EJSON document bodies, it stores raw JSON binaries.
> The github branch is at:
> https://github.com/fdmanana/couchdb/tree/raw_json_docs
> Advantages:
> * what we write to disk is much smaller - a raw JSON binary can easily get up
> to 50% smaller
> (at least according to the tests I did)
> * when serving documents to a client we no longer need to JSON encode the
> document body
> read from the disk - this applies to individual document requests, view
> queries with
> ?include_docs=true, pull and push replications, and possibly other use
> cases.
> We just grab its body and prepend the _id, _rev and all the necessary
> metadata fields
> (this is via simple Erlang binary operations)
> * we avoid the EJSON term copying between request handlers and the db updater
> processes,
> between the work queues and the view updater process, between replicator
> processes, etc
> * before sending a document to the JavaScript view server, we no longer need
> to convert it
> from EJSON to JSON
> The changes done to the document write workflow are minimalist - after JSON
> decoding the
> document's JSON into EJSON and removing the metadata top level fields (_id,
> _rev, etc), it
> JSON encodes the resulting EJSON body into a binary - this consumes CPU of
> course but it
> brings 2 advantages:
> 1) we avoid the EJSON copy between the request process and the database
> updater process -
> for any realistic document size (4kb or more) this can be very expensive,
> specially
> when there are many nested structures (lists inside objects inside lists,
> etc)
> 2) before writing anything to the file, we do a term_to_binary([Len, Md5,
> TheThingToWrite])
> and then write the result to the file. A term_to_binary call with a binary
> as the input
> is very fast compared to a term_to_binary call with EJSON as input (or
> some other nested
> structure)
> I think both compensate the JSON encoding after the separation of meta data
> fields and non-meta data fields.
> The following relaximation graph, for documents with sizes of 4Kb, shows a
> significant
> performance increase both for writes and reads - especially reads.
> http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef63400b94f
> I've also made a few tests to see how much the improvement is when querying a
> view, for the
> first time, without ?stale=ok. The size difference of the databases (after
> compaction) is
> also very significant - this change can reduce the size at least 50% in
> common cases.
> The test databases were created in an instance built from that experimental
> branch.
> Then they were replicated into a CouchDB instance built from the current
> trunk.
> At the end both databases were compacted (to fairly compare their final
> sizes).
> The databases contain the following view:
> {
> "_id": "_design/test",
> "language": "javascript",
> "views": {
> "simple": {
> "map": "function(doc) { emit(doc.float1, doc.strings[1]); }"
> }
> }
> }
> ## Database with 500 000 docs of 2.5Kb each
> Document template is at:
> https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_2_5k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test.couch
> 1996 couchdb/tmp/lib/disk_json_test.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test.couch
> 2693 couchdb-trunk/tmp/lib/disk_ejson_test.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl
> http://localhost:5984/disk_json_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 6m6.740s
> user 0m0.016s
> sys 0m0.008s
> $ time curl
> http://localhost:5985/disk_ejson_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 15m41.439s
> user 0m0.012s
> sys 0m0.012s
> ## Database with 100 000 docs of 11Kb each
> Document template is at:
> https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_11k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test_11kb.couch
> 1185 couchdb/tmp/lib/disk_json_test_11kb.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> 2202 couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl
> http://localhost:5984/disk_json_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 4m19.306s
> user 0m0.008s
> sys 0m0.004s
> $ time curl
> http://localhost:5985/disk_ejson_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 18m46.051s
> user 0m0.008s
> sys 0m0.016s
> All in all, I haven't seen yet any disadvantage with this approach. Also, the
> code changes
> don't bring additional complexity. I say the performance and disk space gains
> it gives are
> very positive.
> This branch still needs to be polished in a few places. But I think it isn't
> far from getting mature.
> Other experiments that can be done are to store view values as raw JSON
> binaries as well (instead of EJSON)
> and optional compression of the stored JSON binaries (since it's pure text,
> the compression ratio is very high).
> However, I would prefer to do these other 2 suggestions in separate
> branches/patches - I haven't actually tested
> any of them yet, so maybe they not bring significant gains.
> Thoughts? :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
