[
https://issues.apache.org/jira/browse/COUCHDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146413#comment-13146413
]
Randall Leeds commented on COUCHDB-1334:
----------------------------------------
Then again, Adam, we've got the SpiderMonkey issues to take care of and we
still need to figure out whether or not that changes 1.2 into 2.0 (unless maybe
we have enough new stuff for 1.2 that we would just release a
not-forward-compatible-with-SM-trunk major version anyway). We should
collectively make a decision about what to do with these branches soon.
> Indexer speedup (for non-native view servers)
> ---------------------------------------------
>
> Key: COUCHDB-1334
> URL: https://issues.apache.org/jira/browse/COUCHDB-1334
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core, JavaScript View Server, View Server
> Support
> Reporter: Filipe Manana
> Assignee: Filipe Manana
> Fix For: 1.2
>
> Attachments: 0001-More-efficient-view-updater-writes.patch,
> 0002-More-efficient-communication-with-the-view-server.patch,
> master-0002-More-efficient-communication-with-the-view-server.patch
>
>
> The following 2 patches significantly improve view index generation/update
> time and reduce CPU consumption.
> The first patch makes the view updater's batching more efficient, by ensuring
> each btree bulk insertion adds/removes a minimum of N (=100) key/value
> pairts. This also makes the index file size grow not so fast with old data
> (old btree nodes basically). This behaviour is already done in master/trunk
> in the new indexer (by Paul Davis).
> The second patch maximizes the throughput with an external view server (such
> as couchjs). Basically it makes the pipe (erlang port) communication between
> the Erlang VM (couch_os_process basically) and the view server more efficient
> since the 2 sides spend less time block on reading from the pipe.
> Here follow some benchmarks.
> test database at http://fdmanana.iriscouch.com/test_db (1 million documents)
> branch 1.2.x
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real 2m45.097s
> user 0m0.006s
> sys 0m0.007s
> view file size: 333Mb
> CPU usage:
> $ sar 1 60
> 22:27:20 %usr %nice %sys %idle
> 22:27:21 38 0 12 50
> (....)
> 22:28:21 39 0 13 49
> Average: 39 0 13 47
> branch 1.2.x + batch patch (first patch)
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real 2m12.736s
> user 0m0.006s
> sys 0m0.005s
> view file size 72Mb
> branch 1.2.x + batch patch + os_process patch
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real 1m9.330s
> user 0m0.006s
> sys 0m0.004s
> view file size: 72Mb
> CPU usage:
> $ sar 1 60
> 22:22:55 %usr %nice %sys %idle
> 22:23:53 22 0 6 72
> (....)
> 22:23:55 22 0 6 72
> Average: 22 0 7 70
> master/trunk
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real 1m57.296s
> user 0m0.006s
> sys 0m0.005s
> master/trunk + os_process patch
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real 0m53.768s
> user 0m0.006s
> sys 0m0.006s
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
